Method and system for providing medical video reports

Abstract

The invention relates to a method and system for providing medical video reports. In particular, it is proposed to extract contextual information (CI) relating to a medical image research (MIS) and to select (S30) a template video script (TVS) from a plurality of candidate template video scripts (CVS) using the contextual information (CI). Candidate template video scripts (CVS) encode instructions for image processing steps of medical image studies (MIS), respectively, to generate videos for human recipients (HR) that visualize medical diagnostics. The selected template video script (TVS) may then be executed to generate a medical video report (MVR).

Description

Technical Field

[0001] Embodiments of the present invention relate to methods and systems for providing medical video reports. In particular, embodiments of the present invention relate to the automatic generation of medical video reports corresponding to medical image studies. In addition, embodiments of the present invention relate to generating medical reports for lay persons such as patients. Background Art

[0002] Medical text reports are the primary medical documents that forward medical diagnoses to stakeholders of a case. Medical reports are not only exchanged between doctors, but also provided to patients. The generation of medical text reports is subject to time and legal constraints. Therefore, medical text reports are usually not guidance readings. In addition, the style of medical reports may vary significantly between different doctors.

[0003] Therefore, most patients have difficulty understanding medical text reports. Often, medical reports use terms and expressions that are unfamiliar and incomprehensible to non-medical professionals. Therefore, when faced with medical reports that are not conducive to patient participation, patients feel uneasy and ignored.

[0004] Even many referring doctors sometimes have difficulty interpreting the medical reports they receive from specialist doctors. In addition, treating doctors may face questions from their patients about the medical reports and have to invest extra time to explain the reports and clarify misunderstandings on the patient's side.

[0005] In order to improve the situation, it has been proposed to use radiology report templates in simplified language. However, these are currently only available for a few standardized treatments. Moreover, preparing such simplified versions in addition to the required medical-legal text reports constitutes additional work for doctors, for whom tight clinical workflows often do not provide sufficient time. Summary of the invention

[0006] It is therefore an object of embodiments of the present invention to provide improved methods and systems that achieve an improved way of communicating clinical results to recipients. In particular, it is an object of embodiments of the present invention to achieve the creation of an artifact that allows the recipient to better understand the medical context and meaning of a particular result. Furthermore, it is an object to achieve the generation of such an artifact without significantly increasing the workload of the diagnostician / radiologist and with minimal disruption to the reading and reporting workflow.

[0007] This object is solved by a method for providing a medical video report, a corresponding system, a corresponding computer program product and a computer-readable storage medium according to embodiments of the present invention.Alternative and / or preferred embodiments are also objects of the present invention.

[0008] In the following, the technical solution according to the present invention is described with respect to the claimed device and with respect to the claimed method. The features, advantages or alternative embodiments described herein can also be assigned to other claimed purposes, and vice versa. In other words, the embodiment of the method of the present invention can be improved by the features described or claimed with respect to the device. In this case, for example, the functional features of the method are realized by the target unit or element of the device.

[0009] According to one aspect, a computer-implemented method for providing a medical video report is provided. The method includes several steps. One step involves receiving a medical image study of a patient. Another step involves determining contextual information related to the medical image study based on the medical image study. Another step involves selecting a template video script from a plurality of candidate template video scripts based on the contextual information, each candidate template video script encoding the following instructions: the instructions are used for image processing steps of the medical image study to generate a video related to the medical image study (or: video visualization information related to the medical image study) for a human recipient. Another step involves executing the selected template video script on the medical image study to generate a medical video report. Another step involves providing the medical video report.

[0010] Medical image studies may include a three-dimensional dataset providing three dimensions in space or two dimensions in space and one dimension in time, a two-dimensional dataset providing two dimensions in space, and / or a four-dimensional dataset providing three dimensions in space and one dimension in time.

[0011] A medical image study may depict a body part of a patient in the sense that it contains three-dimensional image data of the body part of the patient. A medical image study may represent an image volume. A body part of a patient may be included in the image volume.

[0012] Medical image studies include image data in the form of two-dimensional or three-dimensional arrays of pixels or voxels, for example. Such arrays of pixels or voxels can represent intensity, absorption or other parameters as a function of three-dimensional position and can be obtained, for example, by suitable processing of measurement signals obtained by a medical imaging modality.

[0013] Medical imaging modalities correspond to systems for generating or producing medical image data. For example, medical imaging modalities may be computed tomography systems (CT systems), magnetic resonance systems (MR systems), angiography (or C-arm X-ray) systems, positron emission tomography systems (PET systems), ultrasound imaging systems, and the like. Specifically, computed tomography is a widely used imaging method and utilizes "hard" X-rays generated and detected by special rotating instruments. The resulting attenuation data (also referred to as raw data) is presented by computer analysis software that produces detailed images of the internal structures of the patient's body parts. The resulting image set is referred to as a CT scan, which may constitute a plurality of series of continuous images, presenting the internal anatomical structures in cross-sections perpendicular to the axis of the human body. Magnetic resonance imaging (MRI), which is used to provide another example, is an advanced medical imaging technique that utilizes the effect of an effective magnetic field on the movement of protons. In an MRI machine, the detector is an antenna, and the signal is analyzed by a computer to create a detailed image of the internal structure in any part of the human body.

[0014] Medical image studies may be stored in a standard image format such as the Digital Imaging and Communications in Medicine (DICOM) format and stored in a memory or computer storage system such as a Picture Archiving and Communication System (PACS), a Radiology Information System (RIS), a Vendor Neutral Archive (VNA), an Electronic Health Record (HER) storage device or database, etc. Whenever DICOM is mentioned herein, it is understood that DICOM refers to the "Digital Imaging and Communications in Medicine" (DICOM) standard, for example, according to the DICOM PS3.12020c standard (or any later or earlier version of the standard).

[0015] The medical video report can be configured to show the situation, facts and / or findings associated with the medical image study to a human recipient in a video format. In particular, the human recipient can be a patient of the medical image study. In addition, the human recipient can be a physician involved in the patient's treatment, in particular a referring physician or a treating physician.

[0016] The medical video report may include a plurality of individual images in the form of an image sequence. Individual images of the medical video report may include a representation of a medical image study that has been rendered based on image data included in the medical image study. In addition, individual images of the medical video report may include rendered annotations related to observations or findings in the medical image study.

[0017] Furthermore, individual images of the medical video report may be based on image data which were not included in the medical image study.In particular, individual images or sequences of images may be preconfigured or may have been pregenerated.

[0018] A medical video report may include multiple (logical) blocks, each of which is configured to show a given fact or situation related to a medical image study. For example, one block may show the reasons and circumstances for the underlying examination, while other blocks are each dedicated to showing certain findings derived from the examination.

[0019] The context information may include or characterize circumstances of the case relevant to the generation of the medical video report. In particular, the context information may include values ​​of a plurality of predefined parameters (as information). The predefined parameters may be relevant to the generation of the medical video report.

[0020] For example, the context information may include information characterizing the medical image study (e.g., as values ​​of a plurality of predefined parameters). Additionally or alternatively, the context information may include information characterizing the results of an analysis of the medical image study (e.g., as values ​​of a plurality of predefined parameters). Additionally or alternatively, the context information may include information characterizing a human recipient of the medical video report (e.g., in terms of the recipient's level of expertise) (e.g., as values ​​of a plurality of predefined parameters).

[0021] According to some examples, the contextual information may include semantic information extracted from the medical image study, corresponding to, for example, the type of image study, imaging parameters, image data shown, patient, etc.

[0022] The context information may be automatically extracted from the medical image study (and any supplementary information). Specifically, values ​​corresponding to one or more predefined parameters may be extracted from the medical image study (and any supplementary information).

[0023] For example, a DICOM data structure of a medical image study may be queried for values ​​indicating the type of image study and / or the body part or organ imaged in the image study, etc. Similarly, any supplementary information may be mined for relevant parameters such as the patient's age or gender.

[0024] According to some examples, a video script may be envisioned as a set of computer-readable instructions that, when executed by a video rendering engine or the like, cause the generation of a video. The video script may define certain image processing operations by which raw image data of a medical image study may be converted into a video for a human recipient. Furthermore, the video script may be configured to provide a voice-over of natural language text accompanying the video image.

[0025] Specifically, the video script may include instructions for processing raw image data to produce a pre-scripted sequence of scenes or individual images that may form part of a medical video report. This may include pre-processing parameters (e.g., contrast enhancement), image rendering parameters (e.g., volume rendering parameters), parameters defining translation, scaling, or cropping of image data, parameters defining annotations to be added to images rendered for a medical video report, etc.

[0026] Each template video script may define a specific type of medical video report to be generated for a specific use case. Thus, a use case may be characterized by contextual information. A use case may relate to a clinical question or decision underlying a medical image study. Furthermore, a use case may relate to the technical environment (e.g., modality and imaging parameters) under which a medical image study was acquired and / or under which image data was transformed into a medical video report.

[0027] Each candidate template video script may be pre-configured for a specific use case. This may be done by an expert user or by an automated process configured to derive the script from existing (validated) medical video reports.

[0028] Each of the candidate template video scripts can be associated with a specific use case, which can be defined by one or more predefined parameters or by values ​​corresponding to one or more predefined parameters. According to some examples, the predefined parameters can be the same predefined parameters as in the context information.

[0029] According to some examples, the step of selecting an appropriate template video script from the candidate template video scripts may include comparing the context information with the candidate template video scripts, and matching the candidate template video script that (best) adapts the context information to the context information. According to some examples, this may involve comparing a value characterizing the context information with a corresponding value characterizing a use case of the corresponding candidate template video script. Specifically, this may involve comparing the values ​​of a set of predefined parameters of the context information with corresponding values ​​of a set of predefined parameters of the candidate template video script.

[0030] It should be noted that the use of predefined parameters is only an example. According to other examples, the correct template video script can also be selected based on the semantic matching of the context information with the candidate template video script. Specifically, this can include: obtaining (semantic) information from the candidate template video script; and selecting the selected template video script based on the obtained (semantic) information, for example, by comparing the obtained (semantic) information with the context information or corresponding semantic information obtained from / included in the context information. The semantic information obtained for the candidate template video script can correspond to the context information.

[0031] According to some examples, the step of executing the selected template video script may include inputting the selected template video script and the medical image study into a rendering engine configured to process the image data according to the template video script. In other words, the step may include controlling the rendering engine based on the selected template video script.

[0032] According to some examples, providing the medical video report can include making it accessible to the human recipient, for example, by forwarding the medical video report to the human recipient or by providing a download link to the human recipient. In addition, providing the medical video report can include storing the medical video report in an appropriate storage device, from which the human recipient can access the medical video report, for example, by downloading the medical video report.

[0033] Due to the provision of multiple pre-configured template video scripts and the automatic selection of appropriate scripts based on patient cases, instructions for automatically generating medical video reports can be provided in an efficient manner. With this, medical video reports can be provided without the need for doctors to specify instructions for generating medical video reports. Therefore, the workload of doctors is not significantly increased. In addition, the proposed method is seamlessly integrated in the usual reading and reporting workflow, because medical video reports can be automatically generated in parallel with the usual reading and reporting. On the other hand, the report in video format provides the recipient with an instructive summary of the case's circumstances.

[0034] According to some examples, the contextual information includes at least one of: the medical reason for the medical image study, the suspected diagnosis of the patient, the medical imaging modality used for the patient, the body part and / or organ depicted in the medical image study, demographic information of the patient, the expertise level of the human recipient, and / or user input based on the user's interaction with the medical image study or a representation thereof.

[0035] According to some examples, the (semantic) information obtained respectively for the candidate template video scripts reflects (corresponds to) contextual information, and may also indicate at least one of: the medical reason for the medical image study, the suspected diagnosis of the patient, the medical imaging modality used for the patient, the body part and / or organ depicted in the medical image study, the patient's demographic information and / or the expertise level of the human recipient, and a corresponding candidate video template structure is envisioned for the candidate template video script.

[0036] Using the above information, reliable association of appropriate template video scripts can be made for the corresponding cases.

[0037] According to some examples, the step of determining the context information includes: obtaining association data associated with the medical image study; and determining the context information additionally based on the association data.

[0038] The associated data may be obtained by, for example, querying one or more databases based on a digital patient identifier. The one or more databases may be included in a healthcare information system such as a picture archiving or communication system (PACS), an electronic health record (EHR), a radiology information system (RIS), and / or a laboratory information system (LIS).

[0039] The associated data may include at least one of: a (previous) medical report of the patient, a previous medical video report, a further (previous) medical image study of the patient, one or more laboratory values ​​of the patient and / or an electronic health record of the patient.

[0040] By utilizing the associated data, contextual information can be determined more precisely, which in turn can facilitate the selection of an appropriate template video script.

[0041] According to some examples, the medical video report includes a portion that shows a human recipient one or more aspects of the medical image study, such as the type, reasons for conclusions drawn from the medical image study and / or subsequent recommendations based on the medical image study.

[0042] Using this, human recipients can gain greater clarity about what is happening in medical image studies, which improves transparency and patient engagement.

[0043] According to one aspect, the method further comprises obtaining a medical text report related to the medical image research, wherein, in the determining step, context information is extracted from the medical text report.

[0044] The medical text report may include structured or unstructured text related to the patient's medical observations and conclusions based on the patient's examination results (e.g., medical image studies or laboratory reports). In addition, the medical text report may also include one or more images and / or charts extracted from the examination results.

[0045] The medical text report may be an already existing and finalized report that may be actively pulled from a database (e.g., as associated data). According to other examples, the medical text report may be in the process of being generated "on the fly" by a user. Thus, the user may be someone currently reviewing a medical image study, such as a doctor, particularly a radiologist. Thus, a user is typically different from a human recipient. User interactions for generating a medical text report may be recorded and used to determine contextual information.

[0046] If not otherwise indicated, the term "medical report" refers to a textualized medical report or a medical text report and not to a medical video report.

[0047] Medical reports can provide a concise and well accessible representation of the case's circumstances. Therefore, medical reports provide a good basis for extracting contextual information. This, in turn, enables efficient finding of suitable template video scripts.

[0048] According to some examples, the medical text report is based on a text report template, wherein the context information includes the text report template (or a type of the text report template).

[0049] In other words, the template video script is selected according to the text report template for the medical text report. This has the advantage that a medical video report can be generated that optimally supplements the medical text report.

[0050] According to some examples, a text report template is selected from a plurality of preconfigured text report templates, wherein the method includes providing a plurality of candidate template video scripts such that each candidate template video script is associated with one of the plurality of preconfigured text report templates, and in the step of selecting a template video script, selecting a template video script associated with the text report template from the candidate template video scripts.

[0051] In other words, the saved available candidate template video scripts match the report template to be used to generate the medical text report. This makes the selection process of the template video script clear. In addition, it can be ensured that the medical video report optimally supplements the medical text report.

[0052] According to one aspect, the context information includes medical findings of predefined finding types, and the selected template video script encodes instructions for image processing steps of a medical image study to generate a video for a human recipient visualizing the medical findings of the predefined finding types.

[0053] According to some examples, in the step of selecting, the selected template video script is selected based on the predefined discovery type. Thus, according to some examples, each of the candidate template video scripts may encode the following instructions: the instructions are used to visualize different predefined discovery types, wherein in the step of selecting, at least one template video script that encodes instructions for visualizing the predefined discovery type is identified from the candidate template video scripts.

[0054] According to one aspect, a computer-implemented method for providing a medical video report is provided, the method comprising the following steps: receiving a medical image study of a patient; obtaining medical findings related to the image study; selecting a template video script from a plurality of candidate template video scripts based on the finding type, each candidate template video script encoding the following instructions: the instructions are used for image processing steps of the medical image study to generate a video that visualizes the medical findings of a specific finding type for a human recipient; applying the selected video template elements to the medical image study to generate a medical video report; and providing the medical video report.

[0055] Each medical finding may be associated with corresponding image data in a medical image study. The medical finding may indicate a certain condition or pathology of the patient. The condition or pathology may be associated with a diagnosis of the patient.

[0056] The medical findings may be related to anatomical structures that distinguish the patient from other patients. The medical findings may be located within different organs of the patient (e.g., within the patient's lungs, or within the patient's liver) or between the patient's organs. In particular, the medical findings may also be related to foreign bodies.

[0057] In particular, the medical findings may be associated with a neoplasm (also referred to as a "tumor"), in particular a benign neoplasm, an in situ neoplasm, a malignant neoplasm and / or a neoplasm of uncertain / unknown behavior. In particular, the medical findings may be associated with a nodule, in particular a lung nodule. In particular, the medical findings may be associated with a lesion, in particular a lung lesion.

[0058] Medical findings may be obtained from an existing medical text report or a medical text report currently being created by a user. Additional medical findings may be obtained by applying image processing algorithms configured to identify medical findings in a medical image study (i.e., computer-aided detection functions). Furthermore, medical findings may be obtained directly from a user interacting with the medical image study (e.g., by applying measurement tools to a representation / visualization of the medical image study).

[0059] Medical findings may be classified according to their type or category. The type or category is referred to as a "finding type". The finding type may specify the general nature of the medical finding. In addition, the finding type may specify the anatomical structure or organ in which the medical finding has been found. According to some implementations, the finding type may also be conceived as a label for the medical finding. For example, the finding type may be a lung nodule, a liver nodule, a cyst, a rib fracture, an unidentified lesion, etc.

[0060] Template video scripts can be efficiently provided and selected by depending on the type of finding (and pre-configuring new template video scripts that conform to multiple types of finding). With this, appropriate video sequences that strictly reflect the circumstances of the case can be generated.

[0061] According to some examples, the image processing step includes: determining an organ corresponding to a medical finding; segmenting the organ in a medical image study by applying an image segmentation algorithm to generate segmented image data; and rendering a predefined image sequence based on the segmented image data, wherein the medical finding is highlighted in at least a portion of the images of the sequence.

[0062] Using this, medical findings can be appropriately created and brought to the attention of human recipients.

[0063] According to some examples, the contextual information also includes supplemental information related to the medical finding, and the selected video template is further configured to transform the supplemental information into a video that visualizes the medical finding, so that the supplemental information is perceptible to a human recipient, preferably in the form of visual markers. According to some examples, the supplemental information includes at least one of: a criticality of the medical finding, a progression of the medical finding over time, a classification of the medical finding, and / or a medical score of the medical finding.

[0064] According to some examples, the selected template video script may additionally encode instructions for an image processing step of a medical image study to generate a video that visualizes additional medical findings of a different finding type than the predefined finding types.

[0065] According to some examples, additional findings may be related to incidental findings. Using this, additional or incidental findings may be visualized even though the template video script may be primarily related to another finding or another subject, such as a therapeutic video explaining a desired treatment to a human recipient.

[0066] According to some examples, the context information includes user input related to the medical image study. Specifically, the user input may be input via a user interface and may relate to visible medical findings under a representation of the medical image study visualized in the user interface.

[0067] The representation may be a two-dimensional representation image rendered from a medical image study for display to a user in a user interface. The representation may include a plurality of image pixels. In particular, the representation may be a two-dimensional rendering of the medical image. The two-dimensional rendering may typically rely on known rendering processes, such as ray casting, ray tracing, texture rendering, etc. According to some examples, the rendering may cause the image data in combination with the medical image to display the identified medical findings and / or any candidate medical findings.

[0068] The user input may be any input related to specifying a medical finding. The user input may include voice commands or any other, particularly physical input entered into a user interface, particularly a graphical user interface. For example, the user may use an input device like a computer mouse, a trackball device, a smart pen, a touchpad, a touch-sensitive display, etc. In addition, the user input may be captured by eye tracking or by tracking gestures. In particular, the user input may include directly specifying a medical finding in the representation, for example by clicking, drawing an outline, or invoking a measurement tool in a specific location in the representation. The user input may be such that it completely lists the medical findings, or such that it only indicates a portion of the representation or even only indicates a point in the representation that is then automatically associated with a possible medical finding.

[0069] According to some implementations, the user input may include multiple separate user interactions with the user interface (eg, user input regarding representations, display settings, general settings, measurements, etc.).

[0070] According to some examples, the candidate template video scripts respectively include multiple optionally activatable video template structures, each video template structure encodes the following instructions: the instructions are used to provide a portion of a medical video report based on medical image research, and the execution steps include: activating at least one of the video template structures of the selected video script based on context information; and generating a medical video report based on at least one activated video template structure.

[0071] In other words, each template video script may include a plurality of independent blocks or elements that are in principle suitable for visualizing various aspects of the use case described by the context information. Depending on the actual available information, these blocks may be selectively activated. Each activated block may then correspond to a portion of the final medical video report.

[0072] According to some examples, individual structures of a script may be associated with different medical findings and / or standard video sequences for a particular use case. For example, a script for an X-ray scan of a patient's lungs may include structures associated with general instructions about the X-ray exam and structures for possible lung-related findings such as lung nodules or interstitial lung disease.

[0073] Thus, individual template structures can be shared between different template video scripts.For example, each script related to a CT examination as a medical image study can include the same template structure with some general instructions about the CT examination.

[0074] Template video scripts are made more flexible by pre-configuring different structures in the script that can be activated according to the respective use case. Template video scripts can therefore be well integrated into existing workflows.

[0075] According to one aspect, an optionally activatable video template structure of a candidate template video script includes a finding-specific video template structure having instructions for visualizing a medical finding of a predefined finding type. The method also includes obtaining at least a medical finding of a given finding type related to the medical image study from the medical image study and / or any associated information of the medical image study, and the step of activating includes activating at least one finding-specific video template structure of the selected template video script based on the given finding type, the predefined finding type of the at least one finding-specific video template structure matching the given finding type.

[0076] By pre-configuring different instructions for different discovery types, template video scripts can be rendered more flexibly and smoothly integrated into existing workflows.

[0077] According to some examples, the step of activating at least one video template structure includes determining an attribute of the medical finding and activating at least one video template structure based on the attribute. According to some examples, the attribute may include at least one of: the size of the medical finding, the criticality of the medical finding, the progression of the medical finding over time, the classification of the medical finding, and / or the medical score of the medical finding. With this, building blocks of a medical video report that reflect the attributes of the medical finding can be selected. For example, if a nodule is smaller than a certain size, this allows a different video sequence to be generated compared to if the nodule is larger.

[0078] According to an aspect, the step of obtaining at least one medical finding comprises querying whether the medical image study indicates one or more medical findings of a predefined finding type.

[0079] In other words, medical image studies can be proactively queried for possible findings based on the selected template video script. With this, another layer of security can be provided to ensure that relevant findings for the use case are not overlooked.

[0080] According to some examples, the step of querying includes automatically applying an image processing algorithm to the medical image study, the image processing algorithm configured to identify at least one of the predefined medical findings, the selected template video script including a video template structure for the predefined medical findings.

[0081] Automatic invocation of computer-aided inspection tools (in the form of image processing algorithms) also reduces user workload, and reporting workflows can also be automated.

[0082] According to some examples, the step of querying includes notifying the user of at least one of the predefined medical findings, the selected template video script including a video template structure for the predefined medical finding.

[0083] Based on the notification, the user can then view whether the medical image study includes any indication for at least one finding. With this, the user is provided with additional guidance on what to look for.

[0084] According to some examples, the step of querying includes processing a medical text report associated with the medical image study to determine whether the medical text report indicates one or more medical findings of a predefined finding type.

[0085] According to one aspect, the candidate template video scripts each include at least one video template structure associated with a pre-existing video sequence, and the steps performed include retrieving the pre-existing video sequence from a repository configured to store a plurality of pre-existing video sequences and including the pre-existing video sequence in the medical video report.

[0086] According to some examples, the method may also include providing a repository.

[0087] By using pre-existing video sequences, content related to standard video clips independent of image data in medical image research can be kept available. This can involve illustrative or introductory sequences illustrating a certain image examination. Thus, content can be pre-generated and reused, which makes the method more efficient and consumes less computing resources.

[0088] According to one aspect, the method further includes: obtaining natural language text from the contextual information and / or the medical image study; generating a voice-over based on the natural language text; and including the voice-over in the medical video report.

[0089] According to some examples, obtaining the natural language text may include (directly) extracting the natural language text from a structured or unstructured text element included in the medical image study or the context information. Such a text element may be a DICOM tag of the medical image study or a text field of a medical text report. According to other examples, obtaining the natural language text may include transmitting non-text data in the natural language text. According to some examples, obtaining may include applying a text generation function to the medical image study and / or the context information to obtain the natural language text.

[0090] According to some examples, the step of generating a voice-over may include inputting natural language text in a chat function configured to generate speech output based on the natural language text. Thus, the chat function may be configured to filter, summarize, or re-spell the input natural language text to generate an output suitable for accompanying the voice-over of the medical video report. In addition, the step of generating a voice-over may include generating a soundtrack of the video medical report based on the voice-over.

[0091] According to some examples, the text generation function and / or the chat function may include a transformer network, and in particular a large language model such as BERT or ChatGPT.

[0092] A transformer network is a neural network architecture that typically includes an encoder, a decoder, or both. In some instances, the encoder and / or decoder are composed of several corresponding encoding and decoding layers, respectively. Within each encoding and decoding layer there is an attention mechanism. The attention mechanism - sometimes referred to as self-attention - relates data items (e.g., words or pixels) within a series of data items to other data items within the series. For example, the self-attention mechanism allows the model to examine groups of voxels or phrases within a medical image or sentence and determine the relative importance of other groups of voxels or phrases within the medical image or sentence to the group of voxels or phrases being examined.

[0093] In particular, the encoder can be configured to transform the input (medical image or text) into a digital representation. The digital representation can include a vector for each input token (e.g., each word). The encoder can be configured to implement an attention mechanism so that each vector of tokens is influenced by other tokens in the input. In particular, the encoder can be configured so that the representation parses the desired output of the transformer network.

[0094] In particular, the decoder can be configured to transform an input into a sequence of output tokens. In particular, the decoder can be configured to implement a masked self-attention mechanism so that each vector of tokens is only influenced by other tokens on one side of the sequence. Furthermore, the decoder can be autoregressive, meaning that intermediate results (e.g., previously predicted token sequences) are fed back.

[0095] According to some examples, the output of the encoder is input into a decoder.

[0096] Additionally, the transformer network may include a classification module or unit configured to map the output of the encoder or decoder to a set of learned outputs, such as natural language text or voice-over text.

[0097] According to some examples, the training of the transformer model can occur in two stages, a pre-training stage and a fine-tuning stage. In the pre-training stage, the transformer model can be trained on a large data corpus to learn the potential semantics of the problem. Such a pre-trained transformer model can be used for different languages. For some applications described herein, fine-tuning can include further training the transformer network with medical texts and / or medical ontologies (e.g., RADLEX and / or SNOMED) with professional annotation meanings. In particular, using the latter, the transformer model according to some examples can learn typical relationships and synonyms of medical expressions.

[0098] For a review of Transformer Networks, see Vaswani et al., “Attention Is All You Need,” arXiv:1706.03762, June 12, 2017, which is incorporated herein by reference in its entirety.

[0099] By providing voice-over, medical video reports can be made more instructive and accessible to human recipients, while at the same time, the workload of clinical staff is not increased due to the automatic generation of voice-over.

[0100] The advantage of transformer networks is that they can efficiently handle long-range dependencies in the input data due to the attention mechanism. In addition, the encoder used in transformer networks is able to process data in parallel, which saves computational resources in inference. Furthermore, due to autoregression, the decoder of the transformer network is able to iteratively generate sequences of output tokens with great confidence.

[0101] According to some examples, the step of generating a voiceover may include providing a voiceover template and filling the voiceover template based on the natural language text. According to some examples, providing the voiceover template may also include: providing a voiceover template for each video template structure; and selecting at least one voiceover template according to the activated video template structure.

[0102] According to one aspect, in the step of obtaining natural language text, the natural language text is extracted from a medical text report.

[0103] By mining medical text reports, information directly related to medical image research can be collected, and in addition, this does not require much processing to convert this information into voice-over. Specifically, by relying on medical text reports, paragraphs of the report can be adopted into the voice-over word by word.

[0104] According to some examples, the step of generating the voice-over may include listening to speech input into the user interface by a user, and generating the voice-over based on the input speech.

[0105] In other words, the voiceover can be generated at least in part based on the user's direct dictation, which simplifies the workflow. Since doctors tend to dictate their findings to generate medical text reports anyway, this also does not affect the user's workload.

[0106] According to one aspect, the step of obtaining the natural language text includes providing a mapping function configured to map the medical findings onto the structured text, and generating the natural language text by applying the mapping function to the medical findings.

[0107] According to some examples, the mapping function may be a text generation function as described herein. In particular, the mapping function may include a transformer network.

[0108] The use of mapping functions enables structured processing of available information into actionable results that can be easily used by subsequent video generation.

[0109] According to one aspect, the step of generating a voice-over includes: providing a chat function configured to transcribe a natural language text from a first version to a second version; applying the chat function to the natural language text to transcribe the natural language text into the second version; and, in the step of generating a voice-over, generating the voice-over based on the second version of the natural language text.

[0110] The chat function can be configured according to aspects and examples of the chat function described herein. In particular, the chat function can include a transformer network and / or a large language model.

[0111] According to some examples, the first version of the natural language text may be text as extracted from available data. According to some examples, the second version may be suitable for human recipients. According to some examples, the second version may be suitable for human recipients based on information about the human recipients included in the context information. According to some examples, the second version may correspond to a different language, a different level of complexity, in particular simpler language, and / or a different level of text summarization, in particular more concise language, compared to the first version. The simpler language may be characterized by avoiding technical terms included in the first version and / or replacing technical terms included in the first version with lay terms in the second version.

[0112] According to some examples, the second version, and in particular at least one target attribute of the second version, may be determined based on the context information and / or the selected template video script and / or the activated video template structure (e.g., by a suitably configured chat function). The at least one target attribute may include a target complexity level of the language used, a target length of the natural language text obtained in the second version, a target language of the natural language text in the second version, etc.

[0113] By transmitting available text information before generating the audio track of the medical video report on this basis, the voice-over can be automatically adapted to the needs of, for example, a human receiver. With this, a result that can be easily used is generated without requiring user intervention.

[0114] According to one aspect, a medical image study comprises a three-dimensional medical image dataset, a template video script encodes instructions for volumetric image rendering, in particular implementing a rendering process based on path tracing or ray casting, and a medical video report comprises one or more images generated via the volumetric image rendering.

[0115] In ray casting, simulated rays emitted from the eyes of an imaginary observer are transmitted through the examined body or object (see Levoy: "Display of Surfaces from Volume Data", IEEE Computer Graphics and Applications, Issue 8, Vol. 3, May 1988, pp. 29-37). Along the rays, RGBA values ​​are determined for sample points from voxels and combined with the help of alpha synthesis or alpha blending to form pixels of the two-dimensional image. Here, the letters R, G and B in the expression RGBA represent the color components red, green and blue, which constitute the color contribution of the corresponding sample point. A represents the ALPHA value, which represents a measure of transparency at the sample point. The corresponding transparency is used for the superposition of the RGB values ​​at the sample point to form a pixel. The lighting effects are usually taken into account with the help of lighting models within the scope of the method known as "shading".

[0116] Another method for volume rendering is the so-called path tracing method (see Kajiya: "The renderingequation", ACM SIGGRAPH Computer Graphics, Issue 20, Volume 4, August 1986, pages 143 to 150). Here, multiple simulated rays are injected into the volume data of each visualization pixel, and then the simulated rays interact with the volume, that is, the simulated rays are reflected, refracted or absorbed, wherein at least one random ray is generated each time (except in the case of absorption). Therefore, each simulated ray finds its path through the volume data. The more virtual rays used for each visualization pixel, the better the image. Here, in particular, the processing and method described in EP 3 178 068 B1 can be used. The content of EP 3 178 068 B1 is fully incorporated herein by reference.

[0117] Thus, a template video script may specify parameters for the path and / or ray casting process, such as zoom level, viewing angle, transfer function, texture values, number of rays, transparency level, scene lighting, and the like.

[0118] On the one hand, such an approach allows to generate particularly realistic visualizations. This provides the human recipient with an instructive picture of the image inspection and its results. On the other hand, since the volumetric image rendering is triggered automatically, the user does not need to participate, which saves the user from having to familiarize herself or himself with the subtleties of the volume rendering pipeline (which can be complex).

[0119] According to some examples, the step of providing the medical video report may include: providing the medical video report to the user for review in a user interface; receiving feedback from the user indicating approval of the medical video report; and publishing the medical video report to a human recipient based on the feedback. According to some examples, providing the medical video report to the user includes providing the voice-over as text for review.

[0120] According to some examples, providing the medical video report to the user may include visualizing different blocks of the medical video report (which blocks may each be associated with a separate video template structure) through thumbnails according to some examples. According to some examples, the feedback includes consent for the respective blocks, and publishing includes including those blocks in the medical video report for which consent was included in the feedback.

[0121] By showing the video medical report to the user before publishing it to a human recipient, end control can be performed by the user. With this, it can be ensured that the medical video report is of sufficient quality and complies with applicable regulations. At the same time, end control is enabled under a manageable workload. This is the case in particular if the different blocks of the medical video report are presented visually, as this allows a good overview of the content "at a glance".

[0122] According to one aspect, a system for providing a medical video report is provided. The system includes an interface unit and a computing unit. The interface unit is configured to receive a medical image study of a patient and to provide a medical video report. The computing unit is configured to determine contextual information related to the medical image study based on the medical image study. In addition, the computing unit is configured to select a template video script from a plurality of candidate template video scripts based on the contextual information, each candidate template video script encoding instructions for an image processing step of the medical image study to generate a video that visualizes a medical diagnosis for a human recipient. In addition, the computing unit is configured to execute the selected template video script on the medical image study to generate a medical video report. In addition, the computing unit is configured to provide the medical video report.

[0123] The computing unit can be implemented as a data processing system or as part of a data processing system. Such a data processing system may, for example, include a cloud computing system, a computer network, a computer, a tablet computer, a smart phone, etc. The computing unit may include hardware and / or software. The hardware may include, for example, one or more processors, one or more memories, and combinations thereof. One or more memories may store instructions for executing the method steps according to the present invention. The hardware may be configurable and / or operable by software. Typically, all units, subunits, or modules may, for example, exchange data with each other at least temporarily via a network connection or a corresponding interface. Therefore, the individual units may be located separately from each other. In addition, the computing unit may be configured as an edge device.

[0124] According to some examples, the computing unit may include a context extraction module or unit configured to extract context information from a medical image study and / or any associated data. In addition, the computing unit may include a video content module or unit configured to select a template video script based on the context information. In addition, the computing unit may include a rendering module or unit configured to render a medical video report based on the selected script and the medical image study. According to some examples, the computing unit may include a voice-over module or unit configured to generate text based on available information and data, that is, context information, medical image studies, and any associated information (e.g., medical findings identified in the medical image studies). According to some examples, the voice-over module may include: a discovery 2 text engine configured to extract natural language text from available information; and a chat engine configured to transcribe the extracted text into a version suitable for the sound track of the medical video report.

[0125] The interface unit may include an interface for exchanging data with a template database for parsing the template database and retrieving a template video script. The interface unit may also be adapted to interface with one or more users of the system, for example, by receiving queries (e.g., in a graphical user interface) and / or displaying the results of the processing, i.e., information, to the user.

[0126] The system can be adapted to implement the method of the present invention in its various aspects for modifying medical image data. The advantages described in conjunction with the method aspects and examples can also be achieved by correspondingly configured system components. Therefore, the advantages described in conjunction with the method aspects and examples can also be attributed to the corresponding system.

[0127] According to another aspect, the invention relates to a computer program product comprising program elements which, when loaded into a memory of a computing unit of a system configured to provide a medical video report, cause the computing unit to perform one or more steps according to the above method aspects and examples.

[0128] According to another aspect, the present invention relates to a computer-readable medium having program elements stored thereon, which can be read and executed by a computing unit of a system, and the computing unit is configured to provide a medical video report according to one or more method aspects and examples when the program elements are executed by the computing unit.

[0129] The implementation of the invention by a computer program product and / or a computer-readable medium has the advantage that already existing provisioning systems can be easily adapted by software update to work as proposed by the invention.

[0130] A computer program product may be, for example, a computer program or another element beside a computer program like this. The other element may be: hardware, such as a memory device on which the computer program is stored, a hardware key for using the computer program, etc.; and / or software, such as documentation or a software key for using the computer program. The computer program product may also include development materials, a runtime system and / or a database or library. The computer program product may be distributed among several computer instances. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] In view of the following description of the embodiments, the characteristics, features and advantages of the above invention and the manner in which they are achieved become clearer and easier to understand, and the embodiments will be described in detail with respect to the accompanying drawings. This following description does not limit the invention to the embodiments included. In different drawings, the same parts, parts or steps may be marked with the same reference numerals. Generally, the drawings are not drawn to scale. In the following:

[0132] Figure 1 schematically depicts a system for providing a medical video report according to an embodiment,

[0133] Figure 2 schematically depicts a method for providing a medical video report according to an embodiment,

[0134] Figure 3 schematically depicts a data flow diagram in a method for providing a medical video report according to an embodiment,

[0135] Figure 4 schematically depicts a template video script according to an embodiment,

[0136] Figure 5 schematically depicts optional method steps in a method for providing a medical video report according to an embodiment,

[0137] Figure 6 schematically depicts optional method steps in a method for providing a medical video report according to an embodiment,

[0138] Figure 7 schematically depicts a data flow diagram in a method for providing a medical video report according to an embodiment, and

[0139] Figure 8 Data processing functionality according to an embodiment is schematically depicted. DETAILED DESCRIPTION

[0140] Figure 1A system 1 for providing a medical video report MVR to a human receiver HR is depicted. In this regard, the system 1 is adapted to perform a method according to one or more embodiments, for example, as described with reference to Figures 2 to 8 Further described.

[0141] According to some examples, a user U of the system 1 (in the sense of an operator controlling the system 1) may typically relate to a healthcare professional such as a doctor, clinician, technician, radiologist, etc. According to some examples, a human recipient HR (in the sense of a person designated to receive a processing result of the system 1) may typically relate to a patient or (healthcare) person different from the user, such as a referring physician.

[0142] The system 1 includes a user interface 10 (as part of an interface unit), a receiver interface 11 (as part of an interface unit), and a processing system 20 (as part of a computing unit). In addition, the system 1 may include or be connected to a medical information system 40. The medical information system 40 may generally be configured to acquire and / or store and / or forward medical image research MIS and supplementary (non-image) information. For example, the medical information system 40 may include one or more archive / review stations (not shown) for medical image research MIS. The archive / review station may be embodied by one or more databases. In particular, the archive / review station may be implemented in the form of one or more cloud storage modules. Alternatively, the archive / review station may be implemented as local storage or extended storage, for example, as a PACS (Picture Archiving and Communication System). According to some examples, the medical information system 40 may also include one or more medical imaging modalities (not shown), such as a computed tomography system, a magnetic resonance system, an angiography (or C-arm X-ray) system, a positron emission tomography system, a mammography system, an X-ray system, etc.

[0143] The medical image study MIS may be a three-dimensional image data set acquired, for example, using an X-ray system, a computed tomography system, or a magnetic resonance imaging system or other system. The image information may be encoded in a three-dimensional array of m times n times p voxels. The medical image study MIS may include a plurality of image slices stacked in a stacking direction to span the image volume covered by the medical image study MIS.

[0144] Furthermore, the medical image study MIS may include two-dimensional medical image data, wherein image information is encoded in an array of m by n pixels. According to some examples, these two-dimensional medical images may have been extracted from a three-dimensional medical image study MIS.

[0145] In the following, a collection of voxels or pixels may be designated as image data of a corresponding medical image study. Typically, any type of imaging modality and scanner may be used to acquire such image data. Typically, a medical image study MIS may show a body part or anatomical region or anatomical object of a patient, which may include various anatomical structures and organs. Considering the chest region as a body part, a medical image study MIS may, for example, depict a lung lobe, a thoracic cavity, a heart, lymph nodes, etc.

[0146] Medical image research MIS can be formatted according to the DICOM format. DICOM (=Digital Imaging and Communications in Medicine) is an open standard for the communication and management of medical imaging information and related data in healthcare informatics. DICOM can be used to store and send medical images and associated information, enabling the integration of medical imaging devices such as scanners, servers, workstations, printers, network hardware, and picture archiving and communication systems (PACS). DICOM is widely adopted by clinical syndicates, hospitals, and for smaller applications such as doctors' offices or groups. DICOM data objects consist of multiple attributes, including items such as patient name, ID, etc., and also consist of special attributes containing image pixel data and metadata extracted from the image data.

[0147] Supplementary information (or associated data) may be any data that provides additional information related to the patient and / or the medical image study MIS. Supplementary information may include image data, such as other medical image studies MIS of the patient, for example acquired at an earlier point in time than the medical image study MIS under consideration. Furthermore, the supplementary information may include non-image data or data with mixed type content including medical images and non-image content such as text. Non-image data may relate to non-image examination results, such as laboratory data, vital signs records (including, for example, ECG data, blood pressure values, ventilation parameters, oxygen saturation levels), etc. Furthermore, the supplementary information may include structured and unstructured medical text reports MTR related to a previous examination or a current examination of the patient. Furthermore, non-image data may include personal information of the patient, such as gender, age, weight, insurance details, etc.

[0148] The supplemental information may be available in the form of one or more electronic medical reports of the patient. The supplemental information may be stored in the healthcare information system 40. For example, the supplemental information may be stored in a dedicated database of the healthcare information system 40, such as a laboratory information system (LIS) or an electronic health / medical record database.

[0149] The user interface 10 may include a display unit and an input unit. The user interface 10 may be embodied by a mobile device such as a smartphone or a tablet computer. In addition, the user interface 10 may be embodied as a workstation in the form of a desktop PC or a laptop computer. The input unit may be integrated in the display unit, for example, in the form of a touch screen. As an alternative or in addition, the input unit may include a keyboard, a mouse, or a digital pen and any combination thereof. The display unit may be configured to display a representation of a medical image study MIS, to display and edit a medical text report MTR, to play a medical video report MVR, to display a thumbnail of a medical video report MVR, and to receive any user input, for example, for approving a medical video report MVR or a portion thereof.

[0150] The user interface 10 may also include an interface computing unit configured to execute at least one software component for serving the display unit and the input unit to provide a graphical user interface for allowing the user to select the case of the target patient to be reviewed and to perform various inputs. In addition, the interface computing unit may be configured to communicate with the medical information system 40 or the processing system 20 for receiving the medical image research MIS and any supplementary information. The user U may activate the software component via the user interface 10, and may obtain the software component, for example, by downloading the software component from an Internet application store. According to an example, the software component may also be a client-server computer program in the form of a web application running in a web browser. The interface computing unit may be a general-purpose processor, a central processing unit, a control processor, a graphics processing unit, a digital signal processor, a three-dimensional rendering processor, an image processor, an application-specific integrated circuit, a field programmable gate array, a digital circuit, an analog circuit, a combination thereof, or other now known devices for processing image data. The user interface 10 may also be embodied as a client.

[0151] The receiver interface 11 may include a display unit and an input unit. The receiver interface 11 may be implemented by a mobile device such as a smartphone or a tablet computer. In addition, the receiver interface 11 may be embodied as a workstation in the form of a desktop PC or a laptop computer. The input unit may be integrated in the display unit, for example, in the form of a touch screen. As an alternative or in addition, the input unit may include a keyboard, a mouse or a digital pen and any combination thereof. The display unit may be configured to play a medical video report MVR.

[0152] The receiver interface 11 may also include an interface computing unit configured to execute at least one software component for servicing the display unit and the input unit to provide a graphical user interface for allowing the user to receive and play the medical video report MVR. The user can activate the software component via the receiver interface 11, and can obtain the software component, for example, by downloading the software component from an Internet application store. According to an example, the software component may also be a client-server computer program in the form of a web application running in a web browser. The interface computing unit may be a general-purpose processor, a central processing unit, a control processor, a graphics processing unit, a digital signal processor, a three-dimensional rendering processor, an image processor, an application-specific integrated circuit, a field programmable gate array, a digital circuit, an analog circuit, a combination thereof, or other now known devices for processing image data. The receiver interface 11 may also be embodied as a client.

[0153] The processing system 20 may include subunits 21 - 24 configured to process the medical image study MIS and the supplementary information to provide a medical video report MVR.

[0154] The processing system 20 may be a processor. The processor may be a general purpose processor, a central processing unit, a control processor, a graphics processing unit, a digital signal processor, a three-dimensional rendering processor, an image processor, an application specific integrated circuit, a field programmable gate array, a digital circuit, an analog circuit, a combination thereof, or other currently known devices for processing image data. The processor may be a single device or multiple devices operating serially, in parallel, or individually. The processor may be the main processor of a computer such as a laptop or desktop computer, or may be a processor for processing some tasks in a larger system such as a medical information system or a server. The processor is configured by instructions, design, hardware, and / or software to perform the steps discussed herein. The processing system 20 may be included in the user interface 10. Alternatively, the processing system 20 may include a real or virtual computer group, such as a so-called "cluster" or "cloud". Such a server system may be a central server such as a cloud server, or a local server located at a hospital or radiation site, for example. In addition, the processing system 20 may include a memory such as RAM for temporarily loading a medical image research MIS. According to some examples, such a memory may also be included in the user interface 10.

[0155] The subunit 21 is configured to extract relevant context information CI from the medical image study MIS and any supplementary information. The context information CI may include the "reason for the medical image study" and additional relevant information related to the case, such as the identified medical findings and / or information related to the human recipient HR. According to some examples, in order to extract the context information CI, the subunit 21 may be configured to run a correspondingly configured function CI_FU.

[0156] The subunit 22 is configured to select an appropriate template video script TVS from a plurality of candidate template video scripts CVS based on the context information CI. Thus, the template video script TVS encodes the following instructions: the instructions are used to render a medical video report MVR adapted to the corresponding use case (as characterized by the context information CI). In addition, the subunit 22 can be configured to activate and deactivate blocks within the selected template video script TVS based on available information.

[0157] In other words, the subunit 22 may be configured to select the correct template video script TVS and optionally edit the correct template video script TVS based on any supplementary information and medical findings included in the medical image study MIS. Another expression of the subunit 22 may be findings to video engine.

[0158] In order to match the context information CI with an appropriate template video script TVS and to select individual blocks within the template video script TVS, the subunit 22 may be configured to run a correspondingly configured selection function TVS_FU.

[0159] The subunit 23 may be conceived as a rendering module or unit. The subunit 23 may be configured to process instructions included in the template video script TVS to render a medical video report MVR based on the medical image study MIS and any supplementary information. Another expression of the subunit 23 may be a rendering engine. In order to process the medical image study MIS and the supplementary information according to the selected (and optionally edited) template video script TVS, the subunit 23 may be configured to run a correspondingly configured rendering function REN_FU. According to some examples, the rendering function REN_FU may implement a volume rendering algorithm that may be based in particular on path tracing or ray casting.

[0160] The subunit 24 may be conceived as a voice-over module or unit. The subunit 24 is configured to generate an audio track of a medical video report MVR, which includes speech output and additional instructions for a human receiver HR. In other words, the subunit 24 may be configured to extract information from the medical image study MIS, the context information CI and / or any supplementary information and convey this information into speech suitable for accompanying the video frames.

[0161] To this end, according to some examples, the subunit 24 may include two submodules 24_TXT and 24_CHAT. While the submodule 24_TXT may be envisioned as being configured to extract medically relevant information from the available data and transmit it to a discovery-to-text engine in a (machine-readable) text TXT, the submodule 24_CHAT may be viewed as being configured to transmit the text to a chat engine of an appropriate voice output VO (suitable for the needs of the medical video report MVR and human recipients).

[0162] According to some examples, submodule 24_TXT can run a text generation function TXT_FU, which is configured to transform medical findings and other related information into structured or unstructured text TXT. According to some examples, submodule 24_CHAT can run a chat function CHAT_FU, which is configured to transcribe the text into a version suitable as the basis for voice output / voice-over VO.

[0163] The designation of the different subunits 21 to 24 will be interpreted by way of example and not as a limitation. Thus, the subunits 21 to 24 may be integrated to form a single unit (e.g., in the form of a "computing unit") or may be embodied by computer code segments configured to execute corresponding method steps running on a processor or the like of the processing system 20. The same is true for the interface computing unit. Each subunit 21 to 24 and the interface computing unit may be individually connected to other subunits and / or other components of the system 1 that require data exchange to perform method steps.

[0164] The processing system 20 and the interface computing unit together can constitute the computing unit of the system 1. It is worth noting that the layout of this computing unit, that is, the physical distribution of the interface computing unit and the subunits 21 to 24, is arbitrary in principle. Specifically, the processing system 20 can also be integrated in the user interface 10. As already mentioned, alternatively, the processing system 20 can be embodied as a server system such as a cloud server or a local server located, for example, at a hospital or a radiation site. According to such an implementation, the user interface 10 can be designated as a user-oriented "front end" or "client", while the processing system 20 can be conceived as a "back end" or server. For example, the https-protocol can be used to communicate between the user interface 10 and the processing system 20. The computing power of the system can be distributed between the server and the client (i.e., the user interface 10). In a "thin client" system, most of the computing power exists at the server. In a "fat client" system, more computing power and possible data exist on the client.

[0165] Furthermore, the system 1 may include a template database R_CVS serving as a repository for a plurality of candidate template video scripts CVS. The template database R_CVS is a storage device such as a cloud or local storage device serving as an archive of template video scripts TVS.

[0166] Furthermore, the system 1 may include a standard video database R_SEQ, which serves as a repository for easily rendered standard video sequences SEQ, which can be attached to newly rendered video materials based on medical image research MIS. The standard video database R_SEQ is a storage device such as a cloud or local storage device used as an archive of standard video sequences SEQ.

[0167] The various components of system 1 can be connected to each other at least temporarily for data transmission and / or exchange. The user interface 10 communicates with the processing system 20 via a (data) interface 26 to exchange, for example, a medical image study MIS, a medical text report MTR, a final medical video report MVR or any user input. For example, the processing system 20 can be activated based on a request, wherein the request is sent by the user interface 10. In addition, the receiver interface 11 communicates with the processing system 20 via a (data) interface 26 to receive the medical video report MVR. In addition, the processing system 20 can communicate with the medical information system 40 to retrieve the case of the target patient. As an alternative or in addition, the user interface 10 can communicate directly with the medical information system 40. The medical information system 40 can also be activated based on a request, wherein the request is sent by the processing system 20 and / or the user interface 10. The data interface 26 for data exchange can be implemented as a hardware or software interface, such as a PCI bus, USB or FireWire. Data transmission can be implemented using a network connection. The network can be implemented as a local area network (LAN), for example, an intranet or a wide area network (WAN). The network connection is preferably wireless, for example as a wireless LAN (WLAN or Wi-Fi). In addition, the network can include a combination of different network examples. The interface 26 for data exchange is considered to constitute an interface unit of the system 1 together with the components for interfacing with the user.

[0168] Figure 2 A method for providing a medical video report MVR according to an embodiment is described. The corresponding data stream is Figure 3 The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the invention. In addition, individual steps or a series of steps may be repeated.

[0169] In a first step S10, a medical image study MIS is received. This may involve selecting a medical image study MIS from a plurality of cases stored, for example, in the medical information system 40. The selection may be performed manually by the user U, for example by selecting appropriate image data in a graphical user interface running in the user interface 10. Alternatively, the medical image study MIS may be provided to the computing unit 20 by the user U by uploading the medical image study MIS to the computing unit 20.

[0170] Optionally, supplementary information (or associated data) may be obtained for the medical image study MIS. The supplementary information may include a medical text report MTR, demographic information of the patient, medical history of the patient, diagnostic tasks to be performed based on the medical image study MIS, etc. Obtaining the supplementary information may include, for example, querying the medical information system 40 for the supplementary information based on the patient ID of the patient.

[0171] At step S15, a medical text report MTR is obtained. The medical text report MTR may be related to the medical image study MIS in the sense that the medical text report MTR describes findings that can be derived from the medical image study MIS. The medical text report MTR may be an already finalized report that has been previously signed by the user U or any other competent medical personnel. Such a medical text report MTR may be stored in the medical information system 40 and may be automatically obtained by querying the medical information system 40.

[0172] According to some examples, the medical text report MTR may be a report that is currently in the process of being created by the user U. In other words, the medical text report MTR may also relate to a non-final report that the system 1 may analyze "on the fly" when it is created by the user U. Such a medical text report MTR may be created by the user U using the user interface 10, via which the user U may edit the medical text report MTR, for example by typing text or by dictating text into a speech-to-text module (not shown) of the user interface 10.

[0173] Step S15 is optional in the sense that the subsequent steps may also be performed if no medical text report MTR is available. In such a case, the subsequent processing may be based on other available supplementary information. Furthermore, according to some examples, the medical text report MTR may also be included in the supplementary information.

[0174] At step S20, context information CI is derived from available information and data, which may include medical image studies MIS, medical text reports MTR and / or any other supplementary information. Specifically, a correspondingly configured function CI_FU may be applied to the available data, which is configured to derive meaningful context information CI therefrom.

[0175] According to some examples, determining the context information CI may include classifying the current case according to a plurality of predefined use cases based on the available information. Then, the use case into which the current case is classified may be the context information CI. According to some examples, the function CI_FU may be configured to perform the above classification. The use case may involve the type of examination performed or the type of medical imaging study (e.g., CT or MR study, with or without contrast agent, etc.), the patient's suspected disease, the reason for the examination, the identified medical findings, a template for a medical text report MTR, etc.

[0176] At optional sub-step S21, a medical finding of a given finding type is obtained. This may include automatically identifying such a medical finding in available information and data. For example, the medical finding may be identified in a medical text report MTR, for example based on an analysis of the text included in the medical text report MTR. In addition, a finding detection function may be applied to a medical image research MIS, which detection function is configured to identify medical findings in medical image data. In addition, sub-step S21 may include identifying medical findings based on an interaction of a user U with the system 1. Such user interaction may include adding a mark in a representation of the medical image research MIS shown in the user interface 10. In addition, the user interaction may also involve using a measurement tool to identify / measure the medical finding. According to another example, the user input may also include activating an automatic detection function configured to identify medical findings in the medical image research MIS.

[0177] At step S30, a template video script TVS matching the context information CI is retrieved from the template database R_CVS. To this end, a search operation may be performed on the candidate template video script CVS corresponding to the context information CI in the template database R_CVS.

[0178] According to some examples, the template database R_CVS may include candidate template video scripts CVS for each predefined use case. Then, the association linking the candidate template video scripts CVS in the template database R_CVS with the applicable use case may be used to identify the correct template video script TVS in the candidate template video scripts CVS based on the use case of the current medical image research MIS.

[0179] According to some examples, a suitably configured selection function TVS_FU may be used to match the context information CI with a suitable template video script TVS.

[0180] At step S40 , a medical video report MVR is rendered by executing the template video script TVS.

[0181] The actual rendering process in the sense of transferring the image data comprised in the medical image study MIS into a visible image can be performed here by means of a rendering function REN_FU. The rendering function REN_FU can implement, individually or in combination, various methods for visualizing a medical image study MIS, which can be a three-dimensional image data record. By way of example, the rendering function REN_FU can include a ray casting module and / or a path tracing module.

[0182] At step S50, a medical video report MVR is provided. In particular, this may include making the medical video report MVR available to a human receiver HR. This may mean forwarding the medical video report MVR to a receiver interface 11, or saving the medical video report MVR available for downloading via the receiver interface 11. Furthermore, step S50 may include providing the medical video report MVR to a user U for review and approval, for example in a user interface 10.

[0183] exist Figure 4 , a template video script TVS (or candidate template video script CVS) according to an embodiment is shown. Thus, the template video script TVS, CVS can be regarded as a recipe for creating a medical video report MVR for a specific use case. In particular, the template video script TVS, CVS may include instructions for rendering a specific video visualization and / or adding an already existing video clip or a standard video sequence SEQ.

[0184] If you can Figure 4 As seen in , the template video script TVS, CVS may include different blocks or video template structures TVS, each of which includes instructions for providing a portion of the final medical video report MVR. According to some examples, the video template structures VTS may be specific to specific types of medical findings, respectively. Each video template structure VTS may include instructions for optimally highlighting specific medical findings to the human receiver HR. These instructions may include instructions for appropriately segmenting the medical image study MIS, applying suitable textures, using the correct perspective, scene lighting, including markings, camera positions, animations, zoom levels, etc. Another separate video template structure VTS may include instructions for adding a standard video sequence SEQ that can provide more general information to the human receiver HR and is independent of the image data. Such a standard video sequence SEQ may involve instructions about a disease (i.e., a use case), an examination performed to obtain a medical image study MIS, or a recommendation for subsequent treatment.

[0185] Thus, in other words, the template video script TVS can script a sequence of standard videos and / or API calls to a rendering engine. According to a non-limiting example, the respective video template structures VTS of an exemplary template video script TVS can be seen in a pseudo-code representation as follows:

[0186]

[0187] According to some examples, each video template structure VTS can be activated according to the case situation to generate a medical video report MVR. The corresponding optional method steps are Figure 5 It can be performed in step S40 Figure 5 The individual steps or all steps shown in the embodiment of the present invention may not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the present invention. In addition, individual steps or a series of steps may be repeated.

[0188] At step S41, each video template structure VTS of the selected template video script TVS may be selectively activated or deactivated. While the activated video template structure VTS is used to render the medical video report MVR, the deactivated or inactivated video template structure VTS is not used to render the medical video report MVR.

[0189] The activation of the respective video template structure VTS may again depend on the case context, ie the context information CI, supplementary information, medical text report MTR, medical findings, and in particular the respective attributes of the medical findings such as the size of the medical findings, as also shown in the above example.

[0190] At step S42, the activated video template structure VTS is executed, and a medical video report MVR is generated on this basis.

[0191] Furthermore, at step S43, one or more standard video sequences SEQ may be selected, in particular by activating the corresponding video template structure VTS. Again, the selection of the standard video sequence SEQ may be based on the case situation, i.e., context information CI, supplementary information, medical text report MTR, medical findings, and in particular individual attributes of the medical image study MIS, such as the image modality used.

[0192] At step S44, the selected standard video sequence SEQ is retrieved from the standard video database R_SEQ, and included in the medical video report MVR at an appropriate position (which can be determined by the position of the corresponding video template structure VTS in the template video script TVS).

[0193] Optionally, the medical video report MVR may also include an audio track with a voice-over VO that provides additional explanation about the content of the medical video report MVR. Figure 6 A method for providing a voice-over VO according to an embodiment is described. The corresponding data stream is Figure 7 The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the invention. In addition, individual steps or a series of steps may be repeated.

[0194] At step S51, a natural language text TXT is extracted / generated from the available information. The natural language TXT may summarize relevant aspects of the case in a text format, but may not be suitable for direct transmission into a voice-over VO. For example, the text TXT may include repetitive, inappropriate or complex language, or may not be of sufficient length for a medical video report MVR. However, the natural language text TXT may form the basis of a voice-over VO.

[0195] A natural language text TXT can be extracted using a text generation function TXT_FU (another expression can be a mapping function), which is configured to transform (or map) medical findings and other related information into structured or unstructured text TXT. To this end, the text generation function TXT_FU can be configured to process medical image studies MIS, medical text reports MTR, context information CI, detected medical findings, activated video template structures VTS and / or any other supplementary information.

[0196] At step S52, the text TXT generated in step S51 is brought into a version suitable for the voice-over VO.

[0197] To this end, a chat function CHAT_FU may be provided, which is configured to transcribe the text TXT into a version to be used for the voice-over VO. This may include shortening the text TXT, exchanging expressions in the text TXT, switching the language of the text TXT, etc. According to some examples, the chat function CHAT_FU may be configured to simplify the text TXT and, in particular, replace technical terms with layman's explanations that are more easily understood by the human recipient HR.

[0198] In this regard, the chat function CHAT_FU may be configured to take into account specific requirements or constraints of the human recipient HR, such as language, expertise level, and the like.

[0199] Furthermore, the chat function CHAT_FU may be configured to take into account the medical video report MVR and, in particular, the duration of the corresponding part in the activated video template structure VTS and the medical video report MVR.

[0200] At step S53, a voice-over VO is generated. This may include generating a corresponding sound track and including the sound track in the medical video report MVR.

[0201] Several functions that can be used to finally generate a medical video report MVR rely at least in part on the processing of a natural language text TXT. In particular, this is also true for a context function CI_FU configured to determine context information CI based on available information, a function TVS_FU configured to select an appropriate template video script TVS, a text generation function TXT_FU, and a chat function CHAT_FU. According to some examples, one or more of these functions can be implemented as a machine learning model configured as a transformer network.

[0202] exist Figure 8 , a schematic representation of a transformer network according to an embodiment is shown. The transformer architecture follows an encoder-decoder structure and comprises an encoder ENC and a decoder DEC. In short, the task of the encoder ENC is to map the input INPT to a series of continuous representations, which are then fed into the decoder DEC. The decoder DEC receives the output of the encoder ENC and the decoder output OUTR at the previous iteration to generate an output OUT.

[0203] The encoder ENC of this embodiment may include a stack of N=6 identical layers. For ease of reference, only one layer xN is shown in the figure. In addition, N can also be set to different values ​​according to the corresponding task, and in particular to a value greater than N=6. Each layer xN of the encoder ENC includes two sublayers L1 and L3. The first sublayer L1 implements the so-called multi-head self-attention mechanism. Specifically, the first sublayer L1 can be configured to determine the relevance of a particular word relative to other words in the input INPT. This can be represented as an attention vector. In order to avoid any bias, multiple attention vectors for each word can be generated and fed into a weighted average to calculate the final attention vector for each word. The second sublayer L3 is a fully connected feedforward network, which can, for example, include two linear transformations in which a rectified linear unit (Rectified Linear Unit, ReLU) is activated between them. The N=6 layers of the encoder ENC apply the same linear transformation to all words in the input INPT, but each layer uses different weights and bias parameters to do so. Each sublayer L1, L3 is followed by a normalization layer L2, which normalizes the sum calculated between the input fed to the corresponding sublayer L1 and L3 and the output generated by the corresponding sublayer L1, L3 itself. In order to capture information about the relative position of the words in the input INPT, a position encoding PE is generated based on the input embedding INPT-E before being fed to the layer xN. The position encoding PE has the same dimension as the input embedding INPT-E and can be generated using sine and cosine functions of different frequencies. The position encoding PE can then be simply added to the input embedding INPT-E to inject the position information PE. The input embedding INPT-E can be, as usual, a representation of each word in the input INPT, typically in the form of a real-valued vector that encodes the meaning of the word so that closer words in the expected vector space have similar meanings. According to some examples, a neural network can be used to generate the input embedding INPT-E.

[0204] The decoder DEC of this embodiment may also include a stack of N=6 identical layers xN, each layer xN including three sublayers L4, L1, L3, and the three sublayers L4, L1, L3 are followed by a normalization layer L2, as described in conjunction with the encoder ENC. For ease of reference, only one layer xN of the decoder DEC is shown in the figure. In addition, N can also be set differently according to the corresponding task, and in particular, it is set to be greater than N=6. When the sublayers L1 and L3 of the decoder DEC correspond to the corresponding sublayers L1 and L3 of the encoder ENC in their functions, the sublayer L4 receives the previous output OUTR of the decoder DEC (optionally transformed into a corresponding embedding, and if the output is a series of words, it is enhanced with position information), and implements multi-head self-attention on it to weigh the importance of each element of the previous output vector OUTR. Next, the value of the first sublayer L4 from the decoder DEC is input into the L1 sublayer of the decoder DEC. This sublayer L1 of the decoder DEC implements a multi-head self-attention mechanism similar to the multi-head self-attention mechanism implemented in the first sublayer L1 of the encoder ENC. On the decoder side, the multi-head mechanism receives the value from the previous decoder sublayer L4 and the output of the encoder ENC. This allows the decoder to process all words in parallel. As in the encoder ENC part, the output of the L1 sublayer is passed to the feedforward layer L2, which will form the output vector into something that can be easily accepted by another decoder block or linear layer. After all layers xN of the decoder DEC have been processed, the intermediate results are fed to the linear layer L5, which can be another feedforward layer. Linear layer 5 is used to expand the dimension into the expected format for calculating the output vector OUT. Next, the result is passed through the Softmax layer L6, which transforms the result into an output, which can be context information CI, an appropriate template video script TVS, a natural language text TXT, or a text version for voice-over VO, depending on the situation.

[0205] The various embodiments or their various aspects and features may be combined or interchanged with one another wherever it makes sense without limiting or expanding the scope of the invention. Where applicable, advantages described with respect to one embodiment of the invention are also beneficial for other embodiments of the invention. Independent of grammatical term usage, individuals with male or female identities are also included in the terms.

Claims

1. A computer-implemented method for providing a medical video report (MVR), the method comprising: - receiving (S10) a medical image study (MIS) of the patient; - determining (S20) context information (CI) related to the medical image study (MIS) based on the medical image study (MIS); - selecting (S30) a template video script (TVS) from a plurality of candidate template video scripts (CVS) based on the context information, each candidate template video script (CVS) encoding instructions for an image processing step of a medical image study (MIS) to generate a video related to the medical image study (MIS) for a human receiver (HR); - executing (S40) the selected template video script on the medical image study (MIS) to generate the medical video report (MVR); - Providing (S50) said Medical Video Report (MVR).

2. The method according to claim 1, further comprising: - obtaining (S15) a medical text report (MTR) associated with said medical image study (MIS), - wherein, in the step of determining (S20), the context information (CI) is extracted from the medical text report (MTR).

3. A method according to any one of the preceding claims, wherein: - said context information (CI) comprises an indication of a medical finding of a predefined finding type, - The selected Template Video Script (TVS) encodes instructions for an image processing step of a Medical Image Study (MIS) to generate a video for a human receiver (HR) visualizing a medical finding of the predefined finding type.

4. A method according to any one of the preceding claims, wherein: - the candidate template video script (CVS) respectively comprises a plurality of optionally activatable video template structures (VTS), each activatable video template structure (VTS) encoding the following instructions: the instructions are used to provide a part of a medical video report (MVR) based on a medical image study (MIS), - The step of executing (S40) comprises: - activating (S41) at least one of said video template structures (VTS) of the selected template video script (TVS) based on said context information (CI); and - Generating (S42) said Medical Video Report (MVR) based on said Video Template Structure (VTS) performing at least one activation.

5. The method according to claim 4, wherein: - said video template structure (VTS) of the selected template video script (TVS) comprises finding-specific video template structures (VTS) respectively having instructions for visualizing medical findings of said predefined finding types; - obtaining an indication of a medical finding of a given finding type based on said medical image study (MIS) and / or said contextual information (CI) and / or supplementary information associated with said medical image study (MIS); as well as - The step of activating (S41) comprises activating at least one discovery-specific video template structure (VTS) of the selected template video script (TVS) based on the given discovery type, the predefined discovery type of the at least one discovery-specific video template structure (VTS) matching the given discovery type.

6. The method according to claim 5, wherein: The step of obtaining the at least one medical finding (S21) comprises: - querying whether said medical image study (MIS) indicates one or more medical findings of said predefined finding types of said video template structure (VTS) of said selected template video script (TVS).

7. A method according to any one of the preceding claims, wherein: - the selected template video script (TVS) comprises at least a video template structure (VTS) associated with a pre-existing video sequence (SEQ), - The step of executing (S40) comprises: - retrieving (S43) said pre-existing video sequence from a repository (R_SV) configured to store a plurality of pre-existing video sequences (SEQ), - Including (S44) said pre-existing video sequence (SEQ) in said medical video report (MVR).

8. The method according to any one of the preceding claims, further comprising: - obtaining (S51) a natural language text (TXT) from said context information (CI) and / or said medical image study (MIS); - generating (S52) a voice-over (VO) based on the natural language text (TXT); and - Including (S53) the voice-over (VO) in the medical video report (MVR).

9. The method according to claim 8 in combination with claim 2, wherein: In the step (S51) of obtaining the natural language text (TXT), the natural language text (TXT) is extracted from the medical text report (MTR).

10. The method according to any one of claims 8 or 9 in combination with claim 3, wherein: - The step (S51) of obtaining a natural language text (TXT) comprises: - providing a mapping function configured to map medical findings onto structured text, - generating the natural language text (TXT) by applying the mapping function to the medical findings.

11. The method according to any one of claims 8 to 10, wherein: - The step (S52) of generating the voice-over (VO) comprises: - providing a chat function (CHAT_FU) configured to transcribe a natural language text (TXT) from a first version to a second version; - applying said chat function (CHAT_FU) to said natural language text (TXT), to transcribe the natural language text into the second version; and - generating the voice-over (VO) based on the natural language text (TXT) of the second version.

12. A method according to any one of the preceding claims, wherein: - the medical image study (MIS) comprises a three-dimensional medical image dataset, the selected template video script (TVS) encodes instructions for volumetric image rendering, in particular implementing a rendering process based on path tracing or ray casting, and - The medical video report (MVR) comprises one or more images generated via volumetric image rendering.

13. A system for providing a medical video report (MVR), comprising an interface unit and a computing unit, wherein the interface unit is configured to receive (S10) a medical image study (MIS), and the computing unit is configured to: - determining (S20) context information (CI) related to the medical image study (MIS) based on the medical image study (MIS); - selecting (S30) a template video script (TVS) from a plurality of candidate template video scripts (CVS) based on the context information, each candidate template video script (CVS) encoding instructions for image processing steps of a medical image study (MIS) to generate a video related to the medical image study (MIS) for a human receiver (HR); - executing (S40) the selected template video script (TVS) on the medical image study (MIS) to generate the medical video report (MVR); as well as - providing (S50) said medical video report (MVR) via said interface unit.

14. A computer program product comprising program elements which, when loaded into a memory of a computing unit (20) of a system (1), cause the computing unit (20) to execute the steps of the method according to any one of claims 1 to 12.

15. A computer-readable medium storing program elements, which can be read and executed by the computing unit (20) to perform the steps of the method according to any one of claims 1 to 12 when the program elements are executed by the computing unit (20) of the system (1).

Images