Computer-implemented method for operating a medical apparatus, medical apparatus, computer program and electronically readable data carrier

Abstract

The invention relates to a computer-implemented method for operating a medical apparatus (33), in particular a medical imaging apparatus (34), comprising the following steps: - receiving user input information which comprises a natural query input from a user in text form, - applying at least one trained voice analysis function, which comprises a first transformer, to the user input information to obtain output data, wherein the output data comprise at least one control measure, in particular at least one control command and / or programming command, and / or the at least one control measure is determined from the output data, and - carrying out the at least one control measure, allocated to the selected second text embedding, to control the medical apparatus (33).

Description

[0001] Description

[0002] Computer-implemented method for operating a medical device, medical device, computer program and electronically readable data carrier

[0003] The invention relates to a computer-implemented method for operating a medical device, in particular a medical imaging device. The invention also relates to a medical device, in particular a medical imaging device, a computer program, and an electronically readable data carrier.

[0004] With increasing levels of development and the growing variety of available functions, medical technology equipment for examining and / or treating patients is becoming increasingly complex to operate, particularly for users with medical, for example clinical, training who have not received special technical training at the respective medical technology equipment. This also applies in particular to medical imaging equipment, for example magnetic resonance imaging equipment, computed tomography equipment and C-arm X-ray equipment, whose range of functions and performance is constantly growing. A particular problem here is that, due to the technical nature of highly complex medical technology equipment, it is not easy for users to link a specific clinical task or use case with the capabilities of the corresponding system.Features of medical technology devices can be overlooked because their usability in a specific situation is not always immediately apparent to the user.

[0005] In this regard, operating steps, particularly workflows with corresponding workflow steps, are typically described in detail in user manuals, and seminars, workshops, and the like are offered to train users of medical technology equipment, such as physicians and / or medical technicians, in the operation of specific medical technology equipment. However, the problem arises that workflows and operating procedures that have only been learned once through user manuals and / or events / training can be forgotten. For example, users often remember primarily those workflows that they themselves use regularly.In addition, particularly when, for example, minimally invasive interventions are carried out on a patient with or on the medical device, there is usually no time to look up or read information on how to use it.

[0006] The invention is therefore based on the object of enabling a simplified, in particular at least partially automated, usability of the scope of services of medical technology devices, in particular with regard to operability and the best possible implementation of a user request.

[0007] This problem is solved by a computer-implemented method, a medical device, a computer program, and an electronically readable data carrier according to the independent patent claims. Advantageous further developments arise from the dependent patent claims.

[0008] A method according to the invention for operating a medical device, in particular a medical imaging device, comprises the following steps:

[0009] Receiving user input information comprising a natural query input of a user in text form, applying at least one trained speech analysis function, which comprises at least one first transformer, to the user input information to obtain output data, wherein the output data comprise at least one control measure, in particular at least one control command and / or programming command, and / or the at least one control measure is determined from the output data, and

[0010] Providing at least one control measure for controlling the medical technology device.

[0011] In general, a trained function, including a trained speech analysis function, replicates cognitive functions that people associate with the activity of human brains. Through training based on training data (machine learning), a trained function is able to adapt to new circumstances and detect and extrapolate patterns.

[0012] Generally speaking, parameters of a trained function can be adjusted through training. In particular, supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, and / or active learning can be used. Furthermore, representation learning (also known as feature learning) can also be employed. The parameters of the trained function can be adjusted iteratively through multiple training steps.

[0013] For example, a trained function may comprise a neural network, a support vector machine (SVM), a decision tree, and / or a Bayesian network, and / or the trained function may be based on k-means clustering, Q-learning, genetic algorithms, and / or mapping rules. In particular, a neural network may be a deep neural network, a convolutional neural network (CNN), or a deep CNN. Furthermore, the neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network (GAN).

[0014] According to the invention, it is therefore proposed to use artificial intelligence in the form of the trained speech analysis function in order to determine the semantic content of a user's query from the user's free query input and to derive therefrom, directly or indirectly, control measures, in particular control commands and / or programming commands for the medical technology device, which can be used to support the user and / or particularly preferably to automate processes, in particular also with regard to the operation of the medical technology device.

[0015] This takes advantage of the fact that artificial intelligence has recently made significant progress, particularly in the field of language analysis, particularly with regard to the extraction of meaningful content from text sections (semantic coding, particularly as embeddings) and the generation of one's own, derived content (generative models / networks). So-called transformers form the basis of many language analyses. Transformers are architectures of deep machine learning that use so-called attention mechanisms and self-attention mechanisms. They can be used, for example in the form of the first transformer, within the scope of the present invention for the purposes of semantic coding / summarization, for example in embeddings, and / or as translators, for example into a standardized expression or another language, in particular also a control language / machine language.

[0016] Transformers also form the basis for modern language models, which can generate their own content, for example, as a chain of words. Large language models (LLMs) have recently gained particular popularity. Such LLMs can process very large amounts of text data and utilize K-accelerators. A well-known example is generative pre-trained transformers (GPTs).

[0017] Therefore, particularly advantageous embodiments of the present invention provide that the trained language analysis function implements a language model, in particular an LLM, or comprises at least one transformer selected from the group comprising BERT, RoBERTa, USE and SIE, in particular as the first transformer. It should be noted at this point that it is certainly conceivable to use an LLM, for example GPT-4, as a transformer, for example to determine embeddings which would then be extracted as hidden information. However, LLMs were fundamentally developed to generate texts, so that the use of transformers is preferred for determining semantic coding / summaries, in particular embeddings.For example, BERT (Bidirectional Encoder Representation from Transformers) or RoBERTa (Robustly Optimized BERT Pretraining Approach) can be used as transformer models. Both variants have demonstrated good performance in various areas of natural language processing (NLP). It is also conceivable to use USE (Universal Sentence Encoder) and / or SIE (Smooth Inverse Frequency) as transformer models. These two types of transformers are particularly suitable for generating embeddings from sentences.

[0018] It should be noted at this point that the input data for language models and sometimes also for transformers are often referred to as "prompts". Language models, especially LLMs, can be designed to optionally accept input data in a format other than text, for example images to which the input in text form refers.

[0019] The user's query can be entered acoustically, for example via a microphone, and / or directly in text form via an input device, such as a keyboard. In the former case, the acoustic recording of the query is converted into text form by a suitable processing device, such as speech recognition software, as is generally known in the prior art.

[0020] The control measures are preferably available directly in a control language that can be used to control the medical technology device, in particular in a control device of the medical technology device. The first transformer and / or at least one further transformer can therefore be designed to translate into a control language for the medical technology device. The control language can be a machine language. This means that the control measures can be carried out directly. In this sense, according to the invention, programming and / or coding, in particular a workflow comprising control commands and / or programming commands for setting an operating parameter, is generated and executed on the basis of the request input in order to at least partially implement the user's wishes described by the request input and / or to at least partially support the user in carrying them out.

[0021] Various specific ways of operating the medical technology device in response to the user's voice input are conceivable, and these ways can also be combined. It can be provided with particular advantage that at least one control measure, in particular a programming command, comprises the setting of an operating parameter, in particular a recording parameter in an imaging device, and / or at least one control measure comprises the triggering of at least one functional process of the medical technology device, in particular a recording process in an imaging device, and / or at least one control measure comprises an evaluation of medical technology data recorded with the medical technology device, in particular image data recorded with the imaging device.Of course, in addition to or as an alternative to the examples of control measures mentioned here, in particular accompanying and / or independent of them, other control measures are also conceivable, for example the output of information and / or support instructions and / or the request for further input from the user.

[0022] The approach can be applied particularly advantageously to medical imaging devices, which are improved in their functionality and performance, but which also increases the complexity of their operation.

[0023] The imaging system can be a magnetic resonance imaging system and / or a computed tomography system and / or a C-arm X-ray system, particularly an angiography system. Especially in the area of ​​C-arm X-ray systems, the complexity is increasing both in terms of usability and functionality, as well as their relevance to clinical questions, particularly when three-dimensional imaging methods are provided (e.g., C-arm CT). Furthermore, C-arm X-ray systems are frequently used in the context of minimally invasive procedures, for example, to monitor the procedure, where a multitude of options exist in a time-critical context.Not only in this case, imaging devices often also offer capabilities for evaluating acquired image data, particularly for reconstruction, but also for performing further processing and evaluation procedures. Therefore, when an imaging device is used as a medical device, control measures can also relate to the evaluation of image data, which then constitutes the medical device data.

[0024] It can be useful to include knowledge about recorded image data in the operating procedure not only to understand the user's query input, but also to enable the concrete determination and / or parameterization of control measures. This can mean, on the one hand, that the trained speech analysis function, which can also be generally useful, can comprise further input data in addition to the user input information. Such input data can, for example, relate to historical data relating to a current examination and / or treatment process, for example of a current patient, with regard to the operation and use of the medical technology device, in particular the imaging device. Background information on a current examination and / or treatment process of a patient can also be used as additional input data, which will be discussed in more detail in concrete embodiments.Such background information can be retrieved, for example, from further user input and / or an information system, in particular a radiology information system (RIS) and / or a hospital information system (HIS) and / or a patient registration system. Another useful source of information is an electronic patient record. Such background information can provide clues to understanding, particularly regarding the context of the query input, and is particularly advantageous when provided in text form, particularly to supplement the textual query input.

[0025] In a particularly advantageous, first concrete development of the present invention, it can be provided that, as an information-determining and / or preparatory control measure to be carried out first, medical technology data obtained with the medical technology device, in particular image data recorded with the imaging device, are evaluated for processing and / or determining additional information for a subsequent control measure. The core of this first development is therefore to control the medical technology device, which here is preferably designed as an imaging device, in particular a C-arm X-ray device, on the basis of user requests that are defined in more detail by medical technology data, in particular image data. For this purpose, an evaluation of medical technology data, in particular image data, is combined with the trained speech analysis function in order to provide the suitable data for defining subsequent control measures.

[0026] This is always useful when the user's query input relates to a question based on the medical technology data, particularly an image-based task. The trained speech analysis function provides the information-gathering and / or preparatory control measure for evaluating the medical technology data as output data, particularly as control commands and / or programming commands, preferably in a control language of the medical technology device, and carries out the corresponding evaluation.Ultimately, a combination of an automated evaluation of the medical technology data and the trained language analysis function, in particular a language model, takes place in order to enable various automated applications which, in the following control commands, can relate in particular to the display / output of the medical technology data and / or measurements in the medical technology data, in particular image data, and / or the recording of further medical technology data.

[0027] Exemplary overall applications, which will be discussed in more detail below, include automatic collimation processes, automatic zoom processes, the cutting out of partial images, the adjustment of a display and the like. Furthermore, other overall applications include the automatic performance of measurements in the medical technology data, in particular image data, for example the determination of an FFR value of a stenosis, the size of an aneurysm for the selection of a stent, the finding of certain image features, for example supply vessels and the like. Examples of user query inputs that can lead to such evaluation processes with possible subsequent control measures include "zoom on<bestimmtes anatomisches Merkmal> " , "Measure the size of<be- stimmtes anatomisches Merkmal> " , "Give the<bestimmtes Score> out", "What is the largest blood vessel supplying an AVM" and the like.

[0028] An advantageous embodiment of the evaluation provides that the evaluation comprises a segmentation of the image data and / or the application of a trained evaluation function, in particular a trained segmentation function, to the medical technology data. In this way, for example, certain anatomical features or other features contained in the query input, such as implants and / or medical instruments, can be found in the image data and the following control measures can be adapted accordingly. When using trained evaluation functions, a classification of image features, for example as anatomical features or other features, is also possible.For the evaluation of image data, a large number of trained evaluation functions of special architecture already exist in the prior art, which can of course also be used accordingly within the scope of the present invention, also for a subsequent control measure.

[0029] It can be provided with particular advantage within the scope of the present invention that a plurality, in particular all, of the image features depicted in the image data are segmented by the trained segmentation function. It is also particularly expedient if, on the basis of the segmentation result of a trained segmentation function that segments all image features, in particular anatomical and / or other features, the segmented image features are labeled, in particular with text. This means that each segmented image feature is assigned a text that describes which anatomical and / or other feature it is. It is also conceivable to annotate unknown features as such.

[0030] As a trained segmentation algorithm that segments all image features, one could use, for example, the Segment Anything algorithm described in an article by Alexander Kirillov et al., “Segment Anything,” ar- Xiv: 2304. 02643vl, April 5, 2023. However, no annotation, i.e. no addition of labels, takes place in this environment. In principle, the trained segmentation function, which segments all anatomical features in particular, delivers as output sub-images or sub-regions that have been segmented as belonging together. An anatomical or other structure to be annotated, for example an organ, can comprise several such sub-regions, so that a label can be assigned to each of these sub-regions or to the sub-regions as a cluster. It is therefore proposed to apply a classification function, in particular a trained one, to the segmentation result of the trained segmentation function.This makes it possible not only to automatically segment captured images into all relevant sub-areas, especially in real time, but also to annotate them in a standardized way, i.e. to label them.

[0031] In order to train such a classification function, it can be provided

[0032] - to apply the trained segmentation function to a number of training image data sets in order to obtain respective segmentation results into sub-areas,

[0033] - to assign labels to the sub-areas, especially manually, which describe the respective image features displayed, especially in text form, and

[0034] - train the classification function based on the segmentation results as input data and the labels as output data.

[0035] In some embodiments, it is also possible to combine various sub-areas of the segmentation results into clusters manually and / or in a further, particularly unsupervised, learning process, whereby the labels can be assigned to the sub-areas of a cluster as a whole. This simplifies the process, particularly with manual annotation.

[0036] The classification function may, for example, comprise a neural network, in particular a CNN, and / or implement a transformer model to convert the segmentation result into a corresponding labeled segmentation result.

[0037] In the application phase, for example when determining a corresponding control measure that affects segmentation and annotation, the trained segmentation function, for example "Segment Anything", is applied to the image data. The output data of the trained segmentation function is a segmentation result in sub-areas. The trained classification function is applied to the segmentation result, which provides annotated, i.e. labeled, sub-areas (and / or clusters, if applicable) as output data.

[0038] As already mentioned, the sub-areas can be assigned anatomical features, such as "aneurysm", "liver", "clavista" and the like, as well as other features, such as "stent type A", "catheter type B" and the like.

[0039] The evaluation of image data, i.e. the combination of a trained segmentation function that delivers a segmentation result defining sub-areas, and a trained classification function that uses the segmentation result as input data to output annotated, i.e. labeled, sub-areas as output data, can also be used effectively on its own, i.e. independently of the application of the trained speech analysis function to user input information for determining control measures. Such automatic segmentation of recorded image data sets in real time, especially during a preferably minimally invasive procedure, into sub-areas and their standardized annotation allows for a variety of advantageous applications, including, for example,

[0040] - using the knowledge of which medical instruments are visible in the image dataset to adjust image acquisition parameters,

[0041] - tracking instruments for each type of instrument,

[0042] - automatic collimation and / or automatic zoom on relevant image parts, and

[0043] - Automatic identification of potential pathologies in a medical image dataset. The additional information and / or the processed medical technology data can be used to specify and / or modify the subsequent control measure, in particular as input data for an extended speech analysis.

[0044] In a particularly preferred embodiment, it can be provided that the additional information and / or the processed medical technology data together with the user input information and / or the output data of the trained speech analysis function and / or secondary data derived from this output data are used as input data for a trained translator function which determines or adapts at least one secondary control measure, and which at least one secondary control measure is executed. Supplementing the information basis with the additional information and / or the processing of the medical technology data, in particular a processed image data set, thus makes it possible to carry out a speech analysis again in a more targeted manner as a subsequent control measure, on the basis of which secondary control measures are determined, which then ultimately technically process the original user request in the user input information.This advantageously provides a two-stage procedure which, in the first stage, detects whether additional information or evaluable processing is missing and determines this additional information or carries out the processing, so that in the next, second stage, on the basis of a renewed analysis, the actions actually corresponding to the query input can be determined and carried out.

[0045] The trained translation function can expediently also include and use the first transformer of the trained speech analysis function, on the basis of which the (primary) control measures, in particular in the control language of the medical technology device, were determined for evaluation and further analysis. In particular, it can even be provided that the trained speech analysis function is used as the trained translation function. This means that one trained function can be sufficient for both analysis processes, whereby the output data can in particular already include the corresponding (primary) control measures or the secondary control measure. It should be noted that the trained speech analysis function in particular can of course also be designed when no further information orProcessing is required to directly process the user request of the query input in the control measures.

[0046] In exemplary embodiments, it may also be conceivable that the trained translation function uses a different, additional transformer, in particular for transformation directly into a control language of the medical device.

[0047] In a concrete example for the first concrete further development, it can be provided that a user views an image dataset, for example a live fluoroscopy image, and expresses the following as a query input: "I want to zoom in on the aneurysm and the parent artery." The trained speech analysis function determines that it is unknown where the aneurysm and the parent artery are located and that they cannot therefore be zoomed in on (or collimated). As a control measure, a segmentation and annotation to be carried out is therefore output and the image dataset is evaluated according to the control measure. In particular, both the aneurysm and the parent artery are found and annotated.In the following control action, the image data set, the labels, and the user input information are passed as input data to the trained translator function, in particular again the trained language analysis function or at least the same first transformer / LLM, which determines secondary control actions, in particular the primary control actions in a control language for the medical device. Executing the secondary control action zooms in on the image area containing the aneurysm and the parent artery. Optionally, the user can receive a preview beforehand and accept or reject the execution.In a particularly advantageous, second concrete development of the present invention, it can be provided that a first text embedding is determined from the user input information by means of the trained speech analysis function, wherein in order to determine the at least one control measure from the first text embedding, a similarity measure between the first text embedding and a plurality of second text embeddings provided in a database, each of which is assigned at least one database control measure for the medical technology device, is determined and a second text embedding with one of the highest similarity measures is selected, and the at least one database control measure assigned to the selected second text embedding is used as the at least one control measure.

[0048] In this second specific development, it is therefore proposed to summarize the meaning of the query input in a standardized and compact manner using the first transformer in a first text embedding. The control measures are then determined by comparing this semantic content with second text embeddings to which the control measures are assigned. Text embeddings are understood as numerical and / or vector-based and / or string-like representations of words or texts that describe the semantic meaning. They also form the basis of NLP in other contexts, for example of LLMs. Such text embeddings can be determined, for example, using transformers such as BERT, RoBERTa, USE and SIE, as described above.

[0049] The ultimately standardized coding of the semantic content of the query input, performed in this way, enables comparison with correspondingly determined second text embeddings and thus a robust assignment of the user's intention to at least one control measure. Different expressions and formulations of users can be converted into comparable or even identical first text embeddings using the trained language analysis function, in particular the first transformer, which enables a meaningful comparison.

[0050] To determine the second text embeddings, the trained speech analysis function can be applied with particular advantage to predefined sections, in particular headings, of a workflow documentation, in particular a user manual. This means that in a preparatory step, second text embeddings are determined by the trained speech analysis function from a description of workflows assigned to the medical technology device for achieving certain goals and solving certain problems. In this case, a user manual is used in particular, which usually contains corresponding instructions for the user, i.e. workflows, for specific tasks in order to achieve the desired goal. In particular, it can be provided that the predefined sections are headings, which usually describe the task to be processed.However, more extensive predefined sections are also conceivable, for example, if the task is described in more detail in the workflow documentation. The second text embeddings extracted in this way can then be assigned corresponding sentences from at least one control measure that process the task on the part of the medical device and / or at least partially support the user.

[0051] For example, in a simple embodiment, it is conceivable that, as a control measure assigned to at least part of the second text embeddings, an output device of the medical device is controlled to output the predefined section and / or a workflow description derived therefrom. For example, a corresponding page or a corresponding workflow description of the user manual can be output on a screen and / or display of the medical device.

[0052] In a particularly preferred embodiment, however, it can be provided that from the text sections assigned to and / or comprised by the predefined sections, which describe at least one workflow step, at least one control measure assigned to the corresponding second text embedding for implementing at least one of the at least one workflow step is determined by means of a trained analysis function which comprises a second transformer. This can also be done as part of the preparation, but in exemplary embodiments also "online", i.e. when the corresponding second text embedding has been selected. An automatic workflow is therefore generated by the trained analysis function using the at least one control measure directly from a text description of the workflow.In this case, the trained analysis function can, for example, proceed step by step, particularly for each workflow step, through the workflow documentation associated with the second text embedding and determine control measures for each step, particularly in control language for the medical device. In other words, each workflow step described by text in the workflow documentation can be used as input data for the trained analysis function ("prompts"). This automatically enables the derivation of at least one control measure for each second text embedding from workflow documentation, particularly a user manual.

[0053] Preferably, the selection of the second text embedding can be made at least partially on the basis of a user input, in particular with regard to a list of the second text embeddings with the highest similarity measures. This can be particularly useful if a user selection condition is met which indicates that no single second text embedding with a significantly higher similarity measure than the other second text embeddings could be determined, and thus there is uncertainty in the interpretation of the user's wishes. If the user selection condition is not met, the second text embedding with the highest similarity is automatically selected, in particular for confirmation by the user or for the direct implementation of the at least one control measure.If this user selection condition is met (or in other embodiments, generally), the semantic content of the second text embeddings with the highest degree of similarity to the first text embedding can be output as a selection list, for example, a textual description, in particular with an output or visualization of the similarity measures, and a user can select the desired second text embedding. The output of the second text embeddings in the list can, for example, be the headings in the user manual or, in general, the workflow documentation that were used as their basis.

[0054] Furthermore, it can be provided that confirmation of the selected second text embedding is requested by the user, in particular in the case of a selected second text embedding to which a plurality of control measures are assigned. This is particularly expedient if an automatic selection has been made, for example due to a very high degree of similarity or a very high distance to the similarity measures of other second text embeddings, in particular therefore non-fulfillment of the user selection condition. The user is then informed, on the one hand, of which control measures now follow and, on the other hand, can abort by refusing confirmation, for example if there is a misunderstanding or an input error. If confirmation is refused, it can be provided that the list of second text embeddings with the highest similarity measures is output again or nevertheless, in order to allow the user to select a correct second text embedding.

[0055] Specifically, it can be provided that the similarity measure is determined by a classifier, in particular a trained classification function, and / or the similarity measure is determined from the group comprising cosine similarity, Euclidean distance, and Jaccard similarity. Cosine similarity is preferred.

[0056] It should be noted in general , as already indicated several times for the second specific further development , that the control measures can comprise an ordered sequence of workflow steps that are at least partially carried out automatically .

[0057] In this context, expedient embodiments can provide that for at least one workflow step information is requested from the user and used in a subsequent workflow step and / or for at least one workflow step an automatic operation of a control element in a user interface takes place, which is visualized and / or explained acoustically to the user. In the first case, the necessary information is specifically obtained from the user, for example if it is not possible to determine this by evaluating the medical data, in particular image data. This can be done in particular using interaction mechanisms of a user interface of the medical technology device which are already provided. Input and selection windows can be used which are also used for manual operation of corresponding control elements in accordance with the workflow documentation.Other input devices and / or input elements can also be used, for example, acoustic input devices such as microphones. The input can also be explained, for example, "Click an endpoint and drag a rectangle" or "Please enter the text for a note on the image."

[0058] It is also conceivable that for at least one workflow step an automatic operation of a control element in a user interface is carried out, which is visualized and / or explained acoustically to the user. In this way, operation of the medical technology device is made easier for the user and at the same time it is made possible for the user to follow the automated operating process and perceive the explanation, thus improving the operation for future tasks. For example, a highlighted mouse pointer can be moved to a menu item or other control element and this can be operated with a visible animation. In this variant, operating actions are carried out automatically and are explained particularly for a learning process, for example "clicks" are carried out automatically.

[0059] In embodiments, it can be provided that for at least one control measure, in particular one that fulfills a criticality condition, confirmation from the user is requested before it is carried out, with execution only taking place after confirmation. The criticality condition can, for example, indicate whether the control measure has lasting consequences and / or an effect on a patient, in which cases the user can be asked for confirmation before execution for safety reasons. Critical control measures include, for example, those in which an image acquisition is triggered or acquisition parameters, for example X-ray doses, are adjusted, permanent changes are made to medical data, and the like.Confirmation can be performed via an operating means and / or an input means of the medical device, for example, by means of an acoustic input via a microphone and / or actuation of a switch, button, or operating lever (joystick). Embodiments are also conceivable in which, particularly in the case of simple confirmation (for example, pushing the operating lever), each control measure or at least each workflow step must be confirmed by the user.

[0060] In an exemplary embodiment of the second specific development, it can be provided, for example, that the trained language analysis function, e.g., designed as an LLM or implementing such a function, is first applied to predefined sections, e.g., headings, of a user manual and / or other workflow documentation in order to determine the second text embeddings. Then, through further processing of these sources based on the trained analysis function, the workflows described therein can be translated into the control commands assigned to the second text embeddings. The system is now ready for use.

[0061] In the application, the user then submits a query, for example using their voice, a keyboard, or the like. If not already done, this query is converted into text as query input and passed as user input information to the trained speech analysis function, which supplies the corresponding first text embedding as output data. A semantic search then takes place using the second text embeddings by determining a similarity measure between the first text embedding and the second text embeddings, for example a cosine similarity as a vector similarity measure, which allows semantic analysis.The pairs with the highest similarity measures, i.e. the smallest distance between the text embeddings, are found on and, for example, depending on whether the user selection condition is met, are displayed to the user to select a second text embedding and / or forwarded to the automatic selection of a second text embedding.

[0062] After selecting a second text embedding, the medical device is controlled according to the control measures, in particular by using a control language for the medical device directly. It can be provided that the control measures are at least partially confirmed by the user.

[0063] A concrete application example could look like this: The user query is "I would like to add a label to my three-dimensional image dataset." Based on the similarity measures of a first text embedding derived from this with second text embeddings from a user manual, the section "Adding annotations to 3D volumes" is found and the corresponding second text embedding is selected. The medical device asks the user whether they want to execute this workflow. If the user confirms, the medical device checks whether a three-dimensional image dataset is loaded and displayed, whereupon the medical device automatically clicks a menu icon in the user interface. The "Measurements" tab is then automatically clicked, whereupon the annotation icon is automatically clicked.The medical device then requests user input, namely, "Please click on the starting point (head of the arrow) and draw a line to the end point." The user input now includes the position of an arrow that indicates the annotated location in the three-dimensional image dataset. This annotation arrow is displayed accordingly. The user is then asked to enter an annotation text, which is displayed as user input associated with the arrow.

[0064] In exemplary embodiments, the medical technology device can now be operated in such a way that it offers context-sensitive suggestions for further workflows, for example in the case just described "Would you like to adjust the object properties?" If the user accepts the suggestion, the corresponding workflow can be carried out, in particular automatically as a result of control measures.

[0065] A third specific development of the invention provides that the trained speech analysis function uses a plurality of successive speech analysis models to determine the at least one control measure, wherein, in the case of user input information which comprises a clinical query text provided by medically trained personnel as query input, a manufacturer-independent linguistic description text of technical measures necessary for implementing the clinical query on medical technology devices in general, in particular with identification of possible technical measures on the medical technology device to be controlled, is determined as first intermediate data by means of a first speech analysis model, in particular a first large language model, and the output data is determined from the first intermediate data as input data by means of at least one further speech analysis model, in particular a further large language model.

[0066] In such variants of the present invention, it is therefore proposed to proceed in several stages to the linguistic analysis of the query input, at least when this is formulated in clinical, in particular medical, terms. If such a clinical user query is made, a "translation" into technical terms can first take place. For this purpose, a language model, preferably an LLM, is used in particular to mediate between the user and the medical technology device, for example a C-arm X-ray device. In this multi-stage approach, comparable to a compiler, the clinical query is first translated into a "code" or language that is platform-agnostic but uses technical terms, after which a further translation into a platform-specific "code", i.e. the at least one control measure, takes place.

[0067] The first speech analysis model of the trained speech analysis function is therefore independent of the medical technology device (platform independent), and therefore also manufacturer independent, trained and relates clinical use cases to technical terms. It therefore translates the clinical query text into technical terms. With particular advantage, the first speech analysis model can also provide information about the optimal equipment required and compare it with the existing equipment, in particular existing medical technology devices and their features. For example, the output of the first speech analysis model can include an explanation such as: "A computer tomography device would be optimal for this, however, three-dimensional data can also be obtained with the existing C-arm

[0068] X-ray device capture . "

[0069] At least one further language analysis model then relates to the specific medical technology device to be used (application to several medical technology devices is also conceivable in principle) and is thus platform-specific in order to determine the specific implementation of the identified technical measures, in particular taking into account further information, by means of control measures on the medical technology device.

[0070] This third training course further increases the usability of medical technology devices and also enables implicit and lifelong user training. If the procedure described here is understood as providing a "copilot" for the user, this will be used more frequently initially, but will result in a steep learning curve, especially with appropriate user interaction, which will be explained in more detail below.

[0071] It should be noted that the methodology can be used generally, even beyond the specific third training course, to monitor the frequency of user requests and, for example, particularly in connection with the first and at least one further speech analysis model, to check which user requests cannot be implemented with the medical device but might represent useful extensions. Based on this, features can then be prioritized and targeted research and development can take place.

[0072] Preferably, it can be provided that at least some of the output data of at least one of the language analysis models, in particular the linguistic description text of the first intermediate data, is output to the user for confirmation, wherein a respective further processing condition checks whether the confirmation is present and a subsequent step can only be carried out if confirmation is present. This step also serves to train the user, who becomes more familiar with the technical options over time. A linguistic description text is therefore provided as intermediate data which describes the technical implementation of the task of the clinical query text and, optionally, the optimal equipment in comparison to what is available. The user then confirms whether the technical expressions at least essentially solve the clinical query, for example explicitly via an input device or acoustically.The confirmation can, especially in the last case, also be implicit / partial, for example in the form “Well, that might work.”

[0073] Ultimately, with regard to each of the trained speech analysis models, it may be advantageous to use user feedback information received during the confirmation process to further train the respective speech analysis model. In this way, the respective speech analysis model can be continuously improved and optimized.

[0074] In a particularly advantageous embodiment of the invention, it can be provided that the first speech analysis model is trained with and / or on the basis of scientific articles that link specific medical use cases with technical means. Since the first speech analysis model is intended to correlate clinical inquiries / use cases with technical measures, scientific articles or publications that deal with the solution of clinical tasks using technical measures offer an ideal foundation and knowledge base for the first speech analysis model. Clinical use cases are linked to technical features.In connection with information on optimal equipment, in particular for comparison with existing equipment, the first language analysis model can further correlate medical publications and / or information with general information on medical technology equipment, for example, "C-arm X-ray equipment can acquire three-dimensional data".

[0075] Preferably, based on restriction information provided by the user, at least some of the scientific articles can be excluded from the correlation. Thus, for at least one of the language analysis models, in particular the first language analysis model, the information sources that form the basis of the correlation can be disclosed and / or customizable. For example, a user can select sources from which certain corrections are to be made and / or exclude certain sources, for example publication platforms. For example, it is conceivable to exclude fields of knowledge or application practices not followed by the user in order to achieve an output of intermediate data that is better tailored to the user's needs.

[0076] As already indicated, the first language analysis model can expediently determine, as part of the intermediate data, requirement information on required medical technology facilities in general for the technical measures. As part of the first language analysis model or downstream of it, the requirement information can be compared with availability information describing available medical technology facilities. The result of the comparison can be output, in particular which available medical technology facility can provide which technical measures. In particular, the first language analysis model can correlate medical publications and / or information with general equipment information in this context.

[0077] Preferably, it can be provided that a second speech analysis model, specifically trained for at least one of the at least one required medical technology device, is used as a further speech analysis model to determine modification data as second intermediate data from input data, which, in addition to at least a part, which in particular comprises the at least one required medical technology device, of the first intermediate data, comprises case data describing a clinical case associated with the clinical query text and / or patient data describing a patient associated with the clinical query text, wherein the modification data, in particular as a recommendation text, comprise at least one recommended modification and / or concretization of at least one of the technical measures for the at least one required medical technology device, and / or that a third,for at least one of the at least one required medical device, a specifically trained speech analysis model is used to determine the at least one control measure from input data, which includes at least the first intermediate data and in particular also the second intermediate data.

[0078] In a particularly preferred embodiment, three trained speech analysis models can be used which lead step by step to concrete implementation as control measures. The first speech analysis model puts the clinical user query into a technical context, particularly with regard to the medical technology devices required and their equipment. All of this takes place completely independently of the at least one medical technology device actually available and to be used. In the second step, i.e. using the second speech analysis model, the technical description is made more specific with knowledge of the medical technology device to be used and the concrete clinical case, in particular the patient and the examination or...Treatment goal is achieved by including patient data and / or case data in addition to the first intermediate data, and the second speech analysis model is trained specifically for the medical technology device to be used, be it (preferably) the model or the general genre. In other words, the second speech analysis model preferentially correlates diagnostic information (case data) and information about the patient (patient data) as well as information about the medical technology device with the general description of the technical procedure in order to adapt and / or specify technical measures for the specific clinical case. For example, it can be checked whether the patient data and / or the case data meet certain criteria.A modification of technical measures may arise, for example, if the patient has intolerances, e.g., with regard to contrast media or materials, and / or if the patient has special characteristics, e.g., if they are a child and / or obese. The second language analysis model can suggest specific clarifications and / or adjustments (modifications) in text form based on the second intermediate data, for example, something like: "The patient appears to be obese, therefore a protocol with an extended field of view should be considered." It may be provided that the user is given the opportunity to reject and / or edit modifications and / or clarifications.

[0079] As a final, third trained speech analysis model, one can be used which is again specifically trained for the medical technology device, here in particular for the model or at least the manufacturer, and which can concretize the technical measures described in the first and second intermediate data as control measures, in particular in a control language of the medical technology device. In other words, it relates the technical expressions in the first and second intermediate data to the control language specific to the medical technology device or its manufacturer. At the latest here, the at least one first transformer can be used, wherein when the first speech analysis model and / or the second speech analysis model are designed as LLMs, which is preferred according to the invention, these also naturally implicitly use transformers.

[0080] In a suitable embodiment of this third specific development, each speech analysis model can provide, in addition to its output data, reliability information regarding the output data, which is then output to the user. It is therefore conceivable for the speech analysis models to evaluate themselves with regard to the inference and also make this information available to the user. The reliability information can also be processed in other ways, for example to determine the extent to which the user must confirm individual processes and / or the extent to which they receive additional support. For example, if reliability information indicates low reliability, potentially helpful features of the medical technology device can be highlighted, in particular with links to the user manual and / or explanatory videos.The latter support information can also be made available to the user automatically, particularly in the context of tax measures. However, if the reliability information indicates a high degree of reliability, it may be sufficient to have at least one tax measure confirmed by a simple action, for example, a simple click.

[0081] A concrete application example for the third concrete further training could, for example, be based on a query input whose text form “I am wondering which blood vessel is the main supply to this AVM, the upper or the lower one.” This could be expressed by the first language analysis model in technical measures to solve the clinical question, for example as intermediate data like this: “The user would like to visualize two time-resolved three-dimensional digital subtraction angiography (time-resolved 3D-DSA or 4D-DSA) volumes synchronously at the same time. For better comparability, both volumes should have distinguishable colors. Computed tomography with contrast would be most suitable for this task. However, the available C-arm X-ray device with contrast can provide sufficient diagnostic accuracy."If the patient is obese, as described by the patient data, the second language analysis model can add the following as second intermediate data: "Since the patient appears to be obese, a large-volume protocol is recommended." Control measures can, for example if the three-dimensional time-resolved DSA image data sets are already available, include programming commands for acquisition parameters (e.g., large-volume protocol) and / or control commands such as "load the first image data set, then load the fused second image data set, apply automatic registration". Control measures, in particular control and / or programming commands, can of course also be provided for the acquisition and the like, just as for the actual visualization.

[0082] In addition to the method, the present invention also relates to a medical device, in particular a medical imaging device, which has a control device designed to carry out the method according to the invention. All statements regarding the method according to the invention can be applied analogously to the medical device according to the invention and vice versa.

[0083] The medical technology device can have input means, in particular comprising a microphone and / or a keyboard for entering the query input and / or other user inputs, and / or output means, in particular for realizing a user interface.

[0084] The control device can have at least one memory device and at least one processor. Functional units can be implemented by hardware and / or software to carry out steps of the method according to the invention. If the query input is made acoustically, for example via the microphone, the control device can have a conversion unit to convert the query input into text form as user input information. The control device expediently has:

[0085] - a compilation unit for compiling input data, which includes at least the user information, for the trained speech analysis function,

[0086] - an application unit for applying the trained speech analysis function to the input data to obtain output data comprising the at least one control measure and / or from which the application unit derives the at least one control measure, and

[0087] - a control unit for implementing at least one control measure.

[0088] To implement further method steps as described above, further functional units may of course be provided. In particular, the application unit may have subunits that implement the multiple steps in multi-stage procedures and / or request confirmation from a user and / or receive further information, for example, patient data and / or case data, and the like. The control unit may also have subunits, in particular an evaluation subunit for evaluating the medical technology data and the like.In particular, in the first concrete further training, alternating use of the application unit and the control unit may occur, for example if the medical technology data is initially evaluated in a first control measure and the application unit is again used in a further control measure to use the trained translation function, which may correspond to the same LLM as the trained speech analysis function, and to determine or adapt secondary control measures, which in turn can then be implemented by the control unit.

[0089] The control unit can expediently be used generally to control various components of the medical device, and in particular also for other specific uses of the medical device outside of the method described here, for example, by means of operating devices. In an imaging device, the control unit also corresponds in particular to a recording unit that generally controls the recording operation of the imaging device.

[0090] A computer program according to the invention can be loaded directly into a storage means of a control device of a medical technology device and has program means such that when the computer program is executed on the control device, the device is caused to carry out the steps of a method according to the invention. The computer program can be stored on an electronically readable data carrier according to the invention, which therefore comprises control information stored thereon, which comprises at least one computer program according to the invention and is designed such that when the data carrier is used in a control device of a medical technology device, the device is designed to carry out a method according to the invention. The electronically readable data carrier can in particular be a non-transient data carrier, for example a CD-ROM.

[0091] Further advantages and details of the present invention will become apparent from the following exemplary embodiments and from the drawings, which show:

[0092] Fig. 1 shows an embodiment of an artificial neural network,

[0093] Fig. 2 shows an example of a Convolutional Neural Network (CNN),

[0094] Fig. 3 shows a general flow chart of embodiments of the method according to the invention,

[0095] Fig. 4 shows a flow chart of a first concrete implementation of the method of Fig. 3, Fig. 5 shows a flow chart of a second concrete implementation of the method of Fig. 3,

[0096] Fig. 6 is a flow chart of a third concrete implementation of the method of Fig. 3,

[0097] Fig. 7 a medical device according to the invention,

[0098] Fig. 8 shows the functional structure of a control device of the medical device of Fig. 7, and

[0099] Fig. 9 shows an embodiment of the medical device as a C-arm X-ray device.

[0100] Fig. 1 shows an embodiment of an artificial neural network 1. English terms for the artificial neural network 1 are "artificial neural network", "neural network", "artificial neural net" or "neural net".

[0101] The artificial neural network 1 comprises nodes 6 to 18 and edges 19 to 21, each edge 19 to 21 being a directed connection from a first node 6 to 18 to a second node 6 to 18. In general, the first node 6 to 18 and the second node 6 to 18 are different nodes 6 to 18, but it is also conceivable that the first node 6 to 18 and the second node 6 to 18 are identical. For example, in Fig. 1, edge 19 is a directed connection from node 6 to node 9 and edge 21 is a directed connection from node 16 to node 18. An edge 19 to 21 from a first node 6 to 18 to a second node 6 to 18 is referred to as an ingoing edge for the second node 6 to 18 and as an outgoing edge for the first node 6 to 18.

[0102] In this embodiment, the nodes 6 to 18 of the artificial neural network 1 can be arranged in layers 2 to 5, wherein the layers can have an intrinsic order introduced by the edges 19 to 21 between the nodes 6 to 18. In particular, edges 19 to 21 can only be provided between adjacent layers of nodes 6 to 18. In the illustrated embodiment, there is an input layer 2 that only has the nodes 6, 7, and 8, each without an incoming edge. The output layer 5 comprises only the nodes 17, 18, each without an outgoing edge, with hidden layers 3 and 4 furthermore lying between the input layer 2 and the output layer 5. In the general case, the number of hidden layers 3, 4 can be chosen arbitrarily.The number of nodes 6, 7, 8 of the input layer 2 usually corresponds to the number of input values ​​to the neural network 1, and the number of nodes 17, 18 in the output layer 5 usually corresponds to the number of output values ​​of the neural network 1.

[0103] In particular, a (real) number can be assigned to nodes 6 to 18 of the neural network 1. Here, x denotes (n) i is the value of the i-th node 6 to 18 of the n-th layer 2 to 5. The values ​​of nodes 6, 7, 8 of input layer 2 are equivalent to the input values ​​of neural network 1, while the values ​​of nodes 17, 18 of output layer 5 are equivalent to the output values ​​of neural network 1. In addition, each edge 19, 20, 21 can be assigned a weight in the form of a real number. In particular, the weight is a real number in the interval [-1, 1] or in the interval [0, 1, ]. Here, w denotes (m ' n) ifj is the weight of the edge between the i-th nodes 6 to 18 of the m-th layer 2 to 5 and the j-th nodes 6 to 18 of the n-th layer 2 to 5. Furthermore, the abbreviation w^ is used for the weight w ^ n '" +1) defined .

[0104] To calculate output values ​​of neural network 1, the input values ​​are propagated through neural network 1. In particular, the values ​​of nodes 6 to 18 of the (n+l)-th layer 2 to 5 can be calculated based on the values ​​of nodes 6 to 18 of the n-th layer 2 to 5 by

[0105] Here, f is a transfer function, which can also be referred to as an activation function. Well-known transfer functions are step functions, sigmoid functions (e.g., the logistic function, the generalized logistic function, the hyperbolic tangent, the arctangent, the error function, the smoothstep function), or rectifier functions. The transfer function is primarily used for normalization purposes.

[0106] In particular, the values ​​are propagated layer by layer through the neural network 1, with values ​​of the input layer 2 being given by the input data of the neural network 1. Values ​​of the first hidden layer 3 can be calculated based on the values ​​of the input layer 2 of the neural network 1, values ​​of the second hidden layer 4 can be calculated based on the values ​​in the first hidden layer 3, and so on.

[0107] In order to be able to determine the values ​​w^ for the edges 19 to 21, the neural network 1 must be trained using training data . In particular, training data comprise training input data and training output data, which are referred to below as t± . For a training step, the neural network 1 is applied to the training input data to determine calculated output data . In particular, the training output data and the calculated output data comprise a number of values, where the number is determined as the number of nodes 17 , 18 of the output layer 5 .

[0108] In particular, a comparison between the calculated output data and the training output data is used to recursively adjust the weights within the neural network 1 (back propagation algorithm). In particular, the weights can be adjusted according to w'^> = w^' - y-5? n) -x' n)where y is a learning rate and the numbers

[0109] 5 years n) can be calculated recursively as based on 5j n+1) , if the (n+l)-th layer is not the output layer 5, and if the (n+l)-th layer is the output layer 5, where f is the first derivative of the activation function and y? n+1 the comparison training value for the j-th node 17, 18 of the

[0110] Output layer 5 is.

[0111] In the following, an example of a convolutional neural network (CNN) is also given with reference to Fig. 2. Note that the term "layer" is used in a slightly different way than for classical neural networks. For a classical neural network, the term "layer" refers only to the set of nodes that form a layer, thus a specific generation of nodes. For a convolutional neural network, the term "layer" is often used to refer to an object that actively modifies data, in other words, a set of nodes of the same generation and either the set of incoming or outgoing edges.

[0112] Fig. 2 shows an embodiment of a convolutional neural network 22. In the illustrated embodiment, the convolutional neural network 22 comprises an input layer 23, a convolutional layer 24, a pooling layer 25, a fully connected layer 26, and an output layer 27. In alternative embodiments, the convolutional neural network 22 may contain multiple convolutional layers 24, multiple pooling layers 25, and multiple fully connected layers 26, as well as other types of layers. The order of the layers can be chosen arbitrarily, with fully connected layers 26 typically forming the last layers before the output layer 27.

[0113] In particular, within a convolutional neural network 22, the nodes 28 to 32 of one of the layers 23 to 27 can be understood as being arranged in a d-dimensional matrix or as a d-dimensional image. In particular, in the two-dimensional case, the value of a node 28 to 32 with the indices i, j in the n-th layer 23 to 27 can be written as x ( n ) [ i , j ] . It should be noted that the arrangement of the nodes 28 to 31 of a layer 23 to 27 has no effect on the calculations within the convolutional neural network 22 as such, since these effects are determined exclusively by the structure and the weights of the edges.

[0114] A convolutional layer 24 is particularly characterized in that the structure and the weights of the incoming edges form a convolution operation based on a certain number of kernels. In particular, the structure and the weights of the incoming edges can be chosen such that the values ​​x k n ) the node 29 of the convolutional layer 24 as a convolution x k n ) =K k *x (n-1 ) based on the values ​​x^ n-1 ^ the node 28 of the previous layer 23 can be determined, where the convolution * in the two-dimensional case can be defined as

[0115] In it, the k-th kernel K ka d-dimensional matrix, in this embodiment a two-dimensional matrix, which is usually small compared to the number of nodes 28 to 32, for example a 3x3 matrix or a 5x5 matrix. In particular, this implies that the weights of the incoming edges are not independent, but are chosen such that they generate the above convolution equation. In the example for a kernel forming a 3x3 matrix, there are only nine independent weights (where each entry of the kernel matrix corresponds to an independent weight), regardless of the number of nodes 28 to 32 in the corresponding layer 23 to 27. In particular, for a convolutional layer 24, the number of nodes 29 in the convolutional layer 24 is equivalent to the number of nodes 28 in the previous layer 23 multiplied by the number of convolution kernels.

[0116] If the nodes 28 of the preceding layer 23 are arranged as a d-dimensional matrix, the use of the plurality of kernels can be understood as the addition of a further dimension, which is also referred to as the depth dimension, so that the nodes 29 of the convolutional layer 24 are arranged as a (d+1)-dimensional matrix. If the nodes 28 of the preceding layer 23 are already arranged as a (d+1)-dimensional matrix with a depth dimension, the use of a plurality of convolutional kernels can be understood as an expansion along the depth dimension, so that the nodes 29 of the convolutional layer 24 are likewise arranged as a (d+1)-dimensional matrix, wherein the size of the (d+1)-dimensional matrix in the depth dimension is larger than in the preceding layer 23 by the factor formed by the number of kernels.

[0117] The advantage of using convolutional layers 24 is that the spatially local correlation of the input data can be exploited by creating a local connection pattern between nodes of neighboring layers, in particular by each node having connections only to a small region of the nodes of the previous layer. In the illustrated embodiment, the input layer 23 comprises thirty-six nodes 28 arranged as a two-dimensional 6x6 matrix. The convolutional layer 24 comprises seventy-two nodes 29 arranged as two two-dimensional 6x6 matrices, each of the two matrices being the result of a convolution of the values ​​of the input layer 23 with a convolution kernel. Similarly, the nodes 29 of the convolutional layer 24 can be understood as being arranged in a three-dimensional 6x6x2 matrix, the last-mentioned dimension being the depth dimension.

[0118] A pooling layer 25 is characterized in that the structure and weights of the incoming edges as well as the activation function of its nodes 30 define a pooling operation based on a nonlinear pooling function f . For example, in the two-dimensional case, the values ​​x ( n ) the node 30 of the pooling layer 25 based on the values ​​x ( n+1 ) the node 29 of the previous layer 24 as In other words, by using a pooling layer 25, the number of nodes 29, 30 can be reduced by replacing a number of d2x d2 neighboring nodes 29 in the preceding layer 24 with a single node 30, which is calculated as a function of the values ​​of said number of neighboring nodes 29. In particular, the pooling function f can be a maximum function, an averaging function, or the L2 norm. In particular, for a pooling layer 25, the weights of the incoming edges can be fixed and not modified by training.

[0119] The advantage of using a pooling layer 25 is that the number of nodes 29, 30 and the number of parameters are reduced. This leads to a reduction in the amount of calculation required within the convolutional neural network 22 and thus to a control of overfitting. In the illustrated embodiment, the pooling layer 25 is a max-pooling layer in which four neighboring nodes are replaced with just a single node whose value is formed by the maximum of the values ​​of the four neighboring nodes. The max-pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, the max-pooling is applied to each of the two two-dimensional matrices, so that the number of nodes is reduced from seventy-two to eighteen.

[0120] A fully connected layer 26 is characterized in that a plurality, in particular all, of the edges are present between the nodes 30 of the previous layer 25 and the nodes 31 of the fully connected layer 26, wherein the weight of each of the edges can be individually adjusted.

[0121] In this embodiment, the nodes 30 of the preceding layer 25 and the fully connected layer 26 are shown both as two-dimensional matrices and as non-connected nodes (shown as a row of nodes, with the number of nodes reduced for clarity). In this embodiment, the number of nodes 31 in the fully connected layer 26 is equal to the number of nodes 30 in the preceding layer 25. In alternative embodiments, the number of nodes 30, 31 may be different.

[0122] Furthermore, in this exemplary embodiment, the values ​​of the nodes 32 of the output layer 27 are determined by applying the softmax function to the values ​​of the nodes 31 of the preceding layer 26. By applying the softmax function, the sum of the values ​​of all nodes 32 of the output layer 27 is one and all values ​​of all nodes 32 of the output layer are real numbers between 0 and 1. When the convolutional neural network 22 is used to classify input data, the values ​​of the output layer 27 in particular can be interpreted as the probability that the input data falls into one of the different classes.

[0123] A convolutional neural network 22 can also have a ReLU layer, where ReLU is an acronym for "rectified linear units". In particular, the number of nodes and the structure of the nodes within a ReLU layer is equivalent to the number of nodes and the structure of the nodes in the previous layer. The value of each node in the ReLU layer can be calculated, in particular, by applying a rectifier function to the value of the corresponding node in the previous layer. Examples of rectifier functions are f (x ) = max ( 0 , x ) , the hyperbolic tangent or the sigmoid function.

[0124] Convolutional neural networks 22 can be trained, in particular, based on the backpropagation algorithm. To avoid overfitting, regularization methods can be used, such as dropout of individual nodes 28 to 32, stochastic pooling, use of artificial data, weight decay based on the LI or L2 norm, or maximum norm constraints.

[0125] Fig. 3 shows a general flow chart of embodiments of the method described herein. This is used to use freely formulated query inputs from a user in text form to determine specific control measures for a medical technology device, in the embodiments described here an imaging device. In this case, language analysis techniques of artificial intelligence are used, specifically transformers and preferably language models, in particular LLMs. Modern language models usually use transformers. For example, BERT, RoBERTa, USE and / or SIE can generally be used as transformers, while a GPT is preferably used as the LLM. Alternatives to GPT include, for example, LaMDA or MT-NLG.The steps described below are carried out in particular and preferably by a control device of the medical technology device, whereby external components, for example a voice input module, can also be used for other components (for example a C-arm arrangement).

[0126] In a step S 1, user input information is determined or received via an interface, possibly also an internal one. The basis of the user input information is a query input from a user, which is formulated linguistically as free text and, if necessary, converted into text form for the user input information. This applies in particular if the query input is received acoustically, for example via a microphone. Corresponding speech recognition algorithms are basically known in the prior art and can also be used in the context of the present invention. However, it is also possible to choose other input methods, for example to use a keyboard or a similar input device.The query input can already relate to concrete technical questions (for example "I would like to take the images in such a way that an instrument can be seen more clearly"), but is in particular formulated in clinical terms (for example: "I would like to be able to better assess whether and in which of the patient's vessels there is calcification"). It should also be noted at this point that the present invention has been shown to be particularly suitable for use in the field of angiographic examinations and / or interventions, so that the medical technology device designed as an imaging device can in particular be an angiography device.

[0127] In a step S2, the user input information is then evaluated using the trained speech analysis function, which includes at least one first transformer, in order to determine at least one control measure as final output data or derived from the output data. Specific embodiments in this regard are explained in more detail below with reference to Figures 4 to 6.

[0128] The control measures that can be determined in step S2 have been determined in particular in a control language, for example machine language, for the medical technology device and can accordingly be applied directly in the control device for implementation in a step S3. While some of the control measures can relate to the output or retrieval of information to or from the user, at least one of the at least one control measure preferably relates to the setting of an acquisition parameter in the imaging device and / or the triggering of at least one functional process of the imaging device, in particular an acquisition process, and / or an evaluation of image data recorded with the imaging device.

[0129] It should be noted that in exemplary embodiments, it is also possible to return from step S3 to step S2, for example if the following control measures are to be specified and / or if secondary control measures are to be determined and then carried out after or as part of the implementation of primary control measures, which can serve, for example, to determine additional information and / or to process medical data.

[0130] The following concrete implementations, which correspond to the three concrete further developments mentioned above, can of course also be used in addition and features can be transferred analogously, where applicable, from one implementation to the other.

[0131] Fig. 4 shows a flow chart of a first concrete implementation of steps S2 and S3, which correspond to the first concrete further development. In this case, there is a query input from which no concrete automated control measures for implementing the task of the query input can be directly determined without additional information or without prior processing of medical technology data to which the query input refers. This applies, for example, if the query input relates to features contained in the medical technology data, here image data due to the design of the medical technology device as an imaging device, for example anatomical features or other features, for example displayed instruments and / or implants.

[0132] In a first pass of the speech analysis in step S2 (in Fig. 4, in a step S2a), control measures are determined using the trained speech analysis function. At least one of these control measures, which is to be carried out first, determines information and / or is preparatory, relates in the control language of the imaging device to an evaluation of the image data.

[0133] This at least one control measure to be carried out first is then carried out in the following step S3a. In the present case, in which image features in particular play a role in automatically determining secondary control measures that answer the query input, a trained segmentation function is first applied as a trained evaluation function to the image data of at least one relevant image data set in step S3a, for example the aforementioned trained segmentation function "Segment Anything". It should be noted that embodiments are also conceivable in which image data sets of different, mutually registered modalities are evaluated, for example also those of external modalities.Generally speaking, the image datasets can include two-dimensional fluoroscopy image datasets, two-dimensional DSA image datasets, cone-beam CT image datasets, three-dimensional DSA image datasets, computed tomography image datasets, and / or magnetic resonance image datasets. The trained segmentation function provides as output a division of the image dataset into sub-regions (sub-images).

[0134] In a step S3b, a trained classification function is then applied to assign to the sub-areas corresponding labels indicating the respective anatomical and / or other feature as text, for example "neurological vessel segment", "internal carotid artery", "aneurysm", "catheter type A", "stent type A", "liver" and the like.

[0135] It should be noted at this point that when training the classification function, manual clustering and manual annotation, i.e. manual addition of labels, can be used to generate the ground truth.

[0136] The output of the trained classification function is ultimately a processed image dataset (with the segmented sub-areas and the labels) and / or additional information that contains the division into the sub-areas and the associated labels.

[0137] In a step S3c, a subsequent control measure leads back to the general scope of step S2 in that in a step S2b a trained translation function is applied to the user input information as well as the (optionally processed) image data set and / or the additional information, wherein the trained translation function, particularly in the case of an LLM, can also correspond to the trained speech analysis function, but at least preferably also uses the first transformer to determine at least one secondary control measure, which ideally is again present in the control language for the imaging device. The trained translation function uses the data additionally obtained by automatic evaluation of the image data, in particular the processing and / or the additional information, to determine the secondary control commands in such a way that the user request of the request input, which in particular refers to image features, can be fulfilled.This may, for example, include adjusting the imaging device's recording parameters for subsequent recordings and / or adjusting the output of the image data set, for example, zooming, highlighting certain image features, and the like. It is also conceivable that the secondary control measures relate to a further evaluation of the image data set, for example, taking a measurement, determining a score, and the like.

[0138] Optionally, these secondary control measures can be output again for confirmation by the user, whereby the subsequent execution of the secondary control measures in step S3d only takes place after confirmation by the user.

[0139] Fig. 5 shows a second concrete implementation in which text embeddings are used to identify suitable control measures. In step S2c, a first text embedding of the query input is determined in text form using the first transformer of the trained speech analysis function.

[0140] In preparatory steps, a number of second text embeddings were determined, specifically from a user manual and / or other workflow documentation. The first transformer of the trained language analysis function was applied to predefined sections, in particular at least the headings of parts explaining workflows, for example chapters. In addition, control measures assigned to these second text embeddings were determined by applying a trained analysis function, which comprises a second transformer, to the text sections describing the at least one workflow step of the at least one workflow. The control measures are in turn determined directly in the control language of the medical technology device, in this case the imaging device. This creates a database with second text embeddings, each of which is assigned at least one control measure.

[0141] In step S2d, a similarity measure between the first text embedding and the second text embeddings is determined—in this case, a cosine similarity. This ultimately corresponds to a semantic search, since the embeddings describe the semantic content.

[0142] In a step S2e it is now checked whether a user selection condition is met. This can indicate, for example, that no single second text embedding with a significantly higher degree of similarity than the other second text embeddings could be determined. If the user selection condition is not met, the second text embedding with the highest similarity is automatically selected in a step S2f, in particular for confirmation by the user or for the direct implementation of the at least one control measure. If confirmation by the user does not take place, it is optionally possible to switch to step S2f, which is described below (not shown).

[0143] If the user selection condition is met (or in other embodiments generally), in a step S2g the semantic content of the second text embeddings with the highest degree of similarity to the first text embedding is output as a selection list, for example a textual description in each case, in particular with an output or visualization of the similarity measures. For example, a predetermined number of the second text embeddings with the highest similarity measures can be displayed. A user can select the desired second text embedding from the selection list. In this case, the headings in the user manual (or generally the workflow documentation) are displayed in the selection list.

[0144] In a step S3e, the at least one control measure associated with the selected second text embedding is then executed. This can, in particular, also include, for example, selecting menu items in a user interface in a manner that is recognizable to the user and / or requesting further user input. In other embodiments, it is also possible to display the workflow, in particular the corresponding portion of the user manual or the workflow documentation, instead of or in addition to the actual execution of the workflow.

[0145] It should be noted in general at this point that it is conceivable to have at least some of the control measures to be carried out in step S3 confirmed by the user, in particular when a criticality condition is met, which is given, for example, when a function is triggered, for example image recording or adjustment of a patient bed.

[0146] When combining the second concrete implementation with the first concrete implementation, it is also conceivable to use text embedding for both the determination of the primary tax measures and the determination of the secondary tax measures, as described here.

[0147] Fig. 6 shows a third concrete implementation in which the control measures are derived from the query input in text form describing the user's intentions in a step-by-step manner. This is particularly useful when clinical or medical terms are used instead of or alongside technical terms in the query input. For this purpose, the trained language analysis function is expediently divided into three trained language analysis models, which are applied successively. At least the first and second language analysis models are expediently language models, in particular LLMs.

[0148] In a step S2h, a first language analysis model (LLM1) is applied. This model was trained independently of manufacturers and specific medical technology facilities and focuses on medical publications that link technical features with specific clinical use cases. It translates the clinical query into technical terms, which are provided in text form as initial intermediate data. In addition, it provides information on optimal equipment as initial intermediate data and compares it with the available medical technology facilities. Optionally, reliability information can also be output in addition to the initial intermediate data.

[0149] The first intermediate data and, if applicable, the reliability information are output to the user in a step S2 i, so that the user can confirm that the described technical procedure has solved their clinical task. In addition to simple yes / no input options, it is also conceivable to allow intermediate stages, particularly when receiving the response as speech via a microphone. Intermediate stages result in a type of fuzzy feedback. This feedback information, as well as any additional feedback information provided by the user, is used for further training / fine-tuning of the first speech analysis model.

[0150] After confirmation by the user and if the technical measures are feasible, once the at least one required medical device, here an imaging device, is available, in a step S2j a second speech analysis model (LLM2) is applied to the first intermediate data together with the patient data describing the patient and the case data describing the clinical case. The second speech analysis model is trained specifically for the imaging device, in particular its model or at least its manufacturer. It checks whether there is a need for clarification or modification that must be taken into account when determining the control measures. If a patient has metallic implants, for example, the use of low-metal artifact protocols and / or metal artifact correction algorithms can be recommended.Generally speaking, the second language analysis model adds case- and patient-specific specifications and modifications of the technical measures of the first intermediate data in text form, specifically for the imaging device to be used, as second intermediate data.

[0151] The second intermediate data, possibly again with reliability information, are also output to the user for confirmation in a step S2 k . Here, too, feedback information can be used to improve the trained second speech analysis model.

[0152] After confirmation by the user, in a step S21, a third language analysis model (LLM3 or transformer), again specifically trained for the imaging device, is applied to the first intermediate data and the second intermediate data in order to convert the technical expressions and measures into system-specific language, in particular the control language of the imaging device. In other words, the at least one control measure is determined. Reliability information can also be determined for this purpose.

[0153] In a step S2m, a check is performed to determine whether the reliability information indicates a sufficiently high degree of reliability that the user's intentions are reflected. Feedback information, particularly fuzzy confirmations, may also be included alternatively or additionally. If the reliability is sufficiently high, the at least one control measure is immediately adopted in step S2n and executed after confirmation by the user in step S3f.

[0154] Otherwise, in step S2o, additional helpful support functions of the imaging device can be highlighted, for example, with links to a user manual and / or explanatory videos. This allows the user to obtain further information before confirming at least one control action. It should also be noted that in this specific implementation, the option of adapting the information sources of the first speech analysis model can also be offered, for example, by excluding certain publication sources.

[0155] Fig. 7 shows a schematic view of a medical technology device 33 according to the invention, which in this case is designed as an imaging device 34. The medical technology device comprises a control device 35 which controls its operation and is designed to carry out the methods described here. In addition to imaging components 36, it also controls output means 37 provided for forming the user interface, for example comprising a display, input means 38, for example comprising a microphone 39 and / or operating means, and other components.

[0156] Fig. 8 shows the functional structure of the control device 35 in more detail. In addition to a storage means 40, in particular for storing the trained speech analysis function as well as other functions used and processed data, the control device initially comprises a conversion unit 41 assigned to the microphone 39, which converts a spoken, acoustic query input into text form. The user input information thus determined can be forwarded via an internal interface to a compilation unit 42 for compiling input data, which at least includes the user information, for the trained speech analysis function (step S1).

[0157] The control device 35 further comprises an application unit 43 for applying the trained speech analysis function to the input data according to step S2 to obtain output data that includes the at least one control measure and / or from which the application unit 43 derives the at least one control measure. Finally, a control unit 44 is provided, which also serves as a recording unit. It also serves to implement the at least one control measure according to step S3.

[0158] The functional units described here can have subunits for implementation of the substeps described in the concrete implementations, for example an evaluation subunit 45 for carrying out the evaluation according to steps S3a and S3b or other evaluating control measures.

[0159] Fig. 9 shows, as a concrete embodiment of the imaging device 34, a C-arm X-ray device that can be used as an angiography device. The imaging device 34 has a C-arm 46 on which an X-ray emitter 47 and an X-ray detector 48 are arranged opposite one another. The C-arm 46 and thus the resulting recording arrangement can be brought into various positions relative to a patient table 49. Three-dimensional image data sets can also be recorded by rotating the C-arm 46 and / or other movements. In addition, fluoroscopic imaging in various projection geometries is possible.

[0160] As an output means 37, a monitor 50 is shown here by way of example, on which recorded image data can also be displayed; the input means 38, in particular comprising the microphone 39, can be spatially assigned at least in part to this monitor.

[0161] Although the invention has been illustrated and described in detail by the preferred embodiment, the invention is not limited to the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. Regardless of the grammatical gender of a particular term, this includes persons of male, female, or other gender identities.

Claims

Patent claims 1. Computer-implemented method for operating a medical device (33), in particular a medical imaging device (34), comprising the following steps: Receiving user input information comprising a natural query input of a user in text form, applying at least one trained speech analysis function, which comprises at least one first transformer, to the user input information to obtain output data, wherein the output data comprise at least one control measure, in particular at least one control command and / or programming command, and / or the at least one control measure is determined from the output data, and Providing at least one control measure for controlling the medical technology device (33).

2. Method according to claim 1, characterized in that at least one control measure, in particular a programming command, the setting of an operating parameter, in particular a recording parameter in an imaging device (24), and / or at least one control measure the triggering of at least one functional process of the medical technology device (33), in particular a recording process in an imaging device (24), and / or at least one control measure an evaluation of medical technology data recorded with the medical technology device (33), in particular of image data recorded with the imaging device (34).

3. Method according to claim 1 or 2, characterized in that, as an information-determining and / or preparatory control measure to be carried out first, medical technology data obtained with the medical technology device (33), in particular image data recorded with the imaging device (34), are used for processing and / or determining an additional information can be evaluated for subsequent tax measures.

4. Method according to claim 3, characterized in that the evaluation comprises a segmentation of the image data and / or the application of a trained evaluation function, in particular a trained segmentation function, to the medical technology data.

5. Method according to claim 3 or 4, characterized in that the additional information and / or the processed medical technology data together with the user input information and / or the output data of the trained speech analysis function and / or secondary data derived from these output data are used as input data for a trained translator function which determines or adapts at least one secondary control measure, and the at least one secondary control measure is provided.

6. The method according to claim 5, characterized in that the trained translator function comprises the first and / or a second transformer and / or the trained speech analysis function is used as the trained translator function.

7. Method according to one of the preceding claims, characterized in that a first text embedding is determined from the user input information by means of the trained speech analysis function, wherein in order to determine the at least one control measure from the first text embedding, a similarity measure between the first text embedding and a plurality of second text embeddings provided in a database, each of which is assigned at least one database control measure for the medical technology device (33), is determined and a second text embedding with one of the highest similarity measures is selected, the at least one database control measure associated with the selected second text embedding is used as the at least one control measure.

8. Method according to claim 7, characterized in that, in order to determine the second text embeddings, the trained language analysis function is applied to predefined sections, in particular headings, of a workflow documentation, in particular a user manual.

9. The method according to claim 8, characterized in that from the text sections assigned to and / or comprised by the predefined sections, which describe at least one workflow step, at least one control measure assigned to the corresponding second text embedding for implementing at least one of the at least one workflow step is determined by means of a trained analysis function which comprises a second transformer.

10. Method according to one of the preceding claims, characterized in that the trained speech analysis function uses several successive speech analysis models to determine the at least one control measure, wherein, in the case of user input information which comprises a clinical query text provided by medically trained personnel, a linguistic description text, independent of the medical technology device (33) to be controlled, of technical measures necessary for implementing the clinical query on medical technology devices (33) in general, in particular with identification of technical measures possible on the medical technology device (33) to be controlled, is determined as first intermediate data by means of a first speech analysis model, in particular a first large language model, and by means of at least one further speech analysis model, in particular a further large language model, from the first intermediate data as input data the output data are determined.

11. The method according to claim 10, characterized in that at least part of the output data of at least one of the language analysis models, in particular the linguistic description text of the first intermediate data, is output to the user for confirmation, wherein a respective further processing condition checks the presence of the confirmation and a subsequent step is carried out only if confirmation is present.

12. Method according to claim 10 or 11, characterized in that a second speech analysis model, specifically trained for the medical technology device (33) to be controlled, is used as a further speech analysis model for determining modification data as second intermediate data from input data, which, in addition to at least a part of the first intermediate data, which in particular comprises the medical technology device (33) to be controlled, comprises case data describing a clinical case associated with the clinical query text and / or patient data describing a patient associated with the clinical query text, wherein the modification data, in particular as a recommendation text, comprise at least one recommended modification and / or concretization of at least one of the technical measures for the medical technology device (33) to be controlled, and / or that a third,a speech analysis model specifically trained for the medical device (33) to be controlled is used to determine the at least one control measure from input data which comprises at least the first intermediate data and in particular also the second intermediate data.

13. Medical technology device (33), in particular imaging device (34), comprising a control device (35) designed to carry out a method according to one of the preceding claims.

14. Computer program which, when executed on a control device (35) of a medical device (33), causes the device (33) to carry out the steps of a method according to one of claims 1 to 12.

15. An electronically readable data carrier on which a computer program according to claim 14 is stored.

Images