Computer-implemented method for operating a medical device, medical device, computer program and electronically readable data carrier

Abstract

The invention relates to a computer-implemented method for operating a medical technology device (33), in particular a medical imaging device (34), comprising the following steps: - Receiving user input information that includes a natural user request input in text form, - Applying at least one trained language analysis function, comprising at least one first transformer, to the user input information to obtain output data, wherein the output data comprises at least one control action, in particular at least one control command and / or programming command, and / or the at least one control action is determined from the output data, and - Performing at least one of the control measures assigned to the selected second text embedding to control the medical device (33).

Description

[0001] The invention relates to a computer-implemented method for operating a medical device, in particular a medical imaging device. The invention also relates to a medical device, in particular a medical imaging device, a computer program, and an electronically readable data carrier.

[0002] With the increasing level of development and the increasing variety of available functions, medical technology devices for examining and / or treating patients become increasingly complex to operate, particularly for users with medical, for example, clinical, training who have not received specialized technical training at the respective medical technology device. This also applies in particular to medical imaging devices, such as magnetic resonance imaging devices, computed tomography devices, and C-arm X-ray devices, whose range of functions and performance is constantly expanding. A particular problem is that, due to the technical nature of these highly complex medical technology devices, it is not easy for users to connect a specific clinical task or use case with the capabilities of the corresponding system.Features of medical technology devices can be overlooked because their usability in a specific situation is not always immediately apparent to the user.

[0003] In this regard, operating steps, particularly workflows with corresponding workflow steps, are usually described in detail in user manuals, and seminars, workshops, and the like are offered to train users of medical technology devices, such as physicians and / or medical technicians, in the operation of specific medical technology devices. However, the problem arises that workflows and operating procedures that have only been learned once through user manuals and / or events / training can be forgotten. For example, users often remember primarily those workflows that they themselves use regularly. In addition, particularly when minimally invasive procedures are performed on a patient with or on the medical technology device, there is usually no time to look up or read up on operating information.

[0004] The invention is therefore based on the object of enabling a simplified, in particular at least partially automated, usability of the scope of services of medical technology devices, in particular with regard to operability and the best possible implementation of a user request.

[0005] This problem is solved by a computer-implemented method, a medical device, a computer program, and an electronically readable data carrier according to the independent patent claims. Advantageous further developments arise from the dependent patent claims.

[0006] A method according to the invention for operating a medical technology device, in particular a medical imaging device, comprises the following steps: - Receiving user input information that includes a natural query input from a user in text form, - applying at least one trained speech analysis function, which comprises at least one first transformer, to the user input information to obtain output data, wherein the output data comprise at least one control measure, in particular at least one control command and / or programming command, and / or the at least one control measure is determined from the output data, and - Providing at least one control measure for controlling the medical device.

[0007] In general, a trained function, including a trained speech analysis function, replicates cognitive functions that people associate with the activity of human brains. Through training based on training data (machine learning), a trained function is able to adapt to new circumstances and detect and extrapolate patterns.

[0008] Generally speaking, parameters of a trained function can be adjusted through training. In particular, supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, and / or active learning can be used. Representation learning (also known as feature learning) can also be used. The parameters of the trained function can be adjusted iteratively through multiple training steps.

[0009] For example, a trained function may include a neural network, a support vector machine (SVM), a decision tree, and / or a Bayesian network, and / or the trained function may be based on k-means clustering, Q-learning, genetic algorithms, and / or mapping rules. In particular, a neural network may be a deep neural network, a convolutional neural network (CNN), or a deep CNN. Furthermore, the neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network (GAN).

[0010] According to the invention, it is therefore proposed to use artificial intelligence in the form of the trained speech analysis function to determine the semantic content of a user's query from the user's free query input and to derive therefrom, directly or indirectly, control measures, in particular control commands and / or programming commands for the medical device, which can be used to support the user and / or particularly preferably to automate processes, in particular also with regard to the operation of the medical device.

[0011] This exploits the fact that artificial intelligence has recently made significant progress, particularly in the field of language analysis, particularly with regard to the extraction of meaning from text passages (semantic coding, particularly as embeddings) and the generation of one's own, derived content (generative models / networks). Many language analyses are based on so-called transformers. Transformers are deep machine learning architectures that utilize so-called attention mechanisms and self-attention mechanisms. They can be used, for example in the form of the first transformer, within the scope of the present invention for the purposes of semantic coding / summarization, for example in embeddings, and / or as translators, for example into a standardized expression or another language, in particular a control language / machine language.

[0012] Transformers also form the basis for modern language models, which can generate their own content, for example, as a string of words. Large language models (LLMs) have recently gained particular popularity. Such LLMs can process very large amounts of text data and utilize AI accelerators. A well-known example is generative pre-trained transformers (GPTs).

[0013] Therefore, particularly advantageous embodiments of the present invention provide that the trained speech analysis function implements a language model, in particular an LLM, or comprises at least one transformer selected from the group comprising BERT, RoBERTa, USE, and SIF, in particular as the first transformer. It should be noted at this point that it is certainly conceivable to use an LLM, for example GPT-4, as a transformer, for example to determine embeddings that would then be extracted as hidden information. However, LLMs were fundamentally developed to generate texts, so the use of transformers is preferred for determining semantic coding / summaries, in particular embeddings. For example, BERT (Bidirectional Encoder Representation from Transformers) or RoBERTa (Robustly Optimized BERT Pretraining Approach) can be used as transformer models.Both variants have demonstrated good performance in various areas of natural language processing (NLP). The use of USE (Universal Sentence Encoder) and / or SIF (Smooth Inverse Frequency) as transformer models is also conceivable. These two transformer types are particularly suitable for generating embeddings from sentences.

[0014] It should be noted here that the input data for language models, and sometimes also for transformers, is often referred to as a "prompt." Language models, especially LLMs, can optionally be configured to accept input data in a format other than text, such as images to which the textual input refers.

[0015] The user's query can be entered acoustically, for example, via a microphone, and / or directly in text form via an input device, such as a keyboard. In the former case, the acoustic recording of the query is converted into text form by a suitable processing tool, such as speech recognition software, as is generally known in the art.

[0016] The control measures are preferably available directly in a control language that can be used to control the medical technology device, in particular in a control device of the medical technology device. The first transformer and / or at least one further transformer can therefore be designed to translate into a control language for the medical technology device. The control language can be a machine language. This means that the control measures can be carried out directly. In this sense, according to the invention, programming and / or coding, in particular a workflow consisting of control commands and / or programming commands for setting an operating parameter, is generated and executed on the basis of the request input in order to at least partially implement the user's wishes described by the request input and / or to at least partially support the user in implementing them.

[0017] Various specific ways of operating the medical device in response to the user's voice input are conceivable, and these ways can also be combined. Particularly advantageously, it can be provided that at least one control measure, in particular a programming command, comprises the setting of an operating parameter, in particular a recording parameter in an imaging device, and / or at least one control measure comprises the triggering of at least one functional process of the medical device, in particular a recording process in an imaging device, and / or at least one control measure comprises an evaluation of medical data recorded with the medical device, in particular image data recorded with the imaging device.Of course, in addition to or as an alternative to the examples of control measures mentioned here, in particular accompanying and / or independent of them, other control measures are also conceivable, for example the output of information and / or support instructions and / or the request for further input from the user.

[0018] The approach can be applied particularly advantageously to medical imaging systems, which are being improved in terms of functionality and performance, but which also increase the complexity of their operation. The imaging system can be a magnetic resonance imaging system and / or a computed tomography system and / or a C-arm X-ray system, in particular an angiography system. Especially in the area of ​​C-arm X-ray systems, the complexity increases both in terms of operability and functionality, and their assignment to clinical questions, especially when three-dimensional imaging methods are provided (e.g., C-arm CT). In addition, C-arm X-ray systems are frequently used in the context of, for example, minimally invasive procedures to monitor the procedure, where a multitude of options exist in a time-critical context.Not only in this case, imaging devices often also have capabilities for evaluating acquired image data, particularly for reconstruction, but also for performing further processing and evaluation procedures. Therefore, when an imaging device is used as a medical device, control measures can also relate to the evaluation of image data, which then constitutes the medical data.

[0019] It may be expedient to incorporate knowledge of recorded image data into the operating procedure, not only to understand the user's query input, but also to enable the specific determination and / or parameterization of control measures. This may mean, on the one hand, that the trained speech analysis function, which may also be generally useful, may include further input data in addition to the user input information. Such input data may, for example, relate to historical data relating to a current examination and / or treatment procedure, for example of a current patient, regarding the operation and use of the medical device, in particular the imaging device. Background information relating to a current examination and / or treatment procedure of a patient may also be used as additional input data, which will be discussed in more detail in concrete embodiments.Such background information can be retrieved, for example, from further user input and / or an information system, in particular a radiology information system (RIS) and / or a hospital information system (HIS) and / or a patient registration system. Another useful source of information is an electronic patient record. Such background information can provide clues to understanding, particularly regarding the context of the query input, and is particularly advantageous when provided in text form, particularly to supplement the textual query input.

[0020] In a particularly advantageous, first concrete development of the present invention, it can be provided that, as an information-determining and / or preparatory control measure to be carried out first, medical technology data obtained with the medical technology device, in particular image data acquired with the imaging device, are evaluated to prepare and / or determine additional information for a subsequent control measure. The core of this first development is therefore to control the medical technology device, which here is preferably designed as an imaging device, in particular a C-arm X-ray device, based on user requests that are further defined by medical technology data, in particular image data. For this purpose, an evaluation of medical technology data, in particular image data, is combined with the trained speech analysis function in order to provide the suitable data for defining subsequent control measures.

[0021] This is always useful when the user's query input relates to a question based on the medical technology data, particularly an image-based task. The trained speech analysis function provides the information-gathering and / or preparatory control action for evaluating the medical technology data as output data, particularly as control commands and / or programming commands, preferably in a control language of the medical technology device, and performs the corresponding evaluation.Ultimately, a combination of an automated evaluation of the medical technology data and the trained speech analysis function, in particular a language model, takes place in order to enable various automated applications which, in the following control commands, can relate in particular to the display / output of the medical technology data and / or measurements in the medical technology data, in particular image data, and / or the recording of further medical technology data.

[0022] Exemplary overall applications, which will be discussed in more detail below, include automatic collimation processes, automatic zoom processes, the cutting out of partial images, the adjustment of a display, and the like. In addition, other overall applications include the automatic performance of measurements in the medical technology data, in particular image data, for example the determination of an FFR value of a stenosis, the size of an aneurysm for the selection of a stent, the detection of certain image features, for example, supply vessels, and the like. Examples of user query inputs that can lead to such evaluation processes with possible subsequent control measures include "zoom on<bestimmtes anatomisches Merkmal> “, “Measure the size of<bestimmtes anatomisches Merkmal> “, “Give the<bestimmtes Score> > out”, “What is the largest blood vessel supplying an AVM” and the like.

[0023] An advantageous embodiment of the evaluation provides that the evaluation comprises a segmentation of the image data and / or the application of a trained evaluation function, in particular a trained segmentation function, to the medical technology data. In this way, for example, certain anatomical features or other features contained in the query input, such as implants and / or medical instruments, can be located in the image data and subsequent control measures can be adapted accordingly. When using trained evaluation functions, a classification of image features, for example as anatomical features or other features, is also possible.For the evaluation of image data, a large number of trained evaluation functions of special architecture already exist in the prior art, which can of course also be used accordingly within the scope of the present invention, also for a subsequent control measure.

[0024] Particularly advantageously, within the scope of the present invention, it can be provided that a plurality, in particular all, of the image features depicted in the image data are segmented by the trained segmentation function. It is also particularly expedient if, based on the segmentation result of a trained segmentation function that segments all image features, in particular anatomical and / or other features, the segmented image features are labeled, in particular with text. This means that each segmented image feature is assigned a text describing which anatomical and / or other feature it is. It is also conceivable to annotate unknown features as such.

[0025] For example, the Segment Anything algorithm described in an article by Alexander Kirillov et al., "Segment Anything," ar-Xiv:2304.02643v1, April 5, 2023, can be used as a trained segmentation algorithm that segments all image features. However, no annotation, i.e., no addition of labels, takes place in this environment. In principle, the trained segmentation function, which segments all anatomical features in particular, delivers sub-images or sub-regions as output that have been segmented as belonging together. An anatomical or other structure to be annotated, for example, an organ, can comprise several such sub-regions, so that a label can be assigned to each of these sub-regions or to the sub-regions as a cluster.

[0026] It is therefore proposed to apply a specially trained classification function to the segmentation result of the trained segmentation function. This makes it possible not only to automatically segment captured images into all relevant sub-regions, especially in real time, but also to annotate them in a standardized manner, i.e., to label them.

[0027] In order to train such a classification function, it can be provided - to apply the trained segmentation function to a number of training image data sets in order to obtain respective segmentation results into sub-areas, - to assign labels to the sub-areas, in particular manually, which describe the respective image features displayed, in particular in text form, and - train the classification function based on the segmentation results as input data and the labels as output data.

[0028] In some embodiments, it is also possible to combine different sub-areas of the segmentation results into clusters manually and / or in a further, particularly unsupervised, learning process, whereby the labels can be assigned to the sub-areas of a cluster as a whole. This simplifies the process, particularly with manual annotation.

[0029] The classification function may, for example, comprise a neural network, in particular a CNN, and / or implement a transformer model to convert the segmentation result into a corresponding labeled segmentation result.

[0030] In the application phase, for example, when determining a corresponding control measure affecting segmentation and annotation, the trained segmentation function, such as "Segment Anything," is applied to the image data. The output data of the trained segmentation function is a segmentation result divided into sub-regions. The trained classification function is applied to the segmentation result, providing annotated, i.e., labeled, sub-regions (and / or clusters, if applicable) as output data.

[0031] As already mentioned, the sub-areas can be assigned anatomical features, such as “aneurysm”, “liver”, “clavista” and the like, as well as other features, such as “stent type A”, “catheter type B” and the like.

[0032] The evaluation of image data, i.e. the combination of a trained segmentation function that delivers a segmentation result defining sub-areas, and a trained classification function that uses the segmentation result as input data to output annotated, i.e. labeled, sub-areas as output data, can also be used effectively on its own, i.e. independently of the application of the trained speech analysis function to user input information to determine control measures. Such automatic segmentation of recorded image data sets in real time, especially during a preferably minimally invasive procedure, into sub-areas and their standardized annotation allows for a variety of advantageous applications, including, for example: - using the knowledge of which medical instruments are visible in the image dataset to adjust image acquisition parameters, - tracking instruments for each type of instrument, - automatic collimation and / or automatic zoom on relevant image parts, and - automatic identification of potential pathologies in a medical image dataset.

[0033] The additional information and / or the processed medical technology data can be used to specify and / or modify the following control measure, in particular as input data for an extended speech analysis.

[0034] In a particularly preferred embodiment, it can be provided that the additional information and / or the processed medical technology data, together with the user input information and / or the output data of the trained speech analysis function and / or secondary data derived from this output data, are used as input data for a trained translator function that determines or adapts at least one secondary control measure, and the at least one secondary control measure is executed. Supplementing the information basis with the additional information and / or the processing of the medical technology data, in particular a processed image data set, thus makes it possible to perform a speech analysis again in a more targeted manner as the subsequent control measure, on the basis of which secondary control measures are determined, which then ultimately technically process the original user request in the user input information.This advantageously provides a two-stage procedure, which in the first stage detects whether additional information or evaluable processing is missing and determines this additional information or carries out the processing, so that in the next, second stage, on the basis of a renewed analysis, the actions actually corresponding to the query input can be determined and carried out.

[0035] The trained translator function can expediently also include and use the first transformer of the trained speech analysis function, on the basis of which the (primary) control measures, in particular in the control language of the medical device, were determined for evaluation and further analysis. In particular, it can even be provided that the trained speech analysis function is used as the trained translator function. This means that one trained function can be sufficient for both analysis processes, whereby the output data can in particular already include the corresponding (primary) control measures or the secondary control measure. It should be noted that the trained speech analysis function in particular can of course also be designed to process the user query of the query input directly in the control measures if no further information or processing is required.

[0036] In embodiments, it may also be conceivable that the trained translator function uses another, further transformer, for transformation, in particular directly into a control language of the medical device.

[0037] In a concrete example for the first specific development, a user may view an image dataset, for example, a live fluoroscopy image, and enter the query: "I want to zoom in on the aneurysm and the parent artery." The trained speech analysis function determines that the location of the aneurysm and the parent artery is unknown, and therefore cannot be zoomed in (or collimated). As a control measure, a segmentation and annotation to be performed is output, and the image dataset is evaluated according to the control measure. In particular, both the aneurysm and the parent artery are located and labeled.In the subsequent control action, the image dataset, the labels, and the user input information are passed as input data to the trained translator function, in particular, the trained language analysis function or at least the same first transformer / LLM, which determines secondary control actions, in particular the primary control actions in a control language for the medical device. Executing the secondary control action zooms in on the image area containing the aneurysm and the parent artery. Optionally, the user can receive a preview beforehand and accept or reject the execution.

[0038] In a particularly advantageous, second concrete development of the present invention, it can be provided that a first text embedding is determined from the user input information by means of the trained speech analysis function, wherein in order to determine the at least one control measure from the first text embedding - a similarity measure is determined between the first text embedding and a plurality of second text embeddings provided in a database, each of which is assigned at least one database control measure for the medical device, and - a second text embedding with one of the highest similarity measures is selected, and - the at least one database control measure associated with the selected second text embedding is used as the at least one control measure.

[0039] In this second specific development, it is proposed to summarize the meaning of the query input in a standardized and compact manner using the first transformer in a first text embedding. The control measures are then determined by comparing this semantic content with second text embeddings to which the control measures are assigned. Text embeddings are understood as numerical and / or vector-based and / or string-like representations of words or texts that describe the semantic meaning. They also form the basis of NLP in other contexts, for example, LLMs. Such text embeddings can be determined, for example, using transformers such as BERT, RoBERTa, USE, and SIF, as described above.

[0040] The ultimately standardized encoding of the semantic content of the query input, performed in this way, enables comparison with correspondingly determined second text embeddings, thus enabling a robust assignment of the user's intention to at least one control measure. Different user expressions and formulations can be converted into comparable or even identical first text embeddings using the trained language analysis function, in particular the first transformer, which enables meaningful comparison.

[0041] To determine the second text embeddings, the trained speech analysis function can be particularly advantageously applied to predefined sections, in particular headings, of a workflow documentation, in particular a user manual. This means that in a preparatory step, second text embeddings are determined by the trained speech analysis function from a description of workflows assigned to the medical device for achieving specific goals and solving specific problems. In particular, a user manual is used here, which typically contains corresponding instructions for the user, i.e. workflows, for specific tasks in order to achieve the desired goal. In particular, it can be provided that the predefined sections are headings, which usually describe the task to be processed.However, more extensive predefined sections are also conceivable, for example, if the task is described in more detail in the workflow documentation. The second text embeddings extracted in this way can then be assigned corresponding sentences from at least one control measure that process the task on the part of the medical device and / or at least partially support the user.

[0042] For example, in a simple embodiment, it is conceivable that, as a control measure assigned to at least part of the second text embeddings, an output device of the medical device is controlled to output the predefined section and / or a workflow description derived therefrom. For example, a corresponding page or a corresponding workflow description of the user manual can be output on a screen and / or display of the medical device.

[0043] In a particularly preferred embodiment, however, it can be provided that from the text sections assigned to and / or encompassed by the predefined sections, which describe at least one workflow step, at least one control measure assigned to the corresponding second text embedding for implementing at least one of the at least one workflow step is determined by means of a trained analysis function comprising a second transformer. This can also be done during preparation, but in exemplary embodiments also "online", i.e. when the corresponding second text embedding has been selected. Thus, an automatic workflow is generated by the trained analysis function using the at least one control measure directly from a text description of the workflow.In this case, the trained analysis function can, for example, step through the workflow documentation associated with the second text embedding, particularly for each workflow step, and determine control measures for each step, particularly in control language for the medical device. In other words, each workflow step described by text in the workflow documentation can be used as input data for the trained analysis function ("prompts"). This automatically enables the derivation of at least one control measure for every second text embedding from workflow documentation, particularly a user manual.

[0044] Preferably, the selection of the second text embedding can be made at least partially based on user input, in particular regarding a list of second text embeddings with the highest similarity measures. This can be particularly useful if a user selection condition is met, which indicates that no single second text embedding with a significantly higher similarity measure than the other second text embeddings could be determined, thus resulting in uncertainty in interpreting the user's intention. If the user selection condition is not met, the second text embedding with the highest similarity is automatically selected, in particular for confirmation by the user or for the direct implementation of the at least one control measure.If this user selection condition is met (or in other embodiments, generally), the semantic content of the second text embeddings with the highest similarity measure to the first text embedding can be output as a selection list, for example, a textual description, in particular with an output or visualization of the similarity measures, and a user can select the desired second text embedding. The second text embeddings in the list can be output, for example, as the headings in the user manual or, in general, the workflow documentation that were used as their basis.

[0045] Furthermore, it can be provided that confirmation of the selected second text embedding is requested from the user, in particular in the case of a selected second text embedding to which several control measures are assigned. This is particularly expedient if an automatic selection has been made, for example due to a very high degree of similarity or a very high distance to the similarity measures of other second text embeddings, in particular therefore due to non-fulfillment of the user selection condition. The user is then informed, on the one hand, of which control measures now follow and, on the other hand, can abort the process by refusing confirmation, for example in the event of a misunderstanding or an input error. If confirmation is refused, it can be provided to output the list of second text embeddings with the highest similarity measures again or nevertheless in order to allow the user to select a correct second text embedding.

[0046] Specifically, it can be provided that the similarity measure is determined by a classifier, in particular a trained classification function, and / or the similarity measure is determined from the group comprising cosine similarity, Euclidean distance, and Jaccard similarity. Cosine similarity is preferred.

[0047] It should be noted in general, as already indicated several times for the second specific further training, that the control measures can comprise an ordered sequence of workflow steps that are at least partially carried out automatically.

[0048] In this context, expedient embodiments can provide that for at least one workflow step, information is requested from the user and used in a subsequent workflow step and / or for at least one workflow step, an automatic actuation of a control element in a user interface takes place, which is visualized and / or explained acoustically to the user. In the first case, necessary information is specifically obtained from the user, for example if determination by evaluating the medical data, in particular image data, is not possible. This can be done in particular using interaction mechanisms already provided in a user interface of the medical technology device. Input and selection windows can be used, which are also used for manual actuation of corresponding control elements in accordance with the workflow documentation.Other input devices and / or input elements can also be used, for example, acoustic input devices such as microphones. The input can also be explained, for example, "Click an endpoint and drag a rectangle" or "Please enter the text for a note on the image."

[0049] It is also conceivable that for at least one workflow step an automatic operation of a control element in a user interface is carried out, which is visualized and / or explained acoustically to the user. In this way, operation of the medical technology device is simplified for the user and at the same time it is possible for the user to follow the automated operating process and perceive the explanation, thus improving the operation for future tasks. For example, a highlighted mouse pointer can be moved to a menu item or other control element and this can be operated with a visible animation. In this variant, operating actions are carried out automatically and are explained particularly for a learning process, for example "clicks" are carried out automatically.

[0050] In embodiments, it can be provided that for at least one control measure, in particular one that fulfills a criticality condition, confirmation from the user is requested before its execution, whereby execution only takes place after confirmation. The criticality condition can, for example, specify whether the control measure has lasting consequences and / or an effect on a patient, in which cases the user can be asked for confirmation before execution for safety reasons. Critical control measures include, for example, those in which an image acquisition is triggered or acquisition parameters, such as X-ray doses, are adjusted, permanent changes are made to medical data, and the like. Confirmation can be provided via an operating means and / or an input means of the medical device, for example by means of an acoustic input via a microphone and / or by actuating a switch, button, or operating lever (joystick).Embodiments are also conceivable in which, in particular in the case of simple confirmation (for example, pushing the operating lever), each control measure or at least each workflow step must be confirmed by the user.

[0051] In an exemplary embodiment of the second specific development, it can be provided, for example, that the trained language analysis function, e.g., configured as or implementing an LLM, is first applied to predefined sections, such as headings, of a user manual and / or other workflow documentation in order to determine the second text embeddings. Then, through further processing of these sources based on the trained analysis function, the workflows described therein can be translated into the control commands associated with the second text embeddings. The system is now ready for use.

[0052] In the application, the user then submits a query, e.g., using their voice, a keyboard, or the like. If not already done, the query input is converted into text and passed as user input information to the trained speech analysis function, which provides the corresponding first text embedding as input data. A semantic search then takes place using the second text embeddings by determining a similarity measure between the first text embedding and the second text embeddings, for example, a cosine similarity as a vector similarity measure that allows for semantic analysis.The pairs with the highest similarity measures, i.e. the smallest distance between the text embeddings, are found and, for example, depending on whether the user selection condition is met, are displayed to the user to select a second text embedding and / or forwarded to the automatic selection of a second text embedding.

[0053] After selecting a second text embedding, the medical device is controlled according to the control measures, in particular by using a control language for the medical device directly. It can be provided that control measures are at least partially confirmed by the user.

[0054] A concrete application example could look like this: The user query is "I want to add a label to my three-dimensional image dataset." Based on the similarity measures of a first text embedding derived from this with second text embeddings from a user manual, the section "Adding annotations to 3D volumes" is found and the corresponding second text embedding is selected. The medical device facility asks the user if they want to execute this workflow. If the user confirms, the medical device facility checks whether a three-dimensional image dataset is loaded and displayed, whereupon the medical device facility automatically clicks a menu icon in the user interface. The "Measurements" tab is then automatically clicked, whereupon the annotation icon is automatically clicked.The medical device then requests user input, namely, "Please click the starting point (head of the arrow) and draw a line to the endpoint." The user input now includes the position of an arrow that indicates the annotated location in the three-dimensional image dataset. This annotation arrow is displayed accordingly. The user is then asked to enter an annotation text, which is displayed as user input associated with the arrow.

[0055] In embodiments, the medical device can now be operated in such a way that it offers context-sensitive suggestions for further workflows, for example in the case just described "Would you like to adjust the object properties?" If the user accepts the suggestion, the corresponding workflow can be carried out, in particular automatically as a result of control measures.

[0056] A third specific development of the invention provides that the trained speech analysis function uses several successive speech analysis models to determine the at least one control measure, wherein in the case of user input information which includes a clinical query text provided by medically trained personnel as query input, - by means of a first language analysis model, in particular a first large language model, a manufacturer-independent linguistic description text of technical measures necessary for the implementation of the clinical request at medical technology facilities in general, in particular with identification of possible technical measures at the medical technology facility to be controlled, is determined as first intermediate data, and - by means of at least one further language analysis model, in particular a further large language model, the output data are determined from the first intermediate data as input data.

[0057] In such variants of the present invention, it is therefore proposed to proceed in several stages to the linguistic analysis of the query input, at least when it is formulated in clinical, in particular medical, terms. If such a clinical user query is submitted, a "translation" into technical terms can first take place. For this purpose, a language model, preferably an LLM, is used in particular to mediate between the user and the medical technology device, for example a C-arm X-ray device. In this multi-stage approach, the clinical query is first translated, comparable to a compiler, into a platform-agnostic "code" or language that uses technical terms, followed by a further translation into a platform-specific "code", i.e. the at least one control measure.

[0058] The first speech analysis model of the trained speech analysis function is therefore independent of the medical device (platform-independent), and therefore also manufacturer-independent. It is trained and correlates clinical use cases with technical terms. It translates the clinical query text into technical terms. Particularly advantageously, the first speech analysis model can also provide information about the optimal equipment required and compare it with the available equipment, especially existing medical devices and their features. For example, the output of the first speech analysis model can include an explanation such as: "A computed tomography device would be optimal for this, but three-dimensional data can also be acquired using the existing C-arm X-ray device."

[0059] At least one further language analysis model then relates to the specific medical technology device to be used (application to several medical technology devices is also conceivable in principle) and is thus platform-specific in order to determine the specific implementation of the identified technical measures, in particular taking into account further information, by means of control measures on the medical technology device.

[0060] This specific third training course further increases the usability of medical technology devices and also enables implicit and lifelong training for users. If the process described here is understood as providing a "copilot" for the user, this will be used more frequently initially, but will result in a steep learning curve, especially with appropriate user interaction, which will be explained in more detail later.

[0061] It should be noted that the methodology can be used generally, even beyond the specific third training course, to monitor the frequency of user requests and, for example, particularly in connection with the first and at least one further speech analysis model, to check which user requests cannot be implemented with the medical device but might represent useful extensions. Based on this, features can then be prioritized and targeted research and development can be carried out.

[0062] Preferably, it can be provided that at least part of the output data of at least one of the language analysis models, in particular the linguistic description text of the first intermediate data, is output to the user for confirmation, wherein a respective further processing condition checks the presence of the confirmation and a subsequent step can only be carried out if confirmation is present. This step also serves to train the user, who becomes more familiar with the technical possibilities over time. A linguistic description text is therefore provided as intermediate data which describes the technical implementation of the task of the clinical query text as well as, optionally, the optimal equipment in comparison to what is available. The user then confirms whether the technical expressions at least essentially solve the clinical query, for example explicitly via an input device or acoustically.The confirmation can also be implicit / partial, especially in the latter case, for example in the form of “Well, that might work.”

[0063] For each of the trained speech analysis models, it may be advantageous to use user feedback information received during the confirmation process to further train the respective speech analysis model. This allows the respective speech analysis model to be continuously improved and optimized.

[0064] In a particularly advantageous embodiment of the invention, the first speech analysis model can be trained with and / or based on scientific articles that link specific medical use cases with technical means. Since the first speech analysis model is intended to correlate clinical queries / use cases with technical measures, scientific articles or publications that address the solution of clinical tasks using technical measures offer an ideal foundation and knowledge base for the first speech analysis model. Clinical use cases are linked to technical features.In connection with information on optimal equipment, in particular for comparison with existing equipment, the first language analysis model may further correlate medical publications and / or information with general information on medical technology facilities, for example, “C-arm X-ray facilities can acquire three-dimensional data.”

[0065] Preferably, based on restriction information provided by the user, at least some of the scientific articles can be excluded from the correlation. Thus, for at least one of the language analysis models, in particular the first language analysis model, the information sources that form the basis of the correlation can be disclosed and / or customizable. For example, a user can select sources from which certain information is to be corrected and / or exclude certain sources, such as publication platforms. For example, it is conceivable to exclude fields of knowledge or application practices not pursued by the user in order to achieve an output of intermediate data that is better tailored to the user's needs.

[0066] As already indicated, the first language analysis model can expediently determine, as part of the intermediate data, requirement information on required medical technology facilities in general for the technical measures. As part of the first language analysis model or downstream of it, the requirement information can be compared with availability information describing available medical technology facilities. The result of the comparison can be output, in particular which available medical technology facility can provide which technical measures. In particular, the first language analysis model can correlate medical publications and / or information with general equipment information in this context.

[0067] Preferably, it can be provided that a second speech analysis model, specifically trained for at least one of the at least one required medical technology device, is used as a further speech analysis model to determine modification data as second intermediate data from input data, which, in addition to at least a part of the first intermediate data, which in particular comprises the at least one required medical technology device, comprises case data describing a clinical case associated with the clinical query text and / or patient data describing a patient associated with the clinical query text, wherein the modification data, in particular as a recommendation text, comprise at least one recommended modification and / or concretization of at least one of the technical measures for the at least one required medical technology device, and / or that a third,a speech analysis model specifically trained for at least one of the at least one required medical device is used to determine the at least one control measure from input data comprising at least the first intermediate data and in particular also the second intermediate data.

[0068] In a particularly preferred embodiment, three trained speech analysis models can be used, which gradually lead to concrete implementation as control measures. The first speech analysis model puts the clinical user query into a technical context, in particular with regard to the required medical technology devices and their equipment. All of this takes place completely independently of the at least one medical technology device actually available and to be used. In the second step, i.e. using the second speech analysis model, the technical description is made more specific, with knowledge of the medical technology device to be used and the concrete clinical case, in particular the patient and the examination orThe treatment goal is achieved by incorporating patient data and / or case data in addition to the first intermediate data, and by training the second speech analysis model specifically for the medical device to be used, whether (preferably) the model or the general genre. In other words, the second speech analysis model preferentially correlates diagnostic information (case data) and information about the patient (patient data), as well as information about the medical device, with the general description of the technical procedure in order to adapt and / or specify technical measures for the specific clinical case. For example, it can be checked whether the patient data and / or case data meet certain criteria.A modification of technical measures may arise, for example, if the patient has intolerances, e.g., with regard to contrast media or materials, and / or if the patient has special characteristics, such as being a child and / or obese. The second language analysis model can suggest specific clarifications and / or adjustments (modifications) in text form based on the second intermediate data, for example, something like: "The patient appears to be obese, therefore, a protocol with an extended field of view should be considered." It may be provided that the user is given the opportunity to reject and / or edit modifications and / or clarifications.

[0069] As a final, third trained language analysis model, one can be used which is again specifically trained for the medical device, here in particular for the model or at least the manufacturer, and which can concretize the technical measures described in the first and second intermediate data as control measures, in particular in a control language of the medical device. In other words, it relates the technical expressions in the first and second intermediate data to the control language specific to the medical device or its manufacturer. At this point at the latest, the at least one first transformer can be used, wherein when the first language analysis model and / or the second language analysis model are configured as LLMs, which is preferred according to the invention, these naturally also implicitly use transformers.

[0070] In a suitable embodiment of this third specific development, each speech analysis model can provide, in addition to its output data, reliability information regarding the output data, which is then output to the user. It is therefore conceivable for the speech analysis models to evaluate themselves with regard to the inference and also make this information available to the user. The reliability information can also be processed in other ways, for example, to determine the extent to which the user must confirm individual processes and / or the extent to which they receive additional support. For example, if reliability information indicates low reliability, potentially helpful features of the medical device can be highlighted, in particular with links to the user manual and / or explanatory videos.The latter support information can also be made available to the user automatically, especially in the context of tax measures. However, if the reliability information indicates a high degree of reliability, it may be sufficient to have at least one tax measure confirmed by a simple action, for example, a simple click.

[0071] A concrete application example for the third specific training could, for example, be based on a query input whose textual form is "I wonder which blood vessel is the main supply to this AVM, the upper or the lower." This could be expressed by the first language analysis model in technical measures to solve the clinical question, for example, as intermediate data: "The user would like to synchronously visualize two time-resolved three-dimensional digital subtraction angiography (time-resolved 3D-DSA or 4D-DSA) volumes at the same time. For better comparability, both volumes should have distinguishable colors. Computed tomography with contrast would be most suitable for this task. However, the available C-arm X-ray system with contrast can provide sufficient diagnostic accuracy."

[0072] If the patient is obese, as described by the patient data, the second language analysis model can add the following second intermediate data: "Since the patient appears to be obese, a large-volume protocol is recommended." Control measures can, for example, if the three-dimensional time-resolved DSA image datasets are already available, include programming commands for acquisition parameters (e.g., large-volume protocol) and / or control commands such as "load the first image dataset, then load the fused second image dataset, apply automatic registration." Of course, control measures, in particular control and / or programming commands, can also be provided for acquisition and the like, just as for the actual visualization.

[0073] In addition to the method, the present invention also relates to a medical device, in particular a medical imaging device, which has a control device designed to carry out the method according to the invention. All statements regarding the method according to the invention can be applied analogously to the medical device according to the invention, and vice versa.

[0074] The medical technology device can have input means, in particular comprising a microphone and / or a keyboard for entering the query input and / or other user inputs, and / or output means, in particular for implementing a user interface.

[0075] The control device can have at least one memory device and at least one processor. Functional units can be implemented using hardware and / or software to carry out steps of the method according to the invention. If the query input is made acoustically, for example, via a microphone, the control device can have a conversion unit to convert the query input into text form as user input information.

[0076] The control device advantageously comprises: - a compilation unit for compiling input data, which includes at least the user information, for the trained speech analysis function, - an application unit for applying the trained speech analysis function to the input data to obtain output data comprising the at least one control measure and / or from which the application unit derives the at least one control measure, and - a control unit for implementing at least one control measure.

[0077] To implement further method steps as described above, additional functional units can of course be provided. In particular, the application unit can have subunits that implement the multiple steps in multi-stage procedures and / or request confirmation from a user and / or receive further information, for example, patient data and / or case data, and the like. The control unit can also have subunits, in particular an evaluation subunit for evaluating the medical technology data and the like.In particular, in the first concrete further training, alternating use of the application unit and the control unit can also occur, for example if the medical technology data is initially evaluated in a first control measure and the application unit is again used in a further control measure to use the trained translator function, which can correspond to the same LLM as the trained language analysis function, and to determine or adapt secondary control measures, which in turn can then be implemented by the control unit.

[0078] The control unit can expediently be used generally to control various components of the medical device, in particular also for other specific uses of the medical device outside of the method described here, for example, via operating means. In an imaging device, the control unit also corresponds in particular to a recording unit that generally controls the recording operation of the imaging device.

[0079] A computer program according to the invention can be loaded directly into a storage means of a control device of a medical technology device and has program means such that, when the computer program is executed on the control device, the device is caused to carry out the steps of a method according to the invention. The computer program can be stored on an electronically readable data carrier according to the invention, which thus comprises control information stored thereon, which comprises at least one computer program according to the invention and is designed such that, when the data carrier is used in a control device of a medical technology device, the device is designed to carry out a method according to the invention. The electronically readable data carrier can, in particular, be a non-transient data carrier, for example a CD-ROM.

[0080] Further advantages and details of the present invention will become apparent from the exemplary embodiments described below and from the drawings. In the drawings: Fig. 1 an embodiment of an artificial neural network, Fig. 2 an embodiment of a Convolutional Neural Network (CNN), Fig. 3 a general flow chart of embodiments of the method according to the invention, Fig. 4 a schedule of a first concrete implementation of the procedure of Fig. 3, Fig. 5 a schedule of a second concrete implementation of the procedure of Fig. 3, Fig. 6 a schedule of a third concrete implementation of the procedure of Fig. 3, Fig. 7 a medical device according to the invention, Fig. 8 the functional structure of a control device of the medical device of the Fig. 7, and Fig. 9 a design of the medical device as a C-arm X-ray device.

[0081] Fig. 1 shows an embodiment of an artificial neural network 1. English terms for the artificial neural network 1 are “artificial neural network”, “neural network”, “artificial neural net” or “neural net”.

[0082] The artificial neural network 1 comprises nodes 6 to 18 and edges 19 to 21, where each edge 19 to 21 is a directed connection from a first node 6 to 18 to a second node 6 to 18. In general, the first node 6 to 18 and the second node 6 to 18 are different nodes 6 to 18, but it is also conceivable that the first node 6 to 18 and the second node 6 to 18 are identical. For example, in Fig. 1 the edge 19 is a directed connection from the node 6 to the node 9 and the edge 21 is a directed connection from the node 16 to the node 18. An edge 19 to 21 from a first node 6 to 18 to a second node 6 to 18 is called an ingoing edge for the second node 6 to 18 and an outgoing edge for the first node 6 to 18.

[0083] In this embodiment, nodes 6 to 18 of artificial neural network 1 can be arranged in layers 2 to 5, wherein the layers can have an intrinsic order introduced by edges 19 to 21 between nodes 6 to 18. In particular, edges 19 to 21 can only be provided between adjacent layers of nodes 6 to 18. In the illustrated embodiment, there is an input layer 2 that only has nodes 6, 7, and 8, each without an incoming edge. Output layer 5 comprises only nodes 17, 18, each without an outgoing edge, with hidden layers 3 and 4 furthermore lying between input layer 2 and output layer 5. In the general case, the number of hidden layers 3, 4 can be chosen arbitrarily.The number of nodes 6, 7, 8 of the input layer 2 usually corresponds to the number of input values ​​to the neural network 1, and the number of nodes 17, 18 in the output layer 5 usually corresponds to the number of output values ​​of the neural network 1.

[0084] In particular, a (real) number can be assigned to nodes 6 to 18 of the neural network 1. Here, x denotes (n) i the value of the i-th node 6 to 18 of the n-th layer 2 to 5. The values ​​of nodes 6, 7, 8 of input layer 2 are equivalent to the input values ​​of neural network 1, while the values ​​of nodes 17, 18 of output layer 5 are equivalent to the output values ​​of neural network 1. In addition, each edge 19, 20, 21 can be assigned a weight in the form of a real number. In particular, the weight is a real number in the interval [-1, 1] or in the interval [0, 1,]. Here, w denotes (m,n) i,jthe weight of the edge between the i-th nodes 6 to 18 of the m-th layer 2 to 5 and the j-th nodes 6 to 18 of the n-th layer 2 to 5. Furthermore, the abbreviation wi,j(n) for the weight wi,j(n,n+1) defined.

[0085] To calculate output values ​​of neural network 1, the input values ​​are propagated through neural network 1. In particular, the values ​​of nodes 6 to 18 of the (n+1)-th layer 2 to 5 can be calculated based on the values ​​of nodes 6 to 18 of the n-th layer 2 to 5 by xj(n+1)=f(∑ixi(n)⋅wi,j(n)).

[0086] Here, f is a transfer function, which can also be referred to as an activation function. Common transfer functions include step functions, sigmoid functions (e.g., the logistic function, the generalized logistic function, the hyperbolic tangent, the arctangent, the error function, the smoothstep function), or rectifier functions. The transfer function is primarily used for normalization purposes.

[0087] Specifically, the values ​​are propagated layer by layer through neural network 1, with values ​​of input layer 2 being given by the input data of neural network 1. Values ​​of the first hidden layer 3 can be calculated based on the values ​​of input layer 2 of neural network 1, values ​​of the second hidden layer 4 can be calculated based on the values ​​in the first hidden layer 3, and so on.

[0088] To the values wi,j(n) for edges 19 to 21, neural network 1 must be trained using training data. In particular, training data includes training input data and training output data, which are referred to below as t i For a training step, neural network 1 is applied to the training input data to determine calculated output data. Specifically, the training output data and the calculated output data comprise a number of values, where the number is determined as the number of nodes 17, 18 of output layer 5.

[0089] In particular, a comparison between the calculated output data and the training output data is used to recursively adjust the weights within the neural network 1 (back propagation algorithm). In particular, the weights can be adjusted according to w'i,j(n)=wi,j(n)−γ⋅δj(n)⋅xi(n) where γ is a learning rate and the numbers δj(n) can be calculated recursively as δj(n)=(∑kδk(n+1)⋅wj,k(n+1))⋅f'(∑ixi(n)⋅wi,j(n)) based on δj(n+1) if the (n+1)-th layer is not the output layer 5, and δj(n)=(xk(n+1)−tj(n+1))⋅f'(∑ixi(n)⋅wi,j(n)) if the (n+1)-th layer is the output layer 5, where f' is the first derivative of the activation function and yj(n+1) is the comparison training value for the j-th node 17, 18 of the output layer 5.

[0090] In the following, with regard to Fig. 2 also provides an example of a convolutional neural network (CNN). Note that the term "layer" is used in a slightly different way than for classical neural networks. For a classical neural network, the term "layer" refers only to the set of nodes that form a layer, i.e., a specific generation of nodes. For a convolutional neural network, the term "layer" is often used to refer to an object that actively modifies data—in other words, a set of nodes of the same generation and either the set of incoming or outgoing edges.

[0091] Fig. 2 shows an embodiment of a convolutional neural network 22. In the illustrated embodiment, the convolutional neural network 22 comprises an input layer 23, a convolutional layer 24, a pooling layer 25, a fully connected layer 26, and an output layer 27. In alternative embodiments, the convolutional neural network 22 may contain multiple convolutional layers 24, multiple pooling layers 25, and multiple fully connected layers 26, as well as other types of layers. The order of the layers can be chosen arbitrarily, with fully connected layers 26 typically forming the last layers before the output layer 27.

[0092] In particular, within a convolutional neural network 22, the nodes 28 to 32 of one of the layers 23 to 27 can be understood as being arranged in a d-dimensional matrix or as a d-dimensional image. In particular, in the two-dimensional case, the value of a node 28 to 32 with the indices i, j in the n-th layer 23 to 27 can be represented as x (n) [i,j]. It should be noted that the arrangement of nodes 28 to 31 of a layer 23 to 27 has no effect on the calculations within the convolutional neural network 22 as such, since these effects are determined exclusively by the structure and weights of the edges.

[0093] A convolutional layer 24 is particularly characterized in that the structure and weights of the incoming edges form a convolution operation based on a specific number of kernels. In particular, the structure and weights of the incoming edges can be chosen such that the values xk(n) the node 29 of the convolutional layer 24 as a convolution xk(n)=Kk*x(n−1) based on the values ​​x (n-1) the node 28 of the previous layer 23 can be determined, where the convolution * in the two-dimensional case can be defined as xk(n)[i,j]=(Kk*x(n−1))[i,j]=∑i'∑j'Kk[i',j']⋅x(n−1)[i−i',j−j'].

[0094] In it, the k-th kernel K ka d-dimensional matrix, in this embodiment a two-dimensional matrix, which is usually small compared to the number of nodes 28 to 32, for example a 3x3 matrix or a 5x5 matrix. In particular, this implies that the weights of the incoming edges are not independent, but are chosen to generate the above convolution equation. In the example for a kernel forming a 3x3 matrix, only nine independent weights exist (where each entry of the kernel matrix corresponds to an independent weight), regardless of the number of nodes 28 to 32 in the corresponding layer 23 to 27. In particular, for a convolutional layer 24, the number of nodes 29 in the convolutional layer 24 is equivalent to the number of nodes 28 in the preceding layer 23 multiplied by the number of convolution kernels.

[0095] If the nodes 28 of the preceding layer 23 are arranged as a d-dimensional matrix, the use of the plurality of kernels can be understood as the addition of a further dimension, also referred to as the depth dimension, so that the nodes 29 of the convolutional layer 24 are arranged as a (d+1)-dimensional matrix. If the nodes 28 of the preceding layer 23 are already arranged as a (d+1)-dimensional matrix with a depth dimension, the use of a plurality of convolutional kernels can be understood as an expansion along the depth dimension, so that the nodes 29 of the convolutional layer 24 are likewise arranged as a (d+1)-dimensional matrix, wherein the size of the (d+1)-dimensional matrix in the depth dimension is larger than in the preceding layer 23 by the factor formed by the number of kernels.

[0096] The advantage of using convolutional layers 24 is that the spatially local correlation of the input data can be exploited by creating a local connection pattern between nodes of adjacent layers, in particular by each node having connections only to a small range of the nodes of the previous layer.

[0097] In the illustrated embodiment, the input layer 23 comprises thirty-six nodes 28 arranged as a two-dimensional 6x6 matrix. The convolutional layer 24 comprises seventy-two nodes 29 arranged as two two-dimensional 6x6 matrices, each of which is the result of convolving the values ​​of the input layer 23 with a convolution kernel. Similarly, the nodes 29 of the convolutional layer 24 can be understood as being arranged in a three-dimensional 6x6x2 matrix, with the latter dimension being the depth dimension.

[0098] A pooling layer 25 is characterized in that the structure and weights of the incoming edges as well as the activation function of its nodes 30 define a pooling operation based on a nonlinear pooling function f. For example, in the two-dimensional case, the values ​​x (n) the node 30 of the pooling layer 25 based on the values ​​x (n+1) the node 29 of the previous layer 24 as x(n)[i,j]=f(x(n−1)[id1,jd2],...,x(n−1)[id1+d1−1, jd2+d2−1]) In other words, by using a pooling layer 25, the number of nodes 29, 30 can be reduced by replacing a number of d1 × d2 neighboring nodes 29 in the preceding layer 24 with a single node 30, which is calculated as a function of the values ​​of said number of neighboring nodes 29. In particular, the pooling function f can be a maximum function, an averaging function, or the L2 norm. In particular, for a pooling layer 25, the weights of the incoming edges can be fixed and not modified by training.

[0099] The advantage of using a pooling layer 25 is that the number of nodes 29, 30 and the number of parameters are reduced. This leads to a reduction in the amount of computation required within the convolutional neural network 22 and thus to a control of overfitting.

[0100] In the illustrated embodiment, pooling layer 25 is a max-pooling layer in which four neighboring nodes are replaced with a single node whose value is the maximum of the values ​​of the four neighboring nodes. Max-pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, max-pooling is applied to each of the two two-dimensional matrices, reducing the number of nodes from seventy-two to eighteen.

[0101] A fully connected layer 26 is characterized in that a majority, in particular all, of the edges are present between the nodes 30 of the previous layer 25 and the nodes 31 of the fully connected layer 26, wherein the weight of each of the edges can be individually adjusted. In this embodiment, the nodes 30 of the previous layer 25 and the fully connected layer 26 are shown both as two-dimensional matrices and as non-connected nodes (shown as a row of nodes, with the number of nodes reduced for clarity). In this embodiment, the number of nodes 31 in the fully connected layer 26 is equal to the number of nodes 30 in the previous layer 25. In alternative embodiments, the number of nodes 30, 31 may be different.

[0102] Furthermore, in this embodiment, the values ​​of the nodes 32 of the output layer 27 are determined by applying the softmax function to the values ​​of the nodes 31 of the preceding layer 26. By applying the softmax function, the sum of the values ​​of all nodes 32 of the output layer 27 is one, and all values ​​of all nodes 32 of the output layer are real numbers between 0 and 1. When the convolutional neural network 22 is used to classify input data, the values ​​of the output layer 27, in particular, can be interpreted as the probability that the input data falls into one of the different classes.

[0103] A convolutional neural network 22 can also have a ReLU layer, where ReLU is an acronym for "rectified linear units." In particular, the number of nodes and the structure of the nodes within a ReLU layer are equivalent to the number of nodes and the structure of the nodes in the previous layer. The value of each node in the ReLU layer can be calculated, in particular, by applying a rectifier function to the value of the corresponding node in the previous layer. Examples of rectifier functions are f(x)=max(0,x), the hyperbolic tangent, or the sigmoid function.

[0104] Convolutional neural networks 22 can be trained, in particular, based on the backpropagation algorithm. To avoid overfitting, regularization methods can be used, such as dropout of individual nodes 28 to 32, stochastic pooling, use of artificial data, weight decay based on the L1 or L2 norm, or maximum norm constraints.

[0105] Fig. 3 shows a general flowchart of embodiments of the method described herein. This method is used to use freely formulated user query inputs in text form to determine specific control measures for a medical device—in the embodiments described here, an imaging device. For this purpose, language analysis techniques of artificial intelligence are used, specifically transformers and preferably language models, in particular LLMs. Modern language models typically employ transformers. For example, BERT, RoBERTa, USE, and / or SIF can generally be used as transformers, while GPT is preferably used as LLM. Alternatives to GPT include, for example, LaMDA or MT-NLG.

[0106] The steps described below are carried out in particular and preferably by a control device of the medical device, whereby external components, for example a voice input module, can also be used for other components (for example a C-arm arrangement).

[0107] In a step S1, user input information is determined or received via an interface, possibly also an internal one. The basis of the user input information is a user's query input, which is formulated linguistically as free text and, if necessary, converted into text form for the user input information. This applies in particular if the query input is received acoustically, for example via a microphone. Corresponding speech recognition algorithms are generally known in the prior art and can also be used within the scope of the present invention. However, it is also possible to select other input methods, for example, to use a keyboard or a similar input device.The query input may already relate to specific technical questions (e.g., "I would like to capture the images in such a way that an instrument is more clearly visible"), but is particularly formulated in clinical terms (e.g., "I would like to be able to better assess whether and in which vessel of the patient calcification is present"). It should also be noted at this point that the present invention has been shown to be particularly well-suited for use in the context of angiographic examinations and / or interventions, so that the medical device configured as an imaging device can, in particular, be an angiography device.

[0108] In a step S2, the user input information is then evaluated using the trained speech analysis function, which comprises at least one first transformer, in order to determine at least one control measure as final output data or derived from the output data. Specific embodiments in this regard are described below with regard to the Fig. 4 to 6 are explained in more detail.

[0109] The control measures that can be determined in step S2 were determined, in particular, in a control language, for example machine language, for the medical device and can accordingly be applied directly in the control device for implementation in a step S3. While some of the control measures can relate to the output or retrieval of information to or from the user, at least one of the at least one control measure preferably relates to the setting of an acquisition parameter in the imaging device and / or the triggering of at least one functional process of the imaging device, in particular an acquisition process, and / or an evaluation of image data acquired with the imaging device.

[0110] It should be noted that in embodiments, it is also possible to return from step S3 to step S2, for example if subsequent control measures are to be specified and / or if secondary control measures are to be determined and then carried out after or as part of the implementation of primary control measures, which can serve, for example, to determine additional information and / or to process medical data.

[0111] The following concrete implementations, which correspond to the three concrete further developments mentioned above, can of course also be used in addition and features can be transferred analogously, where applicable, from one implementation to the other.

[0112] Fig. Figure 4 shows a flowchart of a first concrete implementation of steps S2 and S3, which correspond to the first concrete further development. In this case, a query input is present from which, without additional information or without prior processing of medical technology data to which the query input refers, no concrete automated control measures for implementing the task of the query input can be directly determined. This applies, for example, if the query input refers to features contained in the medical technology data, in this case image data due to the medical technology facility's design as an imaging facility, such as anatomical features or other features, such as displayed instruments and / or implants.

[0113] In a first pass of the language analysis of step S2, in Fig. 4, in a step S2a, control measures are determined using the trained language analysis function. At least one of these control measures, which is to be carried out first, relates to an evaluation of the image data in the control language of the imaging device.

[0114] This at least one control measure to be implemented first is then carried out in the following step S3a. In the present case, in which image features in particular play a role in automatically determining secondary control measures that answer the query input, a trained segmentation function is first applied to the image data of at least one relevant image data set as a trained evaluation function in step S3a, for example, the previously mentioned trained segmentation function "Segment Anything." It should be noted that embodiments are also conceivable in which image data sets from different, mutually registered modalities are evaluated, for example, including those from other modalities.Generally speaking, the image datasets may include two-dimensional fluoroscopy image datasets, two-dimensional DSA image datasets, cone beam CT image datasets, three-dimensional DSA image datasets, computed tomography image datasets, and / or magnetic resonance image datasets.

[0115] The trained segmentation function provides as output a division of the image dataset into sub-regions (sub-images).

[0116] In a step S3b, a trained classification function is then applied to assign to the sub-areas corresponding labels indicating the respective anatomical and / or other feature as text, for example “neurological vascular segment”, “internal carotid artery”, “aneurysm”, “catheter type A”, “stent type A”, “liver” and the like.

[0117] It should be noted at this point that when training the classification function, manual clustering and manual annotation, i.e. manual addition of labels, can be used to generate the ground truth.

[0118] The output of the trained classification function is ultimately a processed image dataset (with the segmented sub-areas and the labels) and / or additional information containing the division into the sub-areas and the associated labels.

[0119] In a step S3c, a subsequent control measure leads back to the general scope of step S2, in that in a step S2b a trained translator function is applied to the user input information as well as the (optionally processed) image data set and / or the additional information. The trained translator function, particularly in the case of an LLM, can also correspond to the trained language analysis function, but at least preferably also uses the first transformer to determine at least one secondary control measure, which ideally is again present in the control language for the imaging device. The trained translator function uses the additional data obtained by automatic evaluation of the image data, in particular the processing and / or the additional information, to determine the secondary control commands in such a way that the user request of the request input, which in particular references image features, can be fulfilled.This may, for example, include adjusting the imaging device's acquisition parameters for subsequent acquisitions and / or adjusting the output of the image data set, such as zooming, highlighting specific image features, and the like. It is also conceivable that the secondary control measures relate to further evaluation of the image data set, such as taking a measurement, determining a score, and the like.

[0120] Optionally, these secondary control measures can be output again for confirmation by the user, whereby the subsequent execution of the secondary control measures in step S3d only takes place after confirmation by the user.

[0121] Fig. Figure 5 shows a second concrete implementation in which text embeddings are used to identify suitable control measures. In step S2c, the first step is to determine a first text embedding of the query input in text form using the first transformer of the trained speech analysis function.

[0122] In preparatory steps, a number of second text embeddings were determined, specifically from a user manual and / or other workflow documentation. The first transformer of the trained language analysis function was applied to predefined sections, in particular at least the headings for parts explaining workflows, for example, chapters. In addition, control measures assigned to these second text embeddings were determined by applying a trained analysis function, which comprises a second transformer, to the text sections describing the at least one workflow step of the at least one workflow. The control measures are in turn determined directly in the control language of the medical technology device, in this case the imaging device. This creates a database with second text embeddings, each of which is assigned at least one control measure.

[0123] In step S2d, a similarity measure is determined between the first text embedding and the second text embeddings—in this case, a cosine similarity. This ultimately corresponds to a semantic search, since the embeddings describe the semantic content.

[0124] In a step S2e, it is now checked whether a user selection condition is met. This may indicate, for example, that no single second text embedding with a significantly higher similarity measure than the other second text embeddings could be determined. If the user selection condition is not met, the second text embedding with the highest similarity is automatically selected in a step S2f, in particular for confirmation by the user or for the direct implementation of the at least one control measure. If confirmation by the user does not occur, the process can optionally (not shown) switch to step S2f, which is described below.

[0125] If the user selection condition is met (or in other embodiments, generally), in a step S2g, the semantic content of the second text embeddings with the highest degree of similarity to the first text embedding is output as a selection list, for example, a textual description, in particular with an output or visualization of the similarity measures. For example, a predetermined number of the second text embeddings with the highest similarity measures can be displayed. A user can select the desired second text embedding from the selection list. In this case, the headings in the user manual (or generally the workflow documentation) are displayed in the selection list.

[0126] In a step S3e, the at least one control measure associated with the selected second text embedding is then executed. This can, in particular, also include, for example, selecting menu items in a user interface in a way that is recognizable to the user and / or requesting further user inputs. In other embodiments, it is also possible to display the workflow, in particular the corresponding portion of the user manual or workflow documentation, instead of or in addition to the actual execution of the workflow.

[0127] It should be noted in general at this point that it is conceivable to have at least some of the control measures to be carried out in step S3 confirmed by the user, in particular when a criticality condition is fulfilled, which is given, for example, when a function is triggered, for example image recording or adjustment of a patient bed.

[0128] When combining the second concrete implementation with the first concrete implementation, it is also conceivable to use text embedding for both the determination of the primary tax measures and the determination of the secondary tax measures, as described here.

[0129] Fig. Figure 6 shows a third concrete implementation, in which the derivation of the control measures from the query input in text form describing the user's intention is carried out step by step. This is particularly useful when clinical or medical expressions are used instead of or alongside technical expressions in the query input. For this purpose, the trained language analysis function is conveniently divided into three trained language analysis models, which are applied successively. At least the first and second language analysis models are appropriately language models, in particular LLMs.

[0130] In step S2h, a first language analysis model (LLM1) is applied. This model was trained independently of manufacturers and specific medical technology facilities and focuses on medical publications that link technical features to specific clinical use cases. It translates the clinical query into technical terms, which are provided in text form as initial intermediate data. It also provides information on optimal equipment as initial intermediate data and compares it with the available medical technology facilities. Optionally, reliability information can also be output in addition to the initial intermediate data.

[0131] The first intermediate data and, if applicable, the reliability information are output to the user in a step S2i, allowing them to confirm that the described technical procedure has solved their clinical task. In addition to simple yes / no input options, it is also conceivable to allow intermediate stages, particularly when receiving the response as speech via a microphone. Intermediate stages result in a type of fuzzy feedback. This feedback information, as well as any additional feedback provided by the user, is used for further training / fine-tuning of the initial speech analysis model.

[0132] After user confirmation and if the technical measures are feasible, once the at least one required medical device, here an imaging device, is available, a second speech analysis model (LLM2) is applied to the first intermediate data together with the patient data describing the patient and the case data describing the clinical case in a step S2j. The second speech analysis model is trained specifically for the imaging device, in particular its model or at least its manufacturer. It checks whether there is a need for clarification or modification that must be considered when determining the control measures. For example, if a patient has metallic implants, the use of low-metal artifact protocols and / or metal artifact correction algorithms may be recommended.Generally speaking, the second language analysis model adds case- and patient-specific specifications and modifications of the technical measures of the first intermediate data in text form, specifically for the imaging device to be used, as second intermediate data.

[0133] The second intermediate data, possibly again with reliability information, is also output to the user for confirmation in step S2k. Here, too, feedback information can be used to improve the trained second speech analysis model.

[0134] After user confirmation, in a step S21, a third language analysis model (LLM3 or Transformer), again specifically trained for the imaging device, is applied to the first intermediate data and the second intermediate data to convert the technical expressions and measures into system-specific language, in particular the control language of the imaging device. In other words, the at least one control measure is determined. Reliability information can also be determined for this purpose.

[0135] In a step S2m, a check is carried out to determine whether the reliability information indicates a sufficiently high degree of reliability that the user's intentions are reflected. Feedback information, particularly fuzzy confirmations, may also be included alternatively or additionally. If the reliability is sufficiently high, the at least one control measure is immediately adopted in step S2n and executed after confirmation by the user in step S3f.

[0136] Otherwise, additional helpful support functions of the imaging device can be highlighted in step S2o, for example, with links to a user manual and / or explanatory videos. This allows the user to obtain additional information before confirming at least one control action.

[0137] It should also be noted that this concrete implementation may also offer the possibility of adapting the information sources of the first language analysis model, for example, excluding certain publication sources.

[0138] Fig. Figure 7 shows a schematic view of a medical device 33 according to the invention, which is embodied here as an imaging device 34. The medical device comprises a control device 35 that controls its operation, is designed to carry out the methods described here and, in addition to imaging components 36, also controls output means 37 provided for forming the user interface, for example, comprising a display, input means 38, for example, comprising a microphone 39 and / or operating means, and other components.

[0139] Fig. Figure 8 shows the functional structure of the control device 35 in more detail. In addition to a storage means 40, in particular for storing the trained speech analysis function as well as other used functions and processed data, the control device initially comprises a conversion unit 41 assigned to the microphone 39, which converts a spoken, acoustic query input into text form. The user input information thus determined can be forwarded via an internal interface to a compilation unit 42 for compiling input data, which includes at least the user information, for the trained speech analysis function (step S1).

[0140] The control device 35 further comprises an application unit 43 for applying the trained speech analysis function to the input data according to step S2 to obtain output data that includes the at least one control measure and / or from which the application unit 43 derives the at least one control measure. Finally, a control unit 44 is provided, which also serves as a recording unit.

[0141] It also serves to implement at least one control measure according to step S3.

[0142] The functional units described here may have subunits for implementation of the substeps described in the concrete implementations, for example an evaluation subunit 45 for carrying out the evaluation according to steps S3a and S3b or other evaluating control measures.

[0143] Fig.Figure 9 shows a specific embodiment of the imaging device 34, a C-arm X-ray device that can be used as an angiography device. The imaging device 34 has a C-arm 46, on which an X-ray emitter 47 and an X-ray detector 48 are arranged opposite each other. The C-arm 46, and thus the resulting recording arrangement, can be moved into various positions relative to a patient table 49. By rotating the C-arm 46 and / or performing other movements, three-dimensional image data sets can also be acquired. Furthermore, fluoroscopy imaging in various projection geometries is possible.

[0144] As an output means 37, a monitor 50 is shown here by way of example, on which recorded image data can also be displayed; the input means 38, in particular including the microphone 39, can be spatially assigned at least in part to this monitor.

[0145] Although the invention has been illustrated and described in detail by the preferred embodiment, the invention is not limited to the disclosed examples and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention.

[0146] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Alexander Kirillov et al., „Segment Anything“, ar-Xiv:2304.02643v1, 5. April 2023

[0025]

Claims

[1] Computer-implemented method for operating a medical device (33), in particular a medical imaging device (34), comprising the following steps: - Receiving user input information that includes a natural query input from a user in text form, - applying at least one trained speech analysis function, which comprises at least one first transformer, to the user input information to obtain output data, wherein the output data comprise at least one control measure, in particular at least one control command and / or programming command, and / or the at least one control measure is determined from the output data, and - Providing at least one control measure for controlling the medical device (33). [2] Method according to claim 1, characterized bythat at least one control measure, in particular a programming command, comprises the setting of an operating parameter, in particular a recording parameter in an imaging device (24), and / or at least one control measure comprises the triggering of at least one functional process of the medical technology device (33), in particular a recording process in an imaging device (24), and / or at least one control measure comprises an evaluation of medical technology data recorded with the medical technology device (33), in particular of image data recorded with the imaging device (34). [3] Method according to claim 1 or 2, characterized bythat medical technology data obtained with the medical technology device (33), in particular image data recorded with the imaging device (34), are evaluated as an information-determining and / or preparatory control measure to be carried out first, for the purpose of preparing and / or determining additional information for a subsequent control measure. [4] Method according to claim 3, characterized by that the evaluation comprises a segmentation of the image data and / or the application of a trained evaluation function, in particular a trained segmentation function, to the medical technology data. [5] Method according to claim 3 or 4, characterized bythat the additional information and / or the processed medical technology data together with the user input information and / or the output data of the trained speech analysis function and / or secondary data derived from this output data are used as input data for a trained translator function which determines or adapts at least one secondary control measure, and the at least one secondary control measure is provided. [6] Method according to claim 5, characterized by that the trained translator function comprises the first and / or a second transformer and / or the trained language analysis function is used as the trained translator function. [7] Method according to one of the preceding claims, characterized bythat a first text embedding is determined from the user input information by means of the trained speech analysis function, wherein in order to determine the at least one control measure from the first text embedding - a similarity measure is determined between the first text embedding and a plurality of second text embeddings provided in a database, each of which is assigned at least one database control measure for the medical technology device (33), and - a second text embedding with one of the highest similarity measures is selected, - the at least one database control measure associated with the selected second text embedding is used as the at least one control measure. [8] Method according to claim 7, characterized bythat to determine the second text embeddings, the trained language analysis function is applied to predefined sections, in particular headings, of a workflow documentation, in particular a user manual. [9] Method according to claim 8, characterized by that from the text sections assigned to and / or comprised by the predefined sections, which describe at least one workflow step, at least one control measure assigned to the corresponding second text embedding for implementing at least one of the at least one workflow step is determined by means of a trained analysis function which comprises a second transformer. [10] Method according to one of the preceding claims, characterized bythat the trained speech analysis function uses several successive speech analysis models to determine the at least one control measure, wherein, in the case of user input information which comprises a clinical query text provided by medically trained personnel, - by means of a first language analysis model, in particular a first large language model, a linguistic description text, independent of the medical technology device (33) to be controlled, of technical measures necessary for implementing the clinical request on medical technology devices (33) in general, in particular with identification of technical measures possible on the medical technology device (33) to be controlled, is determined as first intermediate data, and - by means of at least one further language analysis model, in particular a further large language model, the output data are determined from the first intermediate data as input data. [11] Method according to claim 10, characterized by that at least part of the output data of at least one of the language analysis models, in particular the linguistic description text of the first intermediate data, is output to the user for confirmation, wherein a respective further processing condition checks the presence of the confirmation and a subsequent step is only carried out if confirmation is present. [12] Method according to claim 10 or 11, characterized bythat a second speech analysis model, specifically trained for the medical technology device (33) to be controlled, is used as a further speech analysis model for determining modification data as second intermediate data from input data, which, in addition to at least a part of the first intermediate data, which in particular comprises the medical technology device (33) to be controlled, comprises case data describing a clinical case associated with the clinical query text and / or patient data describing a patient associated with the clinical query text, wherein the modification data, in particular as a recommendation text, comprise at least one recommended modification and / or concretization of at least one of the technical measures for the medical technology device (33) to be controlled, and / or that a third,a speech analysis model specifically trained for the medical device (33) to be controlled is used to determine the at least one control measure from input data which comprises at least the first intermediate data and in particular also the second intermediate data. [13] Medical device (33), in particular imaging device (34), comprising a control device (35) designed to carry out a method according to one of the preceding claims. [14] Computer program which, when executed on a control device (35) of a medical device (33), causes the device (33) to carry out the steps of a method according to one of claims 1 to 12. [15] An electronically readable data carrier on which a computer program according to claim 14 is stored.

Images