System and method for segmenting medical imaging data

Abstract

A system for segmenting medical imaging data, comprising: an input unit configured to receive text data and medical imaging data, wherein the input text data comprises a segmentation task formulated in natural language with respect to the medical imaging data; a text encoder unit configured to generate a text embed based on the received text data; an imaging encoder unit configured to generate an image embed based on the received medical imaging data; a segmentation unit configured to receive the generated text embed and the generated image embed and to determine a segmentation of the received medical imaging data using a function trained by an artificial intelligence algorithm;and an output interface designed to output the segmentation of the medical imaging data.

Description

[0001] The present invention relates to a system for providing segmentation of medical imaging data based on text inputs. Furthermore, the present invention relates to a computer-implemented method for providing the segmentation.

[0002] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included.

[0003] Segmentation of medical data is a fundamental procedure to enable the identification of different tissue areas and lesions from an imaging procedure (e.g. X-ray).

[0004] In addition to issues related to image resolution, such as identifying lesion boundaries, especially in isodense lesions, a common problem is how the different criteria used by different physicians correlate with segmentation. For example, identifying a tissue area containing tumors can typically result in different segmentations depending on the criteria used. From a CT scan of a patient's lung, defining a lesion (whether the solid core of a nodule, a cyst, the primary tumor of a metastasis, or the entire metastatic area) is crucial to obtaining the desired segmentation.

[0005] Traditionally, this problem is solved by specifying predefined segmentation tasks to expand the segmentation possibilities. Recently, methods that apply machine learning have been developed to perform the procedure more efficiently. However, this conventional approach is often inaccurate and error-prone, as predefined segmentation tasks may not accurately represent a physician's criteria in some cases.

[0006] Furthermore, this conventional approach is limited to specific segmentation tasks. To cover numerous use cases, a variety of segmentation methods are required, which is time-consuming and incomprehensive.

[0007] Against this background, it is an object of the present invention to enable an improved and more adaptable segmentation of medical imaging data, which is based in particular on the information of text inputs in natural language.

[0008] This object is achieved according to the invention by a system having the features of claim 1 and / or by a computer-implemented method having the features of claim 13.

[0009] Preferred embodiments of the invention with advantageous features are the subject of the dependent claims.

[0010] According to a first aspect, the invention provides a system for segmenting medical imaging data, comprising: an input unit configured to receive text data and medical imaging data, wherein the input text data comprises a segmentation task formulated in natural language with respect to the medical imaging data; a text encoder unit configured to generate a text embedding based on the received text data; an imaging encoder unit configured to generate an image embedding based on the received medical imaging data; a segmentation unit configured to receive the generated text embedding and the generated image embedding and to determine a segmentation of the received medical imaging data using a function trained by an artificial intelligence algorithm;and an output interface configured to output the segmentation of the medical imaging data;

[0011] Medical imaging data includes image data resulting from an imaging procedure used in medical diagnostics, such as an X-ray, magnetic resonance imaging, computed tomography, ultrasound, and other digital pathology methods.

[0012] Text data is understood to mean that text input is made in natural language, whereby the various segmentation tasks define a connection with medical imaging data.

[0013] The artificial intelligence algorithm used can be any algorithm that is at least partially based on artificial intelligence methods.

[0014] Artificial intelligence is a computer-based unit or device that is designed to implement various data analysis methods, which are also commonly referred to as artificial intelligence, machine learning, deep learning, computational learning or the like.

[0015] Herein and in the following, an artificial neural network is to be understood as a computing system based on a collection of connected units or nodes divided into layers capable of transmitting signals to each other, and which preferably comprises at least in part a transformer network architecture.

[0016] A trained function is a function that is based on an artificial neural network.

[0017] An image embedding is a representation of an image that maps the image onto an image feature vector. A convolutional neural network (CNN) can be used to create the image embedding.

[0018] A text embedding is a representation of a text that maps the text to a text feature vector. A convolutional neural network (CNN) can be used to create the text embedding.

[0019] Embedding facilitates the processing of the information contained in the image, e.g. through machine learning modeling.

[0020] The segmentation unit is to be understood as an image decoder that is set up to generate an image with the segmentation from the image embedding and with the help of the text embedding.

[0021] The various devices and the output interface generally comprise devices and computer programs that can receive, process, and evaluate data. The devices and the output interface are connected to each other via wires and / or wirelessly to exchange signals. Therefore, the devices comprise at least one central processing unit (CPU) and / or at least one graphics processing unit (GPU) and / or at least one field-programmable gate array (FPGA) and / or at least one application-specific integrated circuit (ASIC) and / or any combination of the aforementioned devices. Each device may further comprise a random access memory operatively connected to the at least one CPU and / or a non-volatile memory operatively connected to the at least one CPU and / or the random access memory.Each facility may be implemented partially and / or entirely in a local device and / or partially and / or entirely in a remote system, such as a cloud computing platform.

[0022] The various devices and the output interface may be implemented in hardware and / or software, wired and / or wireless, and any combination thereof. They may further include an interface to an intranet or the Internet, to a cloud computing service, to a remote server, and / or the like.

[0023] In particular, the text encoder unit, the imaging encoder unit and the segmentation unit can execute a software, a program routine, an app or an algorithm with different data processing capabilities.

[0024] The input unit and the output interface can in particular comprise an interface, for example an Application Programming Interface (API), so that the input unit can read in the text data and imaging data and the output interface can output the processed imaging data with the segmentation.

[0025] In cloud computing-based systems, a large number of devices are connected to the cloud computing system via the Internet. The devices may be located in a remote facility connected to the cloud computing system. The devices may include or consist of, for example, devices, sensors, actuators, robots, and / or machines in one or more industrial facilities. The devices may be household appliances, office equipment in a residential / commercial facility, or the like.

[0026] The cloud computing system can enable the remote configuration, monitoring, control, and maintenance of connected devices (also commonly referred to as "assets"). Furthermore, the cloud computing system can facilitate the storage of large amounts of data regularly collected by the devices, the analysis of this large amount of data, and the provision of insights (e.g., performance indicators, outliers, etc.) and alerts to operators, field technicians, or device owners via a graphical user interface (e.g., web applications). The insights and alerts can enable the control and maintenance of the devices, leading to efficient and resilient device operation. The cloud computing system can also enable the modification of parameters associated with the devices.Finally, the cloud computing system can also issue control commands based, for example, on the insights and warnings obtained via the graphical user interface.

[0027] The cloud computing system may comprise a plurality of servers or processors (also referred to as "cloud infrastructure") that are geographically distributed and interconnected via a network. A dedicated platform (hereinafter referred to as a "cloud computing platform") is installed on the servers / processors, providing the aforementioned functions as a service (hereinafter referred to as a "cloud service"). The cloud computing platform may comprise a plurality of software programs running on one or more servers or processors of the cloud computing system to enable the provision of the requested service(s) to the devices and their users.

[0028] One or more APIs are used in the cloud computing system to offer various cloud services to users.

[0029] The invention further provides, according to a second aspect, a computer-implemented method for segmenting medical imaging data, comprising the steps of: (a) receiving text data and medical imaging data, wherein the input text data comprises a segmentation task formulated in natural language with respect to the medical imaging data; (b) generating a text embedding based on the received text data; (c) generating an image embedding based on the received medical imaging data; (d) determining a segmentation of the received medical imaging data based on the generated text embedding and the generated image embedding using a function trained by an artificial intelligence algorithm; and (e) outputting the segmentation of the received medical imaging data.

[0030] The computer-implemented method according to the second aspect of the invention can, for example, be advantageously carried out with the system according to the first aspect of the invention. The features and advantages described herein in connection with the system are therefore also applicable to the method, and vice versa.

[0031] According to a third aspect, the invention provides a computer-implemented method for providing a trained function, comprising the steps of: (i) receiving text data and medical imaging data of a first data set; (ii) receiving segmentation tasks and segmentations of a second data set, wherein the segmentations correspond to the medical imaging data of the first data set; (iii) training a function based on the data of the first and second data sets, wherein training the function comprises fine-tuning the function; and (iv) providing the trained function.

[0032] The trained function of the first and second aspects of the invention can preferably be trained with the computer-implemented training method according to the third aspect of the invention.

[0033] The received text data and segmentation masks for training may be derived from clinical reports and / or synthetically generated based on keywords regarding at least one lesion type, lesion size, lesion growth, and / or lesion margin characteristics.

[0034] Furthermore, the features, advantages or alternative embodiments contained herein in connection with the method according to the third aspect of the invention can be used to improve the system according to the first aspect of the invention and / or the method according to the second aspect of the invention and vice versa.

[0035] According to a fourth aspect, the invention further provides a computer program product comprising executable program codes configured, when executed, to perform the method according to the second aspect and the third aspect of the present invention. It is contemplated that the program codes for implementing the second aspect and / or the third aspect of the invention can be executed independently of one another.

[0036] According to a fifth aspect, the invention provides a non-transitory computer-readable data storage medium comprising executable program codes configured, when executed, to perform the method according to the second aspect and / or the third aspect of the present invention. It is advantageous if the program codes for performing the second aspect and the third aspect of the invention can be executed independently of one another.

[0037] The non-volatile, computer-readable data storage medium may comprise or consist of any type of computer memory, in particular a semiconductor memory such as a solid-state memory. The data storage medium may also comprise or consist of a CD, a DVD, a Blu-ray disc, a USB memory stick, or the like.

[0038] According to a sixth aspect, the invention provides a medical system comprising a medical scanning system and a system for segmenting medical imaging data, wherein the medical scanning system is configured to generate medical imaging data and wherein the segmentation system is configured to receive the generated imaging data.

[0039] One idea underlying the invention is to provide a system for segmenting medical imaging data, where the various segmentation tasks can be defined using natural language processed by a language model. The system creates text-conditioned segmentation of medical imaging data by combining segmentation algorithms with a language model and training them together.

[0040] The system described above advantageously enables the implementation of a computer-implemented method for segmenting medical imaging data. First, text data and imaging data are received. The text data and imaging data are then processed, and a corresponding text embedding and image embedding are generated. Based on this, an artificial intelligence algorithm is used, which is trained to determine a corresponding segmentation. The segmentation can then be output to an interface to provide a visualization of the segmentation.

[0041] An advantage of the present invention is that the segmentation tasks are text-conditioned. Unlike conventional methods that use predefined segmentation tasks, the invention uses segmentation tasks that are recognized by machine learning. Recognition can be improved by training with larger data sets.

[0042] A further advantage of the invention is that refined criteria can be conveyed through natural language-based segmentation. For example, it is possible to generate different segmentations from the same imaging data that correspond to different text inputs. These changes and adjustments to segmentation masks are prompted by natural language, resulting in quick and easy use that could be implemented in the medical workflow. Subtleties that can only be conveyed using natural language can be taken into account with the present invention.

[0043] Advantageous embodiments and further developments emerge from the further subclaims and from the description with reference to the figures of the drawing.

[0044] According to further embodiments of the invention, the segmentation task that can be determined using the trained function includes, for example, the detection of organs, tissue parts, nerve fibers, and lesions. However, the invention is not limited to a list of segmentation tasks, but can also perform general activities for the detection of body parts from an imaging process.

[0045] According to further embodiments of the invention, it is provided that the text encoder unit is designed to map text data in natural language (e.g., ASCII-encoded input) to a first feature vector; and / or the imaging encoder unit is designed to map medical imaging data to a second feature vector; and / or the segmentation unit is designed to map the first feature vector and the second feature vector to a segmentation.

[0046] According to further embodiments of the invention, it is provided that the text encoder unit is designed to map text data in natural language (e.g., ASCII-encoded input) to a first feature vector; and that the imaging encoder unit is designed to map medical imaging data to a second feature vector; and that the segmentation unit is designed to map the first feature vector and the second feature vector to a segmentation.

[0047] According to further embodiments of the invention, the text encoder unit and / or the imaging encoder unit and / or the segmentation unit are embodied as an artificial neural network or are part of a neural network. The text encoder unit can be based on a transformer network structure known from a Large Language Model (LLM), which includes a "self-attention" mechanism. Natural language processing (NLP) models such as BERT or SentenceTransformer can be used. The imaging encoder unit and the segmentation unit can each be a convolutional neural network (CNN).

[0048] According to further embodiments of the invention, it is provided that the artificial intelligence algorithm comprises a language model, in particular a Large Language Model (LLM), in order to capture semantic information from the text embedding.

[0049] A language model, linguistic model, or linguistic data processing model may include software that implements one or more algorithms to extract information through natural language processing (e.g., in the form of text data), which is interpreted by a computer and thus given meaning. The language model may, in particular, be a large language model (LLM), i.e., a large generative language model using artificial intelligence to understand, process, and generate natural language. Large language models are typically trained with large amounts of data (so-called training at scale) and therefore offer optimal performance in natural language recognition.

[0050] According to further embodiments of the invention, the system further comprises a storage unit configured to store the generated segmentation. The storage unit serves as a database for segmentation tasks, in which the various segmentations are stored along with the associated text information and imaging data. The storage unit can further serve to provide traceability of the segmentation tasks in connection with the imaging data.

[0051] According to further embodiments of the invention, the trained function comprises a pre-trained linguistic data processing model or a pre-trained image processing model, and wherein the trained function is fine-tuned based on medical reports. The training of the function can take place simultaneously with a text data set and an imaging data set. However, it can also be advantageous to use pre-trained models, e.g., conventional computer vision models and linguistic data processing models, particularly when imaging data and text data are difficult or very costly to obtain. To improve training, some embodiments provide for the use of a combination of pre-trained models together with text data and imaging data. Some embodiments provide for the trained function to be fine-tuned.Fine-tuning involves incorporating a model that has been intensively trained for one task into a second model to refine it so that the second model can perform a different task. For the system of the invention to learn various segmentation tasks, it must be trained with information that includes, in particular, how the lesions, organs, or tissue parts can be classified, the relevant features, and the best methods for segmentation. This can be achieved by fine-tuning the Large Language Model (LLM) with the latest clinical guidelines or medical journal articles. Fine-tuning is thus done automatically, resulting in fast and efficient training.

[0052] According to further embodiments of the invention, it is provided that the trained function is based at least partially on a neural network, in particular on a transformer model.

[0053] A transformer model or transformer network is a special type of machine learning model widely used in natural language processing (NLP). It is based on the "Transformers" architecture, which is characterized by the use of "attention mechanisms." These mechanisms allow the model to grasp the context and meaning of words in relation to their position in a sentence or text.

[0054] A key feature of the Transformer architecture is the use of encoder and decoder units. The encoder processes the input information and converts it into a series of vectors containing the input's semantic information. The decoder takes these vectors and generates the model's final output. Both the encoder and decoder consist of several layers, each consisting of self-attention and feed-forward neural networks.

[0055] The "self-attention" mechanism is another central aspect of the Transformer architecture. It allows the model to consider the relationships between different words in a text, regardless of their position. This is particularly useful in natural language processing, as the meaning of a word often depends on its context. By taking context into account, the model can perform accurate and nuanced predictions and analyses.

[0056] Transformer networks are capable of recognizing and learning complex patterns in data, making them a powerful tool for many applications in artificial intelligence (AI). By combining encoder and decoder units with self-attention mechanisms, transformer networks can effectively handle a wide range of natural language processing tasks.

[0057] A description of the functionality and a possible architecture of transformer networks can be found in Ashish Vaswani et al., “Attention Is All You Need” arXiv: 1706.03762 (2017).

[0058] According to further embodiments of the invention, the system further comprises a speech recognition interface configured to convert speech inputs into text inputs and output the text inputs to the input unit. The use of such a speech recognition interface has the advantage that the system of the invention can be used without technical knowledge. To generate segmentation more quickly, it may be advantageous to transmit the segmentation instructions verbally. This may also be advantageous if the invention is used as a support during a surgical procedure.

[0059] According to further embodiments of the invention, the output interface is configured to recognize the various segmentation tasks comprising the text data and to generate a separate segmentation of the input imaging data for each of the segmentation tasks. Especially when comparing different criteria, it is advantageous if the various segmentation masks can be viewed simultaneously in order to graphically identify the differences.

[0060] According to further embodiments of the invention, the medical imaging data originate from a computed tomography scan, and the segmentation task is directed toward identifying lung nodules. Lung nodules are lesions that appear as small, round, or oval tissue nodules in the lung. These lesions can be benign but can also indicate metastasis.

[0061] According to further embodiments of the invention, the segmentation task describes which part of the lung nodules is to be segmented. For example, segmentation tasks related to the identification of lung nodules can be the identification of partially solid lung nodules or the identification of the core of the lung nodules.

[0062] Although some functions are described here and below as being performed by units or devices, this does not necessarily mean that these units or devices are provided as self-contained, related, or separate units or devices. In cases where one or more units or devices, or even a part thereof, are provided as software, the units or devices may be implemented by program code sections or program code segments that may be separate from one another, but may also be interwoven or integrated with one another. Program codes that have program code sections or program code segments distributed across multiple units or devices, as is possible, for example, in cloud computing, are also conceivable.

[0063] Likewise, in cases where one or more units and / or devices are provided as hardware, the functions of one or more units or devices may be provided by one and the same hardware component, or the functions of several units or devices may be provided by one and the same hardware component.

[0064] Facilities may be distributed across multiple hardware components, which do not necessarily correspond to the units or facilities. Therefore, any application, system, method, etc., that exhibits all the features and functions attributed to a particular unit or facilities is considered to include or implement that unit or facilities. In particular, it is possible that all units are implemented by program code executed by, for example, a server or a cloud computing platform.

[0065] The above embodiments and further developments can be combined with one another as desired, where appropriate. Further possible embodiments, further developments, and implementations of the invention also include combinations of features of the invention not explicitly mentioned above or described below with regard to the exemplary embodiments. In particular, those skilled in the art will also add individual aspects as improvements or additions to the respective basic form of the present invention.

[0066] The further scope of applicability of the present method and apparatus will become apparent from the following figures, detailed description, and claims. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the invention, are primarily illustrative, and various changes and modifications within the basic spirit and scope of the invention will be apparent to those skilled in the art.

[0067] The invention will now be described with reference to its advantageous embodiments with reference to the following drawings. They serve to further illustrate embodiments of concepts incorporating the claimed invention and to explain various principles and advantages of these embodiments.

[0068] They show: Fig. 1 shows a system for segmenting medical imaging data according to an embodiment of the invention; Fig. 2 is a schematic block diagram illustrating the flow of a computer-implemented method for segmenting medical imaging data according to an embodiment of the invention; Fig. 3 is a schematic flow diagram illustrating the flow of a computer-implemented method for segmenting medical imaging data according to an embodiment of the invention; Fig. 4 is a schematic block diagram illustrating the flow of a computer-implemented method for providing a trained function according to an embodiment of the invention; Fig. 5 is a representation of various segmentation masks in connection with the identification of pulmonary nodules according to an embodiment of the invention; Fig. 6 is a schematic block diagram illustrating a medical system including a medical scanning system and a medical imaging data segmentation system according to an embodiment of the sixth aspect of the present invention; Fig. 7 is a schematic block diagram illustrating a computer program product according to an embodiment of the fourth aspect of the present invention; and Fig. 8 is a schematic block diagram showing a non-transitory computer-readable data storage medium according to an embodiment of the fifth aspect of the present invention.

[0069] Parts of the various drawings that depict identical or functionally similar elements in the different views are designated by the same reference numerals.

[0070] Elements depicted in the drawings are not necessarily drawn to scale. The drawings serve primarily to illustrate and explain the principles and features of the present invention. Similarly, in some instances, well-known structures and devices are shown in block diagram form to illustrate possible concepts of the present invention.

[0071] The numbering of steps in the procedures is intended to facilitate their description. They do not necessarily imply a particular order of steps unless explicitly stated otherwise. In particular, several steps may be performed simultaneously and / or multiple times, and / or there may be intermediate steps not shown.

[0072] The detailed description of the accompanying drawings contains specific details in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details.

[0073] Fig. 1 shows a system 100 for segmenting medical imaging data according to an embodiment of the invention.

[0074] The system 100 includes an input unit 10, a text encoder unit 20, an imaging encoder unit 30, a segmentation unit 50, an output interface 60, a storage unit 70, and a speech recognition interface 80.

[0075] The input unit 10 is configured to receive text data D1 and medical imaging data D2. The imaging data D2 may be transmitted from a medical scanning device, such as an X-ray device, a magnetic resonance imaging device, or a computed tomography device.

[0076] The text data includes one or more segmentation tasks formulated in natural language regarding the medical imaging data D2. The text data D1 can be entered, for example, by a physician, another person familiar with these devices, or another operator via a user interface (human-machine interface) connected to the input unit 10. The input unit 10 can comprise at least a CPU, a data memory, and an interface, for example, an application programming interface (API). The API can be connected to the medical scanning device and / or the user interface wirelessly and / or via a cable.

[0077] The text encoder unit 20 is configured to generate a text embedding from the text data D1 received from the input unit 10. The text encoder unit 20 is therefore a natural language processing unit. The text encoder unit 20 can execute software, an app, or an algorithm with various data processing capabilities. The text encoder unit 20 can be an artificial neural network (e.g., a transformer network) configured to map natural language text data (e.g., ASCII-encoded input) to a first feature vector.

[0078] The imaging encoder unit 30 is configured to generate an image embedding from the medical imaging data received from the input unit 10. The imaging encoder unit 30 may execute software, an app, or an algorithm in the field of computer vision. The imaging encoder unit 30 may be an artificial neural network (e.g., a convolutional neural network) configured to map medical imaging data to a second feature vector.

[0079] The segmentation unit 50 serves to obtain the generated text embedding TE0 and the generated image embedding BE0 and to determine a segmentation of the received medical imaging data D2 using a function trained by an artificial intelligence algorithm. The segmentation comprises at least one segmentation mask. The segmentation is based on the segmentation task formulated in natural language, which comprises the text data D1. For example, segmentation tasks related to the identification of pulmonary nodules can be the identification of partially solid pulmonary nodules or the identification of the core of the pulmonary nodules.

[0080] In preferred embodiments, the artificial intelligence algorithm comprises a language model, in particular a Large Language Model (LLM), to capture semantic information from the text embedding. Such LLMs can be trained with large amounts of data (so-called training at scale) and therefore provide optimal performance in natural language recognition.

[0081] The segmentation unit 50 can be an image decoder configured to generate an image with the segmentation from the image embedding and with the aid of the text embedding. The segmentation unit 50 can be an artificial neural network (e.g., a convolutional neural network) configured to map the first feature vector and the second feature vector to a segmentation.

[0082] Fig. 1 also illustrates an output interface 60 configured to output the segmentation of the medical imaging data. The output interface 60 may include at least one interface, for example, an application programming interface (API). The API may be connected to a user interface wirelessly and / or via a cable. Furthermore, the output interface 60 may include an interface with a display, through which the segmentation masks can be visualized for an operator or user.

[0083] The output interface 60 can further be configured to provide more than one segmentation task for visualization with respect to the same medical imaging data. This is particularly advantageous for clearly visualizing the differences between different segmentation tasks.

[0084] The storage unit 70 is configured to store the generated segmentation. The storage unit 70 serves as a database for segmentation tasks, in which the various segmentations are stored along with the associated text information and imaging data. From there, the various segmentation tasks can be retrieved and used as needed. The storage unit 70 can also serve to provide traceability of the segmentation tasks in connection with the imaging data. The storage unit 70 can be a data storage medium, such as a DVD, a hard drive, a USB memory, or the like. Alternatively, the storage unit 70 can also be part of a computer, a server, and / or be located in a cloud platform.

[0085] In some embodiments, as in Fig. As shown in Figure 1, the system 100 further includes a speech recognition interface 80. The speech recognition interface 80 is configured to convert speech inputs V0 into text inputs and output the text inputs to the input unit 10. The speech recognition interface 80 enables faster operation of the system 100 and offers an alternative to manually operating a keyboard. The speech recognition interface 80 may include at least a CPU, a data memory, and an interface, for example, an application programming interface (API). The API may be coupled to the input unit 10 wirelessly and / or connected via a cable.

[0086] Fig. Figure 2 shows a schematic flow diagram illustrating the sequence of a computer-implemented method for segmenting medical imaging data according to an embodiment of the invention. The method can advantageously be implemented with the system 100 of Fig. 1 should be carried out.

[0087] In a step S1, text data D1 and medical imaging data D2 are received by an input unit 10. The text data D1 can be entered by a user (e.g., using a suitable user interface). The text data describe a segmentation task. Imaging data D2 can be transmitted from a medical scanning device, e.g., an X-ray device or a computed tomography device.

[0088] In step S2, a text embedding is generated. The text embedding corresponds to a representation of the received text data D1 that can be used for post-processing by a language model.

[0089] In a further step S3, an image embedding is generated based on the received medical imaging data. The image embedding corresponds to a representation of the received imaging data D2, which can be post-processed by computer vision software.

[0090] In step S4, a segmentation of the received medical imaging data D2 is determined based on the generated text embedding and the generated image embedding. This determination is performed using an artificial intelligence algorithm, preferably based on a transformer model. The artificial intelligence algorithm is trained to determine a segmentation from the image embedding and the text embedding. This segmentation includes at least one segmentation mask.

[0091] In step S5, the segmentation is output. This can be done via an appropriate interface so that the segmentation mask corresponding to the segmentation task can be visualized. The output segmentation can further include text information that relates the segmentation mask and the segmentation task.

[0092] Fig. 3 shows, for example, a schematic flow diagram illustrating the flow of a computer-implemented method for segmenting medical imaging data according to an embodiment of the invention.

[0093] Text data D1 and imaging data D2 from an area containing a pulmonary nodule L1 are received. Based on the text data D1 (e.g., "Identification of a partially solid pulmonary nodule"), a text embedding TE0 is generated (marked as step S2). In parallel, an image embedding BE0 is generated based on the imaging data D2 (marked as step S3). The image embedding BE0 and the text embedding TE0 are processed by an artificial intelligence algorithm, and based on this processing, a segmentation is determined (step S4). Based on the segmentation, a segmentation mask SM0 representing the edges of a partially solid pulmonary nodule is created.

[0094] In Fig. 3 also shows a second segmentation mask SM1. The second segmentation mask SM1 can be triggered by a new text input (not shown). Based on the new text input (e.g., "identification of the solid core of the pulmonary nodule"), a new text embedding (not shown) is generated in a step S2. A corresponding segmentation is determined, and the segmentation mask SM1 is generated based thereon. According to the invention, the image embedding BE0 does not require any modification or post-processing when processing the segmentation mask SM1. Using this procedure, and in particular with the same image embedding BE0, for example, the segmentation mask SM0 could identify the edges of a partially solid pulmonary nodule, and the segmentation mask SM1 could identify the solid core of the pulmonary nodule.

[0095] Fig. Figure 4 shows a schematic block diagram of the sequence of a computer-implemented method for training a function for determining various segmentation tasks according to one embodiment of the invention. The method can advantageously be implemented with the system 100 of Fig. 1 should be carried out.

[0096] In step T1, text data and medical imaging data are received from a first data set. The received text data and segmentation masks could be generated automatically from clinical reports and / or synthetically based on keywords regarding at least one lesion type, the size of the lesion, the growth of the lesion, and / or the properties of the lesion edges.

[0097] In a further step T2, segmentation tasks and segmentation masks from a second dataset are received, with the segmentation masks corresponding to the medical imaging data from the first dataset. The segmentation masks are annotated based on predefined images or reports.

[0098] In a step T3, a function is trained based on the data of the received first and second data sets. In some embodiments, the function may include a pre-trained language model, e.g., a generative pre-trained transformer (GPT) language model, or a pre-trained image processing model, e.g., SimCLR or SwAV. The function may also include a language model and image processing model jointly pre-trained using Contrastive Language-Image Pre-Training (CLIP). Databases such as ImageNet COCO, GLUE, SQuAD, or PubMed can be used to generate training data sets. This allows the performance of the function to be optimized.

[0099] In some embodiments, the trained function is fine-tuned. Information including: a) how the lesions, organs, or tissue parts can be classified; b) the relevant features; and c) the best methods for segmentation can advantageously be obtained from the latest clinical guidelines or medical journal articles. Fine-tuning is thus performed automatically, resulting in fast and efficient training.

[0100] In a step T4, the trained function is provided. This trained function can be used to perform the computer-implemented method for segmenting medical imaging data according to the second aspect of the invention.

[0101] Fig. Figure 5 shows a representation of various segmentation masks associated with the identification of a pulmonary nodule L1 according to one embodiment of the invention. The imaging data received by system 100 is arranged in the first row and was acquired during a computed tomography scan. These represent (from left to right) the axial, coronal, and sagittal views of a pulmonary nodule. Fig. 5, two segmentation masks SM0, SM1 are identified. The first segmentation mask SM0 can be generated in conjunction with a text input (e.g., "identification of the partially solid pulmonary nodule"). The second segmentation mask SM1 can be generated in conjunction with a text input (e.g., "identification of the solid core of the pulmonary nodule"). According to the invention, the system 100 can thus be prompted using natural language to generate the corresponding segmentation masks SM0, SM1. The system 100 can preferably also display the segmentation masks SM0, SM1 simultaneously so that the differences are clearly visible.

[0102] Fig. 6 shows a schematic block diagram illustrating a medical system 500 including a medical scanning system 200 and a medical imaging data segmentation system 100 according to an embodiment of the sixth aspect of the present invention.

[0103] The medical scanning system 200 is configured to generate imaging data. The medical scanning system 200 may, in particular, be a device that can perform an imaging method in medical diagnostics. The device may, for example, be a magnetic resonance imaging device, an X-ray device, or a computed tomography device.

[0104] The system 100 for segmenting medical imaging data is a system 100 according to the first aspect of the invention, as described, for example, in Fig. 1. System 100 is configured to acquire the imaging data generated by medical scanning system 200. System 100 may be implemented on medical scanning system 200. Additionally or alternatively, it may also be connected thereto wirelessly or by a cable.

[0105] Fig. Figure 7 shows a schematic block diagram illustrating a computer program product 300 according to an embodiment of the third aspect of the present invention. The computer program product 300 comprises executable program codes 350 configured to perform the method according to any embodiment of the second aspect and / or the third aspect of the present invention, in particular as described in the preceding figures. The program codes for performing the second aspect and / or the third aspect of the invention can be executed independently of one another.

[0106] Fig.Figure 8 shows a schematic block diagram illustrating a non-transitory, computer-readable data storage medium 400 according to an embodiment of the fourth aspect of the present invention. The data storage medium 400 comprises executable program codes 450 configured to, when executed, perform the method according to any embodiment of the second aspect and / or the third aspect of the present invention, in particular as described in the preceding figures. The program codes for performing the second aspect and / or the third aspect of the invention can be executed independently of one another.

[0107] The non-volatile, computer-readable data storage medium may comprise or consist of any type of computer memory, in particular a semiconductor memory such as a solid-state memory. The data storage medium may also comprise or consist of a CD, a DVD, a Blu-ray disc, a hard disk, a USB memory stick, or the like. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Ashish Vaswani et al., “Attention Is All You Need” arXiv: 1706.03762 (2017

[0057]

Claims

[1] System (100) for segmenting medical imaging data (D2), comprising: an input unit (10) configured to receive text data (D1) and medical imaging data (D2), wherein the input text data (D1) comprises a segmentation task formulated in natural language with respect to the medical imaging data (D2); a text encoder unit (20) configured to generate a text embedding (TE0) based on the received text data (D1); an imaging encoder unit (30) configured to generate an image embedding (BE0) based on the received medical imaging data (D2); a segmentation unit (50) configured to receive the generated text embedding (TE0) and the generated image embedding (BE0) and to determine a segmentation of the received medical imaging data (D2) by means of a function trained by an artificial intelligence algorithm; and an output interface (60) configured to output the segmentation of the medical imaging data (D2). [2] System according to claim 1, wherein the segmentation task which can be determined with the trained function comprises in particular a recognition of organs, tissue parts, nerve fibres and lesions. [3] System according to one of the preceding claims, wherein: the text encoder unit (20) is designed to map text data (D1) in natural language to a first feature vector; and / or the imaging encoder unit (30) is designed to map medical imaging data (D2) to a second feature vector; and / or the segmentation unit (50) is designed to map the first feature vector and the second feature vector to a segmentation. [4] System according to one of the preceding claims, wherein the text encoder unit (20) and / or the imaging encoder unit (30) and / or the segmentation unit (50) are designed as an artificial neural network or are part of a neural network. [5] System according to one of the preceding claims, wherein the artificial intelligence algorithm comprises a language model, in particular a large language model, to capture semantic information from the text embedding (TE0). [6] System according to one of the preceding claims, further comprising a storage unit (70) arranged to store the generated segmentation. [7] A system according to any one of the preceding claims, wherein the trained function comprises a pre-trained linguistic data processing model or a pre-trained image processing model, and wherein the trained function is fine-tuned based on medical reports. [8] System according to one of the preceding claims, wherein the trained function is based on a neural network, in particular on a transformer model. [9] System according to one of the preceding claims, further comprising a speech recognition interface (80) which is configured to convert speech inputs (V0) into text inputs (D1) and to output the converted text inputs (D1) to the input unit (10). [10] The system of any preceding claim, wherein the output interface (50) is further configured to recognize the various segmentation tasks comprising the text data (D1) and to generate a separate segmentation of the input imaging data (D2) for each of the segmentation tasks. [11] System according to one of the preceding claims, wherein the medical imaging data (D2) originate from a computed tomography and the segmentation task is directed towards identifying pulmonary nodules (L1). [12] The system of claim 11, wherein the segmentation task describes which part of lung nodules (L1) is to be segmented. [13] Computer-implemented method for segmenting medical imaging data, comprising the steps: Receiving (S1) text data (D1) and medical imaging data (D2), wherein the input text data (D1) comprises a segmentation task formulated in natural language with respect to the medical imaging data (D2); Generating (S2) a text embedding (TE0) based on the received text data (D1); Generating (S3) an image embedding (BE0) based on the received medical imaging data (D2); Determining (S4) a segmentation of the received medical imaging data (D2) based on the generated text embedding (TE0) and the generated image embedding (BE0) by means of a function trained by an artificial intelligence algorithm; and Outputting (S5) the segmentation of the received medical imaging data (D2). [14] Computer-implemented method for providing a trained function, comprising the steps: Receiving (T1) text data and medical imaging data of a first data set; Receiving (T2) segmentation tasks and segmentation masks of a second data set, wherein the segmentation masks correspond to the medical imaging data of the first data set; Training (T3) a function based on the data of the received first and second data sets, wherein the training (T3) of the function comprises fine-tuning the function; and Providing (T4) the trained function. [15] The computer-implemented method of claim 14, wherein the received text data and segmentation masks are automatically generated from clinical reports and / or synthetically generated based on keywords regarding at least one lesion type, the size of the lesion, the growth of the lesion and / or the properties of the lesion edges. [16] Medical System (500), with a medical scanning system (200), wherein the medical scanning system (200) is configured to generate medical imaging data (D2); and with a system (100) for segmenting medical imaging data, wherein the segmentation system (100) is configured to receive the generated imaging data (D2). [17] Computer program product (300) comprising executable program codes (350) which, when executed, are designed to carry out the computer-implemented method according to claim 13 and / or the computer-implemented method according to claim 14. [18] Non-transitory computer-readable data storage medium (400) comprising executable program codes (450) adapted, when executed, to perform the computer-implemented method according to claim 13 and / or the computer-implemented method according to claim 14.

Images