Medical image processing

Abstract

And processing the medical image. A method for segmentation of anomalies. Medical image data representing a medical image is received at a first ML system. The first ML system generates a plurality of image embedding vectors corresponding to a respective plurality of medical image features based on the received medical image data, each of the plurality of image embedding vectors relating to a different respective medical image feature, and includes medical image feature data indicating the presence or absence of the respective medical image feature at each of a plurality of locations in the medical image. An indication of a first medical image feature included in the medical image is received at a second ML system. The second ML system generates a feature vector based on the indication. A comparison of the feature vector to a plurality of image embedding vectors is performed, and a first image embedding vector is identified based on the comparison.

Description

Technical Field

[0001] The invention relates to a method, an apparatus and a computer program for segmenting medical image data. Background Art

[0002] Medical images can be segmented to identify regions of interest, such as organs or medical abnormalities. Segmentation can, among other things, allow quantitative data to be obtained from medical images (e.g., the size of specific medical features), enable accurate planning of radiation therapy, and enable identification of features that may not have been noticed by a medical professional, such as medical abnormalities.

[0003] It would be desirable to provide automated methods for processing medical images to identify such features and their locations in the medical images for purposes such as segmentation. Summary of the invention

[0004] According to a first aspect of the present invention, a computer-implemented method for processing medical image data is provided, the method comprising: receiving medical image data representing a medical image at a first machine learning-based system; generating, by the first machine learning-based system, a plurality of image embedding vectors corresponding to a plurality of corresponding medical image features based on the received medical image data, each of the plurality of image embedding vectors being associated with a different corresponding medical image feature and comprising medical image feature data indicating the presence or absence of the corresponding medical image feature at each of a plurality of positions in the medical image; receiving, by a second machine learning-based system, an indication of a first medical image feature included in the medical image, and generating, by the second machine learning-based system, a feature vector based on the indication; and performing a comparison of the feature vector with the plurality of image embedding vectors, and identifying a first image embedding vector from among the plurality of image embedding vectors based on the comparison.

[0005] Optionally, a first image embedding vector is identified as having a highest similarity with the feature vector from among the plurality of image embedding vectors.

[0006] Optionally, the method comprises determining position data based on the identified first image embedding vector, the position data representing a position in the medical image of the indicated first medical image feature in the medical image data.

[0007] Optionally, the medical image feature data represents a segmentation map of the corresponding medical image feature, and the determined position data comprises the segmentation map represented by the identified first image embedding vector.

[0008] Optionally, the segmentation map indicates a probability that a corresponding medical image feature exists at a corresponding position in the medical image.

[0009] Optionally, the method comprises: determining whether the medical image represents a medical abnormality, and in response to determining that the medical image represents a medical abnormality, displaying the segmentation map on a display device.

[0010] Optionally, the indication received at the second machine learning based system includes region data representing a region of the first medical image feature in the medical image, and the comparison of the feature vector with the plurality of medical image embedding vectors is performed based at least in part on the region data.

[0011] Optionally, the indication received at the second machine learning based system includes data representing a textual prompt indicating a feature of the first medical image.

[0012] Optionally, the method includes performing a training process to train a first machine learning-based system and a second machine learning-based system, the training process including: inputting first training data including multiple medical image data sets into the first machine learning-based system to generate multiple trial image embedding vectors for each medical image data set; and inputting second training data into the second machine learning-based system to generate a trial feature vector for each medical image data set, wherein the second training data includes data representing multiple medical reports, each medical report includes data indicating the presence of one of the multiple medical image features in a corresponding one of the medical image data sets, and each medical report includes data representing an area of ​​the feature indicated as existing in the medical image represented by the corresponding medical image data set, wherein the training process includes jointly training the first machine learning-based system and the second machine learning-based system to minimize the loss function between the trial image embedding vector and the corresponding trial feature vector.

[0013] Optionally, the method comprises inputting each of the medical reports into a natural language processing system to generate a data set representing findings of the medical reports, wherein the data representing the plurality of medical reports is data representing the findings of the medical reports.

[0014] Optionally, the method includes inputting at least one of the plurality of image embedding vectors into a third machine learning-based system to generate natural language text describing findings associated with the medical image data.

[0015] Optionally, the method includes executing a training method to train a third machine learning-based system, the training method comprising: inputting an image embedding vector generated by the first machine learning-based system based on an input medical image dataset into the third machine learning-based system to generate, for each input medical image dataset, experimental natural language text describing findings associated with the input medical image dataset; and training the third machine learning-based system to minimize a loss function between the experimental natural language text and data representing a medical report corresponding to the input medical imaging dataset.

[0016] Optionally, at least one of the plurality of medical image features comprises a medical abnormality.

[0017] According to a second aspect of the present invention, there is provided an apparatus configured to perform the method according to the first aspect.

[0018] According to a third aspect of the present invention, there is provided a computer program, which, when executed by a computer, causes the computer to perform the method according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart schematically illustrating a method for processing medical image data;

[0020] Figure 2 It is a schematic diagram Figure 1 A flowchart of a specific example of the method illustrated in;

[0021] Figure 3 is a flow chart schematically illustrating an example method used by the image encoder 204 to generate a plurality of image embedding vectors;

[0022] Figure 4 is a schematic diagram for training a machine learning based system to perform Figure 1 A flow chart of the method illustrated in the method;

[0023] Figure 5 is a diagram schematically illustrating an apparatus according to an example. DETAILED DESCRIPTION

[0024] Independent of the use of the grammatical terms, individuals having both masculine and feminine identities are also included within the terms.

[0025] Figure 1 A flow chart of a computer-implemented method for processing medical image data 202 is shown.

[0026] In summary, the method comprises:

[0027] - In step 102, medical image data 202 representing a medical image 202 is received at a first machine learning based system;

[0028] - in step 104, the first machine learning-based system generates a plurality of image embedding vectors 206 corresponding to a corresponding plurality of medical image features based on the received medical image data 202, each of the plurality of image embedding vectors 206 being associated with a different corresponding medical image feature and comprising medical image feature data indicating the presence or absence of the corresponding medical image feature at each of a plurality of locations in the medical image 202;

[0029] - in step 106, receiving at the second machine learning based system an indication of a first medical image feature included in the medical image 202, and generating, by the second machine learning based system, a feature vector 212 based on the indication; and

[0030] - In step 108, a comparison of the feature vector 212 with the plurality of image embedding vectors 206 is performed and a first image embedding vector is identified from among the plurality of image embedding vectors 206 based on the comparison.

[0031] Thus, the method enables the generation and identification of medical image feature data indicating the presence or absence of a corresponding medical image feature at each of a plurality of locations in the medical image. Comparison of the feature vector with the plurality of image embedding vectors allows the generation of an image embedding vector, the content of which is associated with the medical image feature indicated at step 106.

[0032] The data included by the identified image embedding vector indicating the presence or absence of the corresponding medical image feature at the location can be used in a variety of ways. As described below, a segmentation map indicating the probability of the indicated medical feature being present at each pixel in the medical image can be generated, and this can also be used by medical professionals for the purposes described in the background section. The identified image embedding vector can be used to determine whether the medical image represents a medical abnormality. Additionally, the identified image embedding vector can be used to generate text describing the findings of the medical image. Further, by generating image embedding vectors for multiple patients with corresponding medical abnormalities and observing clustering patterns in the image embedding vectors, the corresponding medical abnormalities can be classified.

[0033] Reference now Figure 2 The example method 100a is described in detail. Figure 2 Method 100a is Figure 1 A specific example of method 100 includes Figure 1 The method 100 comprises steps.

[0034] As mentioned, method 100 includes, in step 102, receiving medical image data 202 at a first machine learning based system. In method 100a, the first machine learning based system is an image encoder 204. Step 102 may involve receiving medical image data 202 from a memory such as a memory (see, e.g., described in more detail below). Figure 5 The medical image data 202 is retrieved from a storage device (memory 504 in).

[0035] The medical image data 202 may include a 3D array of elements, each element having a value, which collectively represent a 3D medical image. The elements may be voxels, each voxel having at least one value. The at least one value may represent an output signal of a medical imaging technique used to generate the medical image data 202. For example, for magnetic resonance imaging, the value of an element (e.g., a voxel) may represent the rate at which an excited nucleus returns to a state of equilibrium in a region corresponding to the element. As another example, in SPECT imaging, the value of an element may represent the blood flow in a capillary represented by a given voxel. As another example, in CT imaging, the value of an element may correspond to or represent the amount of X-ray attenuation. In some examples, each element may have only one value. However, in other examples, each element may have multiple values ​​or be otherwise associated with multiple values. For example, multiple values ​​of a given element may represent values ​​of corresponding multiple signal channels. For example, each signal channel may represent a different medical image signal or property of an imaged subject. In some examples, the at least one value may include an element (e.g., a voxel) intensity value. For example, an output signal from medical imaging may be mapped to a voxel intensity value, such as a value within a defined range of intensity values. For example, for a grayscale image, the intensity value may correspond to a value in the range of 0 to 255, e.g., where 0 represents a "black" voxel and 255 represents a "white" voxel. As another example, such as in the case of USHORT medical image data, the intensity value may correspond to a value in the range of 0 to 65536. As another example, in a color image (e.g., where different colors represent different properties of the imaged subject), each pixel may have three intensity values, e.g., one for each red, green, and blue channel. It will be appreciated that other values ​​may be used. Although the methods described herein primarily use 3D medical images, it will be appreciated that they may also be applied to 2D medical images.

[0036] As mentioned, at step 104 , the method 100 includes the image encoder 204 generating a plurality of image embedding vectors 206 corresponding to a respective plurality of medical image features based on the received medical image data 202 .

[0037] The method 100a includes inputting the medical image data 202 into an image encoder 204 to generate an image embedding vector 206. In the method 100a, each of the plurality of medical image embedding vectors 206 is associated with a different medical image feature. In the method 100a, the medical image features are different medical abnormalities. The medical abnormalities in the method 100a include ground glass density shadows and lesions. However, in some examples, the plurality of medical image embedding vectors 206 include two image embedding vectors, one associated with normal tissue and one associated with abnormal tissue. Further, in some examples, the medical image features relate to different components of the body that are not medical abnormalities. For example, the image embedding vector may relate to the kidney, the pancreas, or the ribs. As described below with reference to Figure 4 As described above, the medical image features that the image embedding vector 206 focuses on depend on the content of the training data used to train the image encoder 204. Figure 3 Describes an example method used by an image encoder to generate an image embedding vector.

[0038] Each of the plurality of medical image embedding vectors 206 includes medical image feature data indicating the presence or absence of a corresponding medical image feature at each of a plurality of locations in the medical image 202. In the method 100a, the feature data represents a segmentation map 218 of the medical image feature to which the image embedding vector relates, wherein the segmentation map 218 indicates a probability that the corresponding medical image feature exists at a corresponding location in the medical image 202. In the method 100a, each corresponding location is a portion 302 of the medical image 202. The first image embedding vector may represent, for example, a segmentation (probability) map of a lesion. If no lesion is present in the medical image 202, the segmentation map 218 will likely show a probability of 0 for each location in the image. However, in this example, the subject depicted in the medical image 202 has ground-glass opacity, and thus the segmentation map 218 of the image embedding vector for ground-glass opacity may show a non-zero probability for locations having ground-glass opacity. In method 100a, the entries of the given medical image embedding vector are the above-mentioned probabilities, and each component in the given medical image embedding vector corresponds to a different one of the image parts 302. However, in some examples, the medical image feature data is binary data indicating the presence or absence of a medical feature at each of the image parts. "1" may indicate that the feature exists, while "0" indicates that the feature does not exist.

[0039] As mentioned, at step 106 , the method 100 includes receiving, at the second machine learning based system, an indication of a first medical image feature included in the medical image 202 , and generating, by the second machine learning based system, a feature vector 212 based on the indication.

[0040] In method 100a, the indication of the first medical image feature is a text prompt 208 indicating the first medical image feature. For example, a user such as a radiologist may input text such as "ground glass density" to the apparatus 500 including the second machine learning-based system through an input interface 506. Alternatively, the user may select the first medical image feature from a drop-down box listing a plurality of medical image features.

[0041] In method 100a, the indication includes region data representing a region of the first medical image feature in the medical image 202. For example, the radiologist may input text such as "in the right upper lobe" through the input interface 506. A text prompt 208 may include both the region and the first medical image feature; for example, the text prompt 208 may be "ground glass density shadow in the right upper lobe". Alternatively, the region data may be input when the radiologist clicks on a specific region of a template image of a human body (or a portion thereof) stored by the apparatus 500 and displayed by the display device 510; for example, the apparatus 500 may store a mapping between regions in the template image and names of corresponding regions (such as "right upper lobe"), and retrieve the name of the clicked region and use it as the text prompt 208.

[0042] In method 100a, the second machine learning-based system is a text encoder 210. In this example, the text encoder 210 generates a feature vector 212 representing the text "ground glass density shadow in the right upper lobe". The text encoder 210 may include a recurrent neural network that generates the feature vector 212. The feature vector 212 may have the same number of components as each of the image embedding vectors 206. The training of the text encoder 210 is described in detail below.

[0043] As mentioned, at step 108 , method 100 includes performing a comparison of feature vector 212 with plurality of image embedding vectors 206 , and identifying a first image embedding vector from among the plurality of image embedding vectors 206 based on the comparison.

[0044] In method 100a, apparatus 500 incorporating text encoder 210 and image encoder 204 generates, for each image embedding vector, a similarity measure indicating the similarity between the image embedding vector and feature vector 212. The identified (first) image embedding vector is selected as the image embedding vector identified as having the highest similarity measure with the feature vector 212 of all image embedding vectors 206.

[0045] In method 100a, the image embedding vector 206 and the feature vector 212 are represented in a common embedding space, and the similarity measure is a distance measure in the common embedding space between the image embedding vector and the feature vector 212. Any distance measure described herein may be, for example, a Euclidean distance or a cosine distance. By representing the image embedding vector 206 and the feature vector 212 in a common embedding space, the distance measure can be used to identify the image embedding vector whose content is most closely aligned with the feature vector 212. This enables the identification of the image embedding vector that represents the features of the medical image described by the textual prompt 208.

[0046] The comparison is based on region data representing the region, since the feature vector 212 encodes information related to the region of the medical image feature indicated at step 106. In addition to the medical image feature, the indication of the region provides further information on which to base the comparison, which means that the image embedding vector identified by the comparison is more likely to belong to the medical image feature that the medical professional desires to segment.

[0047] The method 100 a includes determining location data representing a location of a medical image feature in the medical image data 202 in the medical image 202 based on the identified image embedding vector 214 .

[0048] The position data may include a segmentation map 218 represented by the identified image embedding vector 214. The segmentation map 218 indicates the probability that a medical image feature exists at a corresponding position in the medical image 202. In this example, the corresponding position is the position of each pixel in the medical image 202.

[0049] In this example, the medical image feature data involved in step 104 includes the probability of the medical image feature existing at each of a plurality of locations in the medical image 202. As described above, each of these locations is a portion of the medical image 202. Each portion of the image includes a plurality of voxels.

[0050] In this example, the medical image feature data is interpolated by the image upscaling device 216 to provide a probability of the presence of the corresponding feature at each voxel of the medical image 202. The interpolation may include, for example, trilinear interpolation. For example, the probability of the presence of the corresponding feature at a given voxel may be calculated by trilinear interpolation of the probability of the presence of the corresponding feature at each of the eight portions of the image closest to the voxel. Thus, the low-dimensional representation of the probability map may be upscaled to provide a full probability map of the medical image 202.

[0051] Indicating a probability of presence, rather than just a binary indication of whether a feature is present or not, means that the segmentation map 218 may show fuzzy boundaries of a medical abnormality.

[0052] The method 100a includes displaying the segmentation map 218 on a display device 510. The segmentation map 218 may be displayed on top of the medical image 202 to indicate areas of the medical image 202 where abnormalities are present. The display device 510 may be a computer monitor or other display screen of a computer, and may be coupled to a processor 502 of the apparatus 500. A medical professional reviewing the segmentation map 218 displayed on the display device 510 may, for example, perform a qualitative assessment of the abnormality using the segmentation map 218 and decide on a course of treatment based on the assessment.

[0053] The location data may additionally or alternatively include natural language text describing a finding 222 associated with the medical image data 202. The method 100a includes inputting at least one of the plurality of image embedding vectors 206, and optionally all of the plurality of image embedding vectors 206, into a text decoder 220, also referred to as a fourth machine learning-based system, to generate the natural language text. The natural language text may include a finding 222, such as "ground glass density shadows are present in the upper lobe." For example, the text decoder 220 may include a recurrent neural network. The training of the text decoder 210 is described below with reference to Figure 4 Described in further detail.

[0054] Medical professionals can use natural language text, for example, to diagnose a medical condition or recommend treatment. In any case, by generating such natural language text, a radiologist or other medical professional can understand the patient condition as represented by the image embedding vector 206 by reading the natural language text.

[0055] Method 100a includes performing a clustering process on a plurality of selected image embedding vectors determined from different medical image datasets. For example, k-means clustering can be used to determine which of a plurality of clusters a particular image embedding vector belongs to. In this way, it can be determined whether two image embedding vectors representing different medical image datasets represent the same medical abnormality.

[0056] The method 100a includes determining whether the medical image 202 represents a medical abnormality using a classifier 224. In one example, the classifier 224 is a large language model, such as ChatGPT, to which the generated natural language text is input, along with a text prompt such as "Does this represent a medical abnormality? Please answer yes or no." In another example, the generated natural language text is input to a natural language processing system, which compares each word in the natural language text describing the findings to a list of words related to medical abnormalities (such as lesions), and outputs an indication (e.g., a binary output) that the medical image 202 represents an abnormality if and only if the natural language text includes a word in the list.

[0057] In another example, the image embedding vectors 206 are all input to a fourth machine learning-based system to determine whether the medical image 202 represents a medical abnormality. The fourth machine learning-based system can be trained using supervised learning, for example, by inputting the image embedding vectors generated by the trained image encoder 204 into the fourth machine learning-based system to generate a trial classification of whether the medical image 202 represents a medical abnormality. Supervised learning includes minimizing a loss function between the trial classification and a ground truth data representing whether the corresponding medical image used to generate the image embedding vector represents a medical abnormality.

[0058] The output of the classifier 224 can be used to decide whether any further processing of the medical image 202 is required. In the event that the output of the classifier 224 indicates that the medical image 202 does not represent an abnormality, the medical image 202 can be removed from the radiology workflow. For example, in response to determining that the medical image 202 represents a medical abnormality, the processor can display the segmentation map 218 on the display device 510, or perform other processing of the medical image 202. In this example, if it is determined that the medical image 202 does not represent a medical abnormality, the processor 502 does not display the segmentation map 218 on the display device 510. The processor 502 can indicate on the display device 510 that the medical image 202 does not represent a medical abnormality in other ways, such as using text such as "No abnormality detected in the image."

[0059] By determining whether the medical image 202 represents a medical abnormality and performing further processing in response to determining that the medical image 202 represents an abnormality, the efficiency of analysis of the medical image 202 may be improved. Removing medical images that do not represent a medical abnormality from the radiology workflow reduces redundant processing of medical images.

[0060] Figure 3 A flowchart of an example process used by the image encoder 204 in method 100a to generate an image embedding vector 206 is shown. The image encoder 204 is trained to generate multiple image embedding vectors 206 based on the medical image data 202. Its training process is described in detail below. The example image encoder 204 described below is similar to the group vision transformer described in "GroupViT: Semantic Segmentation Emerges from Text Supervision" (arXiv.org, 2022) by Xu et al. However, other transformer-based attention mechanisms, such as the Swin Transformer, can be used.

[0061] The image encoder 204 first divides the image into equal-sized portions 302, such as equal-sized rectangular portions 302. The image encoder 204 assigns a label to each portion 302. Initially, the labels are all different. Figure 3 In the illustrative 2D example shown in , there are a total of 6×6=36 parts 302, labeled 1 to 36. In practice, the number of parts 302 will typically be much larger than this. In the case where the medical image data 202 is 3D, each part 302 may include a cube of voxels in the medical image 202. For example, each part 302 may include a voxel cube having a side length of 3 voxels.

[0062] For each label, each portion 302 is assigned a probability that the portion 302 belongs to that label. The initial assigned probabilities simply indicate that the portion 302 belongs to only one label and not to any other label. For example, Figure 3 The upper left portion shown in has a 100% probability of belonging to label 1 and a 0% probability of belonging to any other label. The portion immediately to the right of this portion has a 100% probability of belonging to label 2 and a 0% probability of belonging to any other label.

[0063] The medical image data 202 of each portion 302 (eg, voxel) is input to a transformer layer 304. The transformer layer 304 determines a similarity measure for each pair of portions 302. The similarity measure represents the visual similarity between the portions 302.

[0064] The similarity measure is input to the grouping layer 306. The grouping layer 306 groups the different portions 302 together based on the similarity measure to determine a number of tags that is less than the number of tags that existed before the grouping. Figure 3 In the example shown in , 36 initial labels are reduced to 14 intermediate labels after the first grouping layer 306. The number of labels before and after grouping is a fixed parameter of the image encoder 204.

[0065] The intermediate labels represent specific image features. For example, Figure 3 The middle label 1 in the middle image of represents the empty black space. The middle label 3 represents the tissue outside the lungs of the image subject. The other middle labels represent other image features.

[0066] By means of transformation and grouping, for each intermediate label, each image portion 302 is associated with a probability of belonging to the intermediate label. Figure 3The middle label shown in represents the middle label associated with the highest probability for that portion. The black space at the top of the image is likely to have a probability close to 100% belonging to middle label 1, and a probability close to 0% belonging to any other middle label. Meanwhile, for example, portion 302, which has a higher probability of belonging to middle label 10 than any other middle label, may have a probability of 40% belonging to middle label 10, 30% belonging to middle label 9, and 10% belonging to middle label 8. The probabilities for a given image portion need not add up to 100%.

[0067] The operations of transforming and grouping can be repeated a fixed number of times until the desired number of tags exists. Figure 3 In the case shown in , after the final stage of transformation and grouping, there are two final labels. Depending on the content of the training data, the two final labels can represent specific medical image features; these form the multiple medical image features mentioned above. For example, Figure 3 The final label 1 in represents normal tissue, while the final label 2 represents abnormal tissue. Again, for simplicity, only the label associated with the highest probability is shown for each portion 302 of the medical image 202. Each portion 302 has a probability of belonging to the final label 1 (i.e., representing normal tissue) and a probability of belonging to the final label 2 (i.e., representing abnormal tissue).

[0068] Each image embedding vector generated by the image encoder 204 is associated with one of the final labels or medical image features. In this example, each component of the image embedding vector represents the probability that the corresponding image portion 302 belongs to the label or medical image feature associated with the image embedding vector.

[0069] Figure 4 A flow chart of an example training process 400 for training the image encoder 204, text encoder 210, and text decoder 220 used in the method 100a is shown. Figure 4 In the example shown in , a large language model (e.g., ChatGPT) is trained in a separate process and is used in the training process 400 in its trained form.

[0070] The training process 400 includes inputting a plurality of medical image datasets 202 into the image encoder 204 to generate a plurality of trial image embedding vectors 228 for each medical image dataset 202. The trial image embedding vectors 228 referred to herein are image embedding vectors generated for the training process 400, as opposed to image embedding vectors generated by the training image encoder 204 during the inference method 100.

[0071] The medical image datasets 202 may be selected from a medical image library. The selection process may involve selecting each medical image that contains one of the plurality of medical image features. For example, where the plurality of medical image features are a plurality of medical abnormalities, images that include one of the medical abnormalities may be selected, while images that include other abnormalities may be excluded. Where the plurality of medical image features include normal tissue and abnormal tissue, a plurality of medical image datasets 202 depicting normal tissue may be selected, and an approximately equal number of medical image datasets 202 depicting abnormal tissue may be selected.

[0072] The inventors have recognized that because medical reports typically include indications of medical image features together with associated images, they can be used as training data, where the indications in the report serve as reference truth data. This avoids the difficulty of labeling medical image training data - human implementers of the training process are typically not qualified to perform accurate assessments of medical images. Therefore, each medical image data set 202 is associated with a medical report 224 of the medical image 202. The medical report 224 includes data indicating the presence of one of the multiple medical image features. For example, the medical report 224 may include text such as "there is a low-density lesion". Each report also includes data indicating the area of ​​the feature indicated as present. For example, the medical report 224 may include text such as "right lobe of the liver". The report may be a report written by a medical professional or another person. The report may include data indicating the presence of a feature in the same sentence as the data representing the area of ​​the feature. For example, the medical report 224 may include a sentence such as "there is a low-density lesion in the right lobe of the liver".

[0073] By means of the first loss function described below, the regional data can guide the image encoder 204 to focus on the image portion including the medical image feature. For example, when the untrained image encoder 204 is only told that there is a low-density lesion somewhere in the medical image 202 used in training, it is more likely to mistakenly identify the image portion that does not include the low-density lesion as including the low-density lesion. Specifically, it may not output any image embedding vector indicating a high probability for the image portion 302 including the low-density lesion and a low probability for the image portion 302 not including the low-density lesion. At the same time, when the untrained image encoder 204 is told that there is a low-density lesion in the right lobe of the liver, it is more likely to pay attention to the correct part of the image. It is more likely to output at least one image embedding vector that correctly indicates a high probability for the image portion 302 including the low-density lesion and a low probability for other places. Including regional data can enable the image encoder 204 to be trained using fewer training examples because the image encoder 204 can more easily identify the image portion that it needs to focus on. It may also improve the accuracy of the image embedding vectors produced by the trained image encoder 204, and therefore improve the accuracy of the segmentation maps 218 described above.

[0074] In this example, the report is input to a natural language processing system to generate a data set representing the findings 222 of the medical report 224. The natural language processing system may be, for example, Figure 4 , such as ChatGPT. Before being input to the LLM 226, the medical report 224 may include information that is not useful for training the text encoder 210 and the image encoder 204. For example, the medical report 224 may include the name of the patient and measurements about the patient, such as their height and weight. The medical report 224 may be input to the LLM 226 along with a prompt such as "Summarize the findings of this medical report 224." Data representing the output of the LLM 226 may then be input to the text encoder 210. This will exclude redundant information that may mislead the text encoder 210.

[0075] The text encoder 210 is configured to generate a trial feature vector 230 based on the input findings summary 222. The trial feature vector 230 effectively encodes information about the medical image features (eg, abnormalities) described in the medical report 224 and the regions in which the image features exist.

[0076] The training process 400 includes jointly training the image encoder 204 and the text encoder 210 to minimize a first loss function between a trial image embedding vector 228 and a corresponding trial feature vector 230. In this example, the trial image embedding vectors 228 generated for a given medical image are averaged to generate an average embedding vector 232. For a corresponding image portion, each component of the average embedding vector 232 represents the probability that the image portion 302 averaged over all medical image features includes the given medical image feature. It should be noted that since the probabilities for a given image portion 302 do not need to add up to 1, the average of the probabilities is not necessarily constant and is likely to vary across images. It should also be noted that in this example, the image embedding vectors are not manually assigned to the corresponding medical image features prior to training - instead, this is implemented through training, and the specific assignment of medical image features to image embedding vectors is not intentionally set by the user.

[0077] The first loss function measures the similarity between the generated trial feature vector 230 and the average image embedding vector 232. For example, the similarity can be measured by Euclidean distance or cosine distance. The lower the similarity level between the average image embedding vector 232 and the feature vector of the given medical image, the higher the first loss function value.

[0078] In some examples, a contrast loss function may be used as the first loss function. For example, the medical images used in the training may be grouped into pairs, where each medical image pair includes a normal medical image representing only normal tissue and an abnormal medical image representing a medical abnormality. The contrast loss function additionally measures the similarity between the average image embedding vector 232 generated for the normal medical image and the average image embedding vector 232 generated for the abnormal medical image. The higher the level of similarity between the average image embedding vector 232 of the normal medical image and the average image embedding vector 232 of the abnormal medical image, the higher the value of the contrast loss function.

[0079] In any case, the first loss function can be used to update the parameters of the image encoder 204 and the text encoder 210. The weights of both the transformer layer 304 and the grouping layer 306 of the image encoder 204 can be updated. However, the number of labels before and after each grouping layer 306 is typically a fixed parameter of the image encoder 204. Further, in this example, the parameters of the text decoder 220 are not updated using the first loss function.

[0080] Through training, the image encoder 204 may be trained to generate an image embedding vector based on the medical image 202 , the image embedding vector representing information to be included in a summary of the findings 222 of the medical report 224 for the medical image 202 .

[0081] The assignment of image embedding vectors to medical image features (e.g., different medical abnormalities) occurs automatically and is implemented through training. The medical report 224 of an abnormal medical image typically includes a finding describing the abnormality. Therefore, in order to minimize the first loss function, it is advantageous for the image encoder 204 to generate at least one image embedding vector for each medical image, the image embedding vector representing a high probability value for the image portion 302 that includes the abnormality. Since the same abnormality is likely to be described in several medical reports 224 for the corresponding medical image, generating such a vector reduces the overall value of the first loss function for these medical images. Therefore, in order to minimize the first loss function, the image encoder will output an image embedding vector that appropriately describes the abnormalities that are typically mentioned in the medical report. Therefore, the actual medical image features represented by the image embedding depend firstly on the content of the medical report, and secondly on the number of image embeddings generated for each medical image.

[0082] Thus, the image encoder 204 may be trained to generate a plurality of image embedding vectors corresponding to a corresponding plurality of medical image features based on the received medical image data 202. In particular, the image encoder 204 may generate an image embedding vector for each of a plurality of different types of abnormalities, and thus generate a segmentation map.

[0083] Further, using the first loss function, the text encoder can be trained to generate a feature vector based on the indication of the medical image features. The first loss function is minimized in the case where the average image embedding vector 232 matches the trial feature vector 230 generated by the text encoder. Since the medical report describes the features (e.g., abnormalities) represented in the medical image 202, in order to minimize the first loss function, the text encoder will generate a feature vector representing the information encoded in the finding summary generated by the LLM 226. If it does not do this, and instead randomly generates a feature vector, the image embedding vector generated by the image encoder 204 will not match the feature vector generated by the text encoder 210.

[0084] Some automatic segmentation methods can be used to identify well-defined medical image features. However, some features, including ground glass opacities and bones in X-rays, do not exhibit well-defined boundaries when represented in medical images. In other words, the transition between normal tissue and tissue affected by the medical image feature is gradual. Therefore, when segmenting such features, methods for segmenting images that involve determining the boundary between two segments of the image are inappropriate.

[0085] To address this problem, machine learning-based systems can be trained using medical images for which practicing physicians have manually indicated the boundaries of abnormalities. However, this is very labor-intensive.

[0086] The above-described training process 400 and resulting runtime method 100a address this problem by using readily available medical reports as training data.

[0087] As mentioned above, method 100a includes inputting at least one of the image embedding vectors to a text decoder 220 to generate natural language text describing findings 222 associated with the medical image data 202. We now describe an example training method for training the text decoder 220. The training method may form part of a training process 400.

[0088] The training method includes inputting at least one image embedding vector generated by the image encoder 204 based on the input medical image dataset 202 to the text decoder 220. Based on the input image embedding vector, the text decoder 220 generates a trial natural language text describing a finding 222 related to the input medical image dataset 202.

[0089] The training dataset used in the training using the first loss function can also be used to train the text decoder 220. As described above, these training datasets include the medical image dataset 202 and the corresponding medical reports 224. The text decoder 220 can be trained after the image encoder 204 and the text encoder 210 have been trained. Alternatively, in alternating training sessions, the image encoder 204 / text encoder 210 can be trained using the first loss function and the text decoder 220 can be trained using the second loss function. In any case, in some examples, during training using the second loss function, the parameters of the image encoder 204 and the text encoder 210 are not updated.

[0090] In one example, the text decoder 220 generates a trial natural language text based on a plurality or all of the image embedding vectors generated by the image encoder 204 for a given input medical image dataset 202 .

[0091] The training method includes training the text decoder 220 to minimize the second loss function between the test natural language text generated for each input medical image data set 202 and the data representing the corresponding medical report 224. In one example, the data representing the medical report 224 includes a summary of the findings generated by the LLM 226 for training using the first loss function. For example, a vector representing the natural language text generated by the LLM 226 can be compared with a vector representing the test natural language text generated by the text encoder 210. In this article, the natural language text can be represented by a vector by, for example, using a mapping between words in the natural language text and vectors such as word2vec, and averaging the vectors to generate a vector representing the entire natural language text. For example, the comparison can involve calculating the Euclidean or cosine distance between the vectors. Based on the comparison, the second loss function value for a given input medical image data set 202 can be calculated. The greater the distance between the vectors, the higher the second loss function value, that is, the more the test natural language text does not match the summary of the findings generated by the LLM 226.

[0092] The second loss function may be used to update parameters (eg, weights) of the text decoder 220. In this example, it is not used to update parameters of the image encoder 204.

[0093] By training using a second loss function, the text decoder 220 can be trained to generate natural language text based on at least one image embedding vector output by the image encoder 204 based on the medical image 202, wherein the natural language text represents information to be included in a summary of the findings of the medical report 224 of the medical image 202.

[0094] Figure 5 500 according to an example is shown. The apparatus 500 may be implemented as a processing system and / or a computer. The apparatus 500 includes an input interface 506, an output interface 508, a processor 502, a memory 504, and a display device 510. The processor 502 and the memory 504 may be configured as described above. Figures 1 to 4 The method 100, 100a and / or the training process 400 may be performed by any of the examples described. The memory may store instructions that, when executed by the processor 502, cause the processor 502 to perform the above-referenced Figures 1 to 3 The method 100, 100a of any of the examples described, and / or according to the above reference Figure 4 The training method 400 of any of the described examples. The instructions may be stored on any computer-readable medium, such as any non-transitory computer-readable medium.

[0095] For example, the input interface 506 may receive the medical image data 202, and the processor 502 may implement the above reference Figures 1 to 3 The method 100a described above is used to identify the image embedding vector. The apparatus 500 may also display the segmentation map 218 of the identified image embedding vector 214 on the display device 510. In some examples, the natural language text generated by the text decoder 220 may be transmitted to a structured storage device (not shown) so that the natural language text is stored in the structured storage device.

[0096] The above examples should be understood as illustrative examples of the present invention. It should be understood that any feature described in connection with any one of the examples may be used alone or in combination with the other features described, and may also be used in combination with one or more features of any other example or with any combination of any other example. Furthermore, equivalents and modifications not described above may also be adopted without departing from the scope of the present invention as defined in the appended claims.

Claims

1. A computer-implemented method for processing medical image data, the method comprising: receiving, at a first machine learning-based system, medical image data representing a medical image; a first machine learning-based system generating, based on the received medical image data, a plurality of image embedding vectors corresponding to a respective plurality of medical image features, each of the plurality of image embedding vectors being associated with a different respective medical image feature and comprising medical image feature data indicating a presence or absence of the respective medical image feature at each of a plurality of locations in the medical image; receiving, at a second machine learning based system, an indication of a first medical image feature included in the medical image, and generating, by the second machine learning based system, a feature vector based on the indication; as well as A comparison of the feature vector to a plurality of image embedding vectors is performed, and a first image embedding vector is identified from among the plurality of image embedding vectors based on the comparison.

2. The method of claim 1, wherein the first image embedding vector is identified as having the highest similarity with the feature vector from among the plurality of image embedding vectors.

3. The method according to claim 1 or claim 2 comprises determining position data representing the position of the indicated first medical image feature in the medical image data in the medical image based on the identified first image embedding vector.

4. A method according to claim 3, wherein the medical image feature data represents a segmentation map of the corresponding medical image feature, and the determined position data includes the segmentation map represented by the identified first image embedding vector. The method according to claim 4 , wherein the segmentation map indicates the probability that the corresponding medical image feature exists at the corresponding position in the medical image.

6. The method according to claim 4 or claim 5, comprising: Determining whether a medical image represents a medical abnormality, and In response to determining that the medical image represents a medical abnormality, the segmentation map is displayed on a display device.

7. A method according to any one of claims 1 to claim 6, wherein the indication received at the second machine learning based system includes region data representing a region of the first medical image feature in the medical image, and the comparison of the feature vector with the plurality of medical image embedding vectors is performed at least in part based on the region data.

8. The method of any one of claims 1 to 7, wherein the indication received at the second machine learning based system comprises data representing a textual prompt indicating a feature of the first medical image.

9. The method according to any one of claims 1 to 8, comprising performing a training process to train the first machine learning based system and the second machine learning based system, the training process comprising: Inputting first training data including a plurality of medical image datasets into a first machine learning-based system to generate a plurality of trial image embedding vectors for each medical image dataset; and inputting second training data into the second machine learning-based system to generate a trial feature vector for each medical image dataset, wherein the second training data comprises data representing a plurality of medical reports, each medical report comprising data indicating the presence of one of the plurality of medical image features in a corresponding one of the medical image datasets, and each medical report comprising data representing a region of the feature indicated as being present in the medical image represented by the corresponding medical image dataset, The training process includes jointly training a first machine learning-based system and a second machine learning-based system to minimize a loss function between a test image embedding vector and a corresponding test feature vector.

10. The method of claim 9, comprising inputting each of the medical reports into a natural language processing system to generate a data set representing findings of the medical reports, wherein the data representing the plurality of medical reports is data representing findings of the medical reports.

11. The method of any one of claims 1 to 10, comprising inputting at least one of the plurality of image embedding vectors into a third machine learning-based system to generate natural language text describing findings associated with the medical image data.

12. The method of claim 11, comprising executing a training method to train the third machine learning based system, the training method comprising: inputting the image embedding vectors generated by the first machine learning-based system based on the input medical image datasets to a third machine learning-based system to generate, for each input medical image dataset, a trial natural language text describing findings associated with the input medical image dataset; and A third machine learning-based system is trained to minimize a loss function between trial natural language text and data representing medical reports corresponding to an input medical imaging dataset.

13. The method of any one of claims 1 to 12, wherein at least one of the plurality of medical image features comprises a medical abnormality.

14. An apparatus configured to perform the method according to any one of claims 1 to 13.

15. A computer program which, when executed by a computer, causes the computer to perform the method according to any one of claims 1 to 13.

Images