Generating natural language text describing changes between medical imaging datasets

Abstract

A framework for generating natural language text describing a change between first medical imaging data and second medical imaging data is disclosed. Imaging data representative of the first medical imaging data and the second medical imaging data or representative of a difference between the first medical imaging data and the second medical imaging data is obtained. The imaging data is input into a first trained machine learning model to generate an image feature vector representative of a difference between the first medical imaging data and the second medical imaging data. Data representative of the image feature vector is input into a second trained machine learning model to generate natural language text describing a change between the first medical imaging data and the second medical imaging data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority of European Patent Application No. 23191392.2 filed on August 14, 2023, the contents of which are incorporated by reference. Technical Field

[0003] The present disclosure relates to a method and apparatus for analyzing changes between two medical imaging data sets. Background Art

[0004] When a patient's condition is diagnosed using baseline medical images, follow-up imaging is often recommended to monitor the progression of the condition. After the follow-up images are taken, the radiologist can review the follow-up images and the baseline images and write a report describing any progression of the condition. However, interpreting the follow-up images and / or writing the reports are time-consuming tasks.

[0005] In addition, when comparing a baseline image taken at diagnosis with a follow-up image taken subsequently, the image may include artifacts that are not related to the progression of the condition. For example, artifacts may be visible where the lighting or imaging angle used to capture the baseline image is different than the lighting or imaging angle used to capture the follow-up image. Other unrelated changes may be evident in the image. This may make the interpretation and / or reporting of the follow-up and baseline images difficult.

[0006] It would be desirable to automate or at least provide computer-implemented assistance in the interpretation and / or reporting of follow-up medical images. Summary of the invention

[0007] A framework for generating natural language text describing a change between first medical imaging data and second medical imaging data is described herein. Imaging data representing the first medical imaging data and the second medical imaging data or representing a difference between the first medical imaging data and the second medical imaging data is obtained. The imaging data is input into a first trained machine learning model to generate an image feature vector representing the difference between the first medical imaging data and the second medical imaging data, and data representing the image feature vector is input into a second trained machine learning model to generate natural language text describing a change between the first medical imaging data and the second medical imaging data. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] A more complete appreciation of the present disclosure and its many attendant aspects will be readily obtained as the present disclosure becomes better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0009] Figure 1is a flow chart schematically illustrating a method for generating natural language text describing changes between first medical imaging data and second medical imaging data;

[0010] Figure 2a is a flow chart schematically illustrating a method for obtaining differential imaging data from first medical imaging data and second medical imaging data;

[0011] Figure 2b is an illustration of the difficulties associated with identifying medical abnormalities from differential imaging data obtained by visual inspection;

[0012] Figure 3 is a more detailed schematic diagram of the Figure 1 A flow chart of the method illustrated in the method;

[0013] Figure 4 is a schematic diagram of the process used to train a machine learning model to perform Figure 1 A flow chart of the method illustrated in the method;

[0014] Figure 5 is a more detailed schematic diagram of the Figure 4 A flow chart of the method illustrated in the method;

[0015] Figure 6 is a diagram schematically illustrating an apparatus according to an example. DETAILED DESCRIPTION

[0016] Independent of the grammatical use of the term, individuals with both male and female identities are included in the term.

[0017] refer to Figure 1 , illustrates a computer-implemented method 100 for generating natural language text describing changes between first medical imaging data 302 and second medical imaging data 304 .

[0018] In a broad overview, the method 100 includes: in step 102, obtaining imaging data 306 representing first medical imaging data 302 and second medical imaging data 304 or representing the difference between the first medical imaging data 302 and the second medical imaging data 304 (see Figure 2a ); In step 104, the imaging data 306 is input into a first trained machine learning model 340 to generate an image feature vector Zi (e.g., Figure 3 In step 106, the data representing the image feature vector Zi is input into the second trained machine learning model 350 to generate natural language text describing the changes between the first medical imaging data 302 and the second medical imaging data 304 (e.g., see Figure 3 A second trained machine learning model 350 in ).

[0019] Thus, natural language text describing the changes between the first medical imaging data 302 and the second medical imaging data 304 may be generated.

[0020] By using the first trained machine learning model 340 to generate an image feature vector Zi based on the imaging data 306, an image feature vector Zi can be generated that encodes features related to the changes between the first medical imaging data 302 and the second medical imaging data 304. Thus, for example, the progression of the patient's condition depicted in the first medical imaging data 302 and the second medical imaging data 304 can be extracted and encoded into the image feature vector Zi. The second trained machine learning model 350 uses this image feature vector to generate a natural language text describing the changes. Therefore, the generated natural language text describing the changes is related to the (one or more) related changes between the first medical imaging data and the second medical imaging data. Such generated natural language text can provide automation for the interpretation and / or reporting of the second medical imaging (such as a follow-up image). For example, such text can represent or be used to generate a report of the follow-up image. Alternatively or additionally, such generated natural language text can allow assistance to be provided in the interpretation and / or reporting of the follow-up image. For example, a radiologist can use such generated text to guide his interpretation and / or reporting of the follow-up image.

[0021] The natural language text describing the changes can be used by the medical professional in a variety of ways. For example, the medical professional can recommend a course of treatment or decide that further follow-up imaging is needed.

[0022] Reference now Figure 3 An example method 300 is described in detail. The method 300 is a specific example of the method 100 and includes the steps of the method 100.

[0023] As mentioned, the method 300 includes, in step 102, obtaining imaging data 306 representing the first medical imaging data 302 and the second medical imaging data 304 or representing the difference between the first medical imaging data 302 and the second medical imaging data 304. In this example, the imaging data 306 includes differential imaging data representing the difference between the first medical imaging data 302 and the second medical imaging data 304.

[0024] Each of the first medical imaging data 302 and the second medical imaging data 304 can include an array of elements each having a value. The elements can be pixels, each pixel having at least one value. The at least one value can correspond to or otherwise represent an output signal of a medical imaging technique used to generate the medical imaging data. For example, for magnetic resonance imaging, the value of an element (e.g., a pixel) can correspond to or represent the rate at which an excited nucleus in a region corresponding to the element returns to an equilibrium state. In some examples, each element can have only one value. However, in other examples, each element can have multiple values ​​or be otherwise associated with multiple values. For example, multiple values ​​of a given element can represent the values ​​of a corresponding plurality of signal channels. For example, each signal channel can represent a different medical imaging signal or the properties of an imaging object. In some examples, at least one value can include an element (e.g., a pixel) intensity value. For example, an output signal from medical imaging can be mapped to a pixel intensity value, such as a value within a defined range of intensity values. For example, for a grayscale image, an intensity value can correspond to a value within a range of 0 to 255, where, for example, 0 represents a "black" pixel and 255 represents a "white" pixel. As another example, for example, in the case of USHORT medical image data, the intensity values ​​may correspond to values ​​in the range of 0 to 65536. As another example, in a color image (e.g., where different colors represent different properties of the imaged object), each pixel may have three intensity values, e.g., one each for the red, green, and blue channels. It will be appreciated that other values ​​may be used.

[0025] In some examples, the first medical imaging data 302 is for a patient and has been captured at a first time, and the second medical imaging data 304 is for the patient and has been captured at a second later time. For example, the first medical imaging data 302 may form part of a baseline study of the patient. Figure 3 As shown in, the first medical text report 312 can be written for a baseline study. The second medical imaging data 304 can form part of a follow-up study of the patient. The follow-up study can be intended to track the progression (or regression) of the medical abnormality 308 and / or identify the occurrence (or disappearance) of the medical abnormality 308 after the baseline study. In this example, the first medical imaging data 302 and the second medical imaging data 304 are for a specific area of ​​the patient. For example, the first medical imaging data 302 and the second medical imaging data 304 can be for the abdominal area of ​​the patient or for the limbs of the patient.

[0026] Generating natural language text describing the changes, where the first medical imaging data 302 is for the patient and has been captured at a first time, and the second medical imaging data 304 is for the patient and has been captured at a second later time, enables computer-implemented interpretation and / or reporting of follow-up medical images. This in turn enables the progression of the patient's condition to be monitored without necessarily requiring human analysis of the first medical imaging data 302 and the second medical imaging data 304. Alternatively or additionally, this may enable computer-implemented assistance in the interpretation and / or reporting of follow-up medical images or monitoring of the progression of the patient's condition by a medical professional.

[0027] In some examples, the method 300 includes obtaining first medical imaging data 302 and second medical imaging data 304. For example, the first medical imaging data 302 and the second medical imaging data 304 may be obtained from a storage device (such as a memory (see, e.g., Figure 6 The first medical imaging data 302 may include a plurality of first intensity values, and the second medical imaging data 304 may include a plurality of corresponding second intensity values. That is, each of the first intensity values ​​in the first medical imaging data 302 corresponds to a second intensity value in the second medical imaging data 304. Each of the first intensity values ​​may be associated with a corresponding pixel position, and each of the second intensity values ​​may be associated with one of the pixel positions to provide such a correspondence.

[0028] In this example, the method 300 includes image preprocessing 332 of the first medical imaging data 302 and the second medical imaging data 304. For example, the image preprocessing 332 can include image registration. Image registration generally involves converting a plurality of first intensity values ​​and a plurality of second intensity values ​​into a coordinate system. Image registration can include associating each of the first intensity values ​​with its corresponding second intensity value, and corresponding pixel locations can be defined during image registration.

[0029] In some examples, method 300 includes, for each of the plurality of first intensity values, comparing the first intensity value to a corresponding second intensity value to obtain a differential intensity value. In these examples, differential imaging data 306 includes the obtained differential intensity value. For example, Figure 2a As illustrated in , comparing the first intensity value to the second intensity value may include performing a subtraction operation using the first intensity value and the second intensity value. For example, comparing the first intensity value to the second intensity value may include subtracting the second intensity value from the first intensity value, or vice versa. Subtracting the second intensity value from the first intensity value, or vice versa, may allow for a computationally simple method of obtaining differential imaging data 306. Figure 2a and Figure 2bIn the differential imaging data 306 shown in FIG. 1 , regions where the intensity values ​​of the second medical imaging data 304 are lower than the intensity values ​​of the first medical imaging data 302 are shown in light gray, and regions where the intensity values ​​of the second medical imaging data 304 are higher than the intensity values ​​of the first medical imaging data 302 are shown in dark gray. A medical abnormality 308 is shown on each of the first medical imaging data 302, the second medical imaging data 304, and the differential medical imaging data. Between the time when the first medical imaging data 302 was captured and the time when the second medical imaging data 304 was captured, the medical abnormality 308 has undergone a slight change in appearance.

[0030] Figure 2b Two copies of differential medical imaging data are shown, with the location of the abnormality indicated on the second copy. Figure 2b As can be appreciated from the differential imaging data 306 shown in FIG. 1 , it may be difficult to identify the medical abnormality 308 from only a visual inspection of the differential imaging data 306. In fact, it may also be difficult for a computer vision system to identify the medical abnormality 308 using only the differential imaging data 306. Some examples described herein enable a first trained machine learning model 340 to identify changes between the first medical imaging data 302 and the second medical imaging data 304 by training the machine learning model using a medical text report associated with the medical imaging data in addition to the medical imaging data. This will be further referenced Figure 5 Give a description.

[0031] As mentioned, the method 300 includes, at step 104, inputting the imaging data 306 into the first trained machine learning model 340 to generate an image feature vector Zi representing the difference between the first medical imaging data 302 and the second medical imaging data 304. In this example, the imaging data 306 is differential imaging data 306, and the image feature vector Zi represents the differential imaging data.

[0032] In this example, the first trained machine learning model 340 has been trained to generate an image feature vector Zi based on an input of given differential imaging data representing the difference between given first medical imaging data and given second medical imaging data. Figure 5 The training process is described in further detail.Any machine learning model described herein may include a neural network, such as a convolutional neural network or a transformer.

[0033] In this example, generating the image feature vector Zi includes inputting the differential imaging data 306 into a first component trained machine learning model of a first trained machine learning model 340 to generate one or more attention maps 322. In this example, the first component trained machine learning model includes an image encoder 342. The image encoder 342 has been trained to generate one or more attention maps based on the input of the given differential imaging data 306. By using the following reference Figure 5

[00106] Using the further loss function 470 described above to train the first trained machine learning model 340, the image encoder 342 may be able to generate an attention map indicating features of the differential imaging data 306 that may represent information related to the medical text report.

[0034] Each attention map can indicate the extent to which a particular portion or segment of the differential imaging data 306 contributes to the encoding or a particular portion of the encoding performed by the image encoder 342. For example, each attention map can relate to a different segment or a different context of the differential imaging data 306. For example, one or more attention maps can each highlight a particular segment or (one or more) portions of the differential imaging data that the image encoder 342 determines corresponds to a corresponding particular context. For example, the particular context can be that the (one or more) portions correspond to a medical abnormality. The segment can include a sub-plurality of differential intensity values. The image encoder 342 can segment the differential imaging data 306 into attention maps based on the contribution of each particular segment or context to the encoding. For example, the image encoder can be configured to segment the differential imaging data 306 based on the corresponding locations of possible abnormalities appearing in the differential imaging data 306.

[0035] In this example, the method 300 includes inputting data representing one or more attention maps 322 into a second component trained machine learning model 344 of the first trained machine learning model 340 to generate an image feature vector Zi. In this example, the second component trained machine learning model 344 has been trained to generate an image feature vector Zi based on an input of a given attention map.

[0036] The data representing the one or more attention maps 322 may include the one or more attention maps 322. However, in this example, the method 300 includes generating data representing the one or more attention maps 322 by applying a dimensionality reduction operation to each of the one or more attention maps 322. Applying the dimensionality reduction operation may reduce the number of pixels in the differential imaging data 306 and / or the amount of data used to store the attention maps. The dimensionality reduction operation may include an average pooling operation. In general, the medical abnormality 308 may be represented by multiple adjacent or nearby pixels. Therefore, applying the dimensionality reduction operation may reduce the number of pixels used to represent the differential imaging data 306 without substantially removing or suppressing information related to the medical abnormality 308 represented by the pixels before applying the dimensionality reduction operation. In some examples, the dimensionality reduction operation (such as average pooling) is a fixed operation applied to the pixels in the sense that the dimensionality reduction operation itself does not involve applying a machine learning model to the one or more attention maps 322.

[0037] In this example, the second component machine learning model includes a multilayer perceptron (MLP) 1. The multilayer perceptron has been trained to generate an image feature vector Zi based on an input of data representing one or more given attention maps 322 (whether a dimensionality reduction operation has been applied to the given attention map 322). Generating the image feature vector Zi may include inputting data representing one or more attention maps 322 (whether a dimensionality reduction operation has been applied to the attention map 322) into the MLP 1 to generate the image feature vector Zi.

[0038] The image feature vector Zi can accurately represent the features of the imaging data 306 that will be relevant to the medical professional in analyzing the first medical imaging data 302 and the second medical imaging data 304. Figure 5 Explain how the first machine learning model 440 can be trained to generate image feature vectors Zi having such properties.

[0039] As mentioned, the method 300 includes, at step 106 , inputting data representing the image feature vector Zi into a second trained machine learning model 350 to generate natural language text describing changes between the first medical imaging data 302 and the second medical imaging data 304 .

[0040] The second trained machine learning model 350 may have been trained to generate natural language text describing the changes between given first medical imaging data and given second medical imaging data based on the input of given data representing the image feature vector Zi. Figure 5 The training process is described in further detail.

[0041] By “describing the change,” it is meant that the natural language text generally describes the change in such a way that the description of the change is understandable to a medical professional. Thus, the present invention may enable a description of the change between the first medical imaging data 302 and the second medical imaging data 304 to be automatically generated. The medical professional may use the description of the change to, for example, diagnose a medical condition or recommend a treatment. In any case, by generating such natural language text, a radiologist or other medical professional may understand what change(s) have occurred between the first medical imaging data 302 and the second medical imaging data 304 without having to evaluate the first medical imaging data 302 or the second medical imaging data 304. Furthermore, a radiologist visually inspecting the first medical imaging data 302 and the second medical imaging data 304 or the differential imaging data 306 may miss changes that have occurred. By Figure 1 and Figure 3 The method illustrated in generates natural language text describing the changes,between two medical imaging datasets may allow such overlooked,objects to be avoided.

[0042] In some examples, the change indicates the appearance or progression of the medical abnormality 308. For example, the change may indicate the initial appearance or growth of a tumor between the first medical imaging data 302 and the second medical imaging data 304. In these examples, by generating natural language text describing the change, the initial appearance of the medical abnormality 308 may be identified, or the progression of the medical abnormality 308 may be tracked. In examples where the first medical imaging data 302 is for a patient and has been captured at a first time and the second medical imaging data 304 is for the patient and has been captured at a second, later time, the change in the patient's condition may be monitored.

[0043] In this example, the method 300 includes generating a second medical text report 314 associated with the second medical imaging data 304 based on a first medical text report 312 associated with the first medical imaging data 302 and a natural language text describing a change. The first medical text report 312 may describe the first medical imaging data 302 or any feature in the first medical imaging data 302. In this example, the first medical text report 312 includes at least one sentence, such as "There is a low-density lesion in the right lobe of the liver." The second medical text report 314 may describe the second medical imaging data 304 or any feature in the second medical imaging data 304. In this example, the second medical text report 314 includes at least one sentence, such as "A low-density lesion in the right lobe of the liver is seen again. There is an increase in size." The second medical text report 314, such as the natural language text describing a change, may describe any change that has occurred between the first medical imaging data and the second medical imaging data. In this example, the natural language text describing a change includes at least one sentence, such as "There is an increase in the size of the low-density lesion in the right lobe of the liver."

[0044] By generating the second medical text report 314 based not only on the natural language text describing the changes but also on the first medical text report 312, the first medical text report 312 can serve as a guide for generating the second medical text report 314. Therefore, the second medical text report 314 can be provided, which focuses on the changes compared to the information described in the first medical text report.

[0045] In this example, generating a second medical text report 314 may include inputting the first medical text report 312 and the natural language text describing the change into a natural language processing model. In this example, the natural language processing model is a trained natural language processing machine learning model. Specifically, in this example, the trained natural language processing machine learning model is a trained large language model, such as ChatGPT or GPT-4. In this example, the trained large language model has been trained to generate a second medical text report 314 based on (among other aspects) the input of a given first medical text report 312, a given natural language text describing the change, and a text prompt including instructions for generating a second medical text report 314. For example, generating a second medical text report 314 may include inputting the following into a large language model:

[0046] "Based on the baseline radiology report below relating to the baseline radiology image and the description below of changes that have occurred between the baseline radiology image and the follow-up radiology image, write a follow-up radiology report describing the follow-up radiology image.

[0047] Here is the baseline radiology report. [First Medical Text Report 312]

[0048] Here is a description of the changes that have occurred between the baseline radiograph and the follow-up radiograph. [Natural language text describing the changes]", or alternatively.

[0049] "Here is the baseline radiology report associated with the baseline radiology images. [First Medical Text Report 312]

[0050] Here is a description of the changes that have occurred between the baseline radiograph and the follow-up radiograph. [Natural language text describing the changes]

[0051] Write a follow-up radiology report describing the follow-up radiological images. ”

[0052] Or any other appropriate prompt. In some examples, the trained large language model has not been trained for the specific purposes described above, but has instead been trained for general natural language processing.

[0053] By using a natural language processing model to generate the second medical text report 314, the efficiency of generating the second medical text report 314 can be improved compared to a medical professional writing the second medical text report 314. By using a trained natural language processing machine learning model or a large language model to generate the second medical text report 314, the content of the second medical text report 314 can be more relevant to the changes described in the natural language text.

[0054] refer to Figure 4 and Figure 5 , illustrates a computer-implemented training method 400, 400a for training a machine learning model for generating a natural language text describing changes between a first medical imaging data 302 and a second medical imaging data 304. Figures 1 to 3 The first trained machine learning model 340 and the second trained machine learning model 350 described may (or have) been based on Figure 4 and Figure 5 At the same time, the trained natural language processing machine learning model (e.g., ChatGPT) may have been trained in a separate process and can be used in the process of training method 400 in its trained form. Figure 5 The computer-implemented training method 400a illustrated in FIG. Figure 4 A specific example of a computer-implemented training method 400 is illustrated in FIG.

[0055] The training method 400, 400a includes: in step 202, providing a machine learning model.

[0056] The machine learning model includes (a) a first machine learning model 440, which is configured to generate an image feature vector Zi based on an input of given imaging data representing given first medical imaging data and given second medical imaging data or representing the difference between given first medical imaging data and given second medical imaging data. In this example, the given imaging data is given differential imaging data representing the difference between given first medical imaging data and given second medical imaging data. The first machine learning model 440 can have an architecture similar to or identical to the first trained machine learning model 340, for example, it can include a first component machine learning model and a second component machine learning model. This architecture is Figure 5 The machine learning model also includes (b) a second machine learning model 450 configured to generate a natural language text describing the changes between a given first medical imaging data and a given second medical imaging data based on an input of data representing the image feature vector Zi. The second machine learning model 450 can have an architecture similar to or identical to the second trained machine learning model 350.

[0057] In this example, the machine learning model includes (c) a text encoder machine learning model. The text encoder machine learning model is configured to generate a text feature vector Zt based on an input of data, wherein the data represents a given first medical text report associated with a given first medical imaging data and a given second medical text report associated with a given second medical imaging data. The data representing the given first medical text report and the given second medical text report may include data representing the first medical text report and data representing the second medical text report. Additionally or alternatively, the data representing the given first medical text report and the given second medical text report may include data representing a given natural language text describing a change.

[0058] The training method 400, 400a comprises: in step 204, providing training data.

[0059] The training data includes a plurality of training imaging data sets 406, each training imaging data set 406 representing a first training medical imaging data 402 and a second training medical imaging data 404, or representing a difference between the first training medical imaging data 402 and the second training medical imaging data 404. For each training imaging data set 406, the training data further includes ground truth natural language text describing changes between the first training medical imaging data 402 and the second training medical imaging data 404. In this example, the training imaging data is training differential imaging data representing a difference between the first training medical imaging data 402 and the second training medical imaging data 404.

[0060] In this example, providing training data includes generating ground truth natural language text for each training differential imaging data set 406. In this example, generating ground truth natural language text is performed by inputting a first training medical text report 412 associated with the first training medical imaging data 402 and a second training medical text report 414 associated with the second training medical imaging data 404 into a third trained machine learning model 460. In this case, for each training differential imaging data set 406, the training data may include the first training medical text report 412 and the second training medical text report 414. The third training machine learning model 460 may be a trained large language model. The third training machine learning model 460 may be similar to the above referenced Figures 1 to 3 The trained large language model described (e.g., ChatGPT or GPT-4) may be the same as the one described above, or it may be a different large language model. The trained large language model may have been trained to generate ground truth natural language text based on (among other things) input of a given first training medical text report associated with a given first training medical imaging data, a given second training medical text report associated with a given second training medical imaging data, and a text prompt including instructions to generate ground truth natural language text. For example, generating the natural language text may include inputting the following to the large language model:

[0061] “Write a description of any discrepancies between the following baseline and follow-up radiology reports.

[0062] Here is the baseline radiology report. [Given the first training medical text report]

[0063] Here is the follow-up radiology report. [Given Second Training Medical Text Report]",

[0064] or alternatively,

[0065] “The following radiology report accordingly describes two medical images of the same patient taken at different times.

[0066] Here is the first radiology report. [Given the first training medical text report]

[0067] Here is the second radiology report. [Given the second training medical text report]

[0068] Write a description of any changes that occurred between the time the medical images were taken. ”

[0069] or any other appropriate hint.

[0070] By training the machine learning model with the ground truth natural language text describing the changes between the first training medical imaging data 402 and the second training medical imaging data 404, during the inference process, the second machine learning model 450 can accurately output natural language text describing the changes. Inputting the first training medical text report 412 and the second medical text report 314 into the third trained machine learning model 460 can provide an automatic and accurate way to generate ground truth natural language text.

[0071] The training method 400, 400a includes: in step 206, training a machine learning model based on training data.

[0072] Training the machine learning model includes training the machine learning model based on the training data so as to minimize a loss function 460 between natural language text generated by the machine learning model for the training imaging data set 406 and corresponding ground truth natural language text of the training imaging data set 406. As described above, in this example, the training imaging data 406 is training differential imaging data 406.

[0073] Minimizing loss function 460 can include reducing the distance in the change description embedding space between the change description text feature vector and the basic truth value text feature vector. The change description text feature vector is a vector representation in the change description embedding space of the natural language text generated for a given training differential imaging data set 406. The basic truth value text feature vector is a vector representation in the change description embedding space of the basic truth value natural language text. For example, loss function 460 can include an item that decreases with this distance reduction. For example, any distance mentioned herein can include Euclidean distance or cosine distance.

[0074] In this example, the training method 400a includes training a machine learning model based on training data so as to minimize a further loss function 470 between the following vectors: a text feature vector Zt generated by a text encoder machine learning model 480 for a first training medical text report 412 and a second training medical text report 414 and an image feature vector Zi generated by a first machine learning model 440 for a corresponding training differential imaging dataset 406.

[0075] As mentioned above, the text encoder machine learning model 480 is configured to generate a text feature vector Zt. In this example, the text feature vector Zt is generated by inputting the first training medical text report 412 and the second training medical text report 414 into the third trained machine learning model 460 to generate a natural language text. This natural language text can be the same as the ground truth natural language text described above. Then, the natural language text is input to the text encoder machine learning model 480. The text encoder machine learning model 480 may include a multi-layer perceptron (MLP) 2. The text encoder machine learning model 480 then uses MLP2 to generate a text feature vector Zt. The text feature vector Zt and the image feature vector Zi can be represented in a common embedding space. The common embedding space can be the same as the change description embedding space, or it can be different.

[0076] In some examples, minimizing the further loss function 470 may include reducing the distance in a common embedding space between the image feature vector Zi generated for a given set in the plurality of training differential imaging datasets 406 and the text feature vector Zt generated for the given set. That is, the further loss function 470 may tend to attract the image feature vector Zi to the text feature vector Zt generated for the same training differential imaging dataset 406 in the common embedding space.

[0077] The further loss function 470 generally has the following effect: Figures 1 to 3 During the method 300 (i.e., during inference), the image feature vector Zi generated by the first trained machine learning model 340 captures features that would be considered relevant when writing natural language text describing changes between the first medical imaging data 302 and the second medical imaging data 304. This is because, by means of training using a further loss function 470, the first machine learning model 440 is trained to generate image feature vectors Zi that are close to the text feature vector Zt in a common embedding space, regardless of the content of the set of training data. In the case where the text feature vector Zt is generated using natural language text generated from the first training medical imaging data 402 and the second training medical imaging data 404, the image feature vector Zi effectively represents the same information by means of the further loss function. The same applies to one or more attention maps 322.

[0078] In some examples, the further loss function 470 is a contrastive loss function 460. Minimizing the contrastive loss function 460 includes, in addition to the above properties of the further loss function, increasing the distance in the common embedding space between the image Zi or text feature vector Zt generated for one set in the plurality of training differential imaging datasets 406 and the image Zi or text feature vector Zt generated for another set in the plurality of training differential imaging datasets 406. That is, the contrastive loss function 460 tends to separate feature vectors belonging to different sets from each other. The contrastive loss function generally has a functional form:

[0079] max S(Zi 1d ,Zt 1d )+S(Zi 2d ,Zt 2d )

[0080] min S(Zi 1d ,Zt 2d )+S(Zi 2d ,Zt 1d )

[0081] Wherein, S(x,y) is a function whose value decreases as the distance between x and y increases, 1d is a first set of multiple training data sets, and 2d is a second set of the multiple training data sets different from the first set.

[0082] Therefore, the contrast loss function 460 generally has the following effect: the image feature vector Zi generated for a differential imaging data set 306 (in the inference stage, i.e. Figure 1 and Figure 3 ) is different from the image feature vector Zi generated for another differential imaging dataset 306. This generally has the effect that the natural language text describing the changes generated by the second trained machine learning model 350 is also different for the two different differential imaging datasets 306. This can help generate more accurate natural language text describing the changes because it generally reduces the likelihood of generating the same natural language text describing the changes for two significantly different differential imaging datasets 306 (e.g., two differential imaging datasets 306 showing different medical abnormalities).

[0083] The training method 400, 400a may include training a machine learning model by alternately applying a loss function 460 and a further loss function 470 to the training data. For example, in a first stage, the first machine learning model 440 and the text encoder machine learning model 480 may process the training data to update the parameters of each of the first machine learning model 440 and the text encoder machine learning model 480. In the first stage, the parameters of the second machine learning model 450 may be frozen, that is, the parameters are not updated during the first stage. In the second stage, the second machine learning model 450 may process the training data to update the parameters of the second machine learning model 450. In the second stage, the parameters of the first machine learning model 440 and the text encoder machine learning model 480 may be frozen, that is, the parameters are not updated during the second stage. The first stage and the second stage may be repeated alternately and iteratively until each of the first machine learning model 440, the text encoder machine learning model 480, and the second machine learning model 480 has been trained. The parameters of the third trained machine learning model 460 may generally remain fixed during both the first stage and the second stage because the third trained machine learning model 460 has been trained. The machine learning model can process each training dataset in this two-stage manner.

[0084] refer to Figure 6 , illustrates an apparatus 600 according to an example. The apparatus 600 includes an input interface 606, an output interface 608, a processor 602, and a non-transitory memory device 604. The processor 602 and the non-transitory memory device 604 may be configured according to the above reference Figures 1 to 5 Any of the examples described herein performs the method 100, 300 or the training method 400, 400a. The memory may store instructions or computer readable program codes that, when executed by the machine or processor 602, cause the processor 602 to perform the above-described Figures 1 to 3 The method 100, 300 of any of the described examples and / or the method according to the above reference Figures 4 to 5 The training method 400, 400a of any of the described examples. The instructions may be stored on any computer-readable medium, such as any non-transitory computer-readable medium.

[0085] For example, the input interface 606 can receive the imaging data 306, and the processor 602 can implement the above reference Figures 1 to 3The method 300 described above can be used to generate a natural language text describing the changes, and the processor 602 can output the natural language text describing the changes or the second medical text report 314 via the output interface 608. In some examples, the natural language text describing the changes can be transmitted to a structured storage device (not shown) so that the output data is stored in the structured storage device. In some examples, the natural language text describing the changes or the second medical text report 314 can be transmitted to a display device (not shown) to allow a user to review the output data. In some examples, the output data can be stored in the memory 604 alternatively or additionally.

[0086] As another example, alternatively or additionally, the input interface 606 may receive training data, and the processor 608 may implement the above reference Figure 4 and Figure 5 The training of the machine learning model described in the embodiment of the present invention is performed, and the processor 602 can output the trained machine learning model or data representing the trained machine learning model via the output interface 608. In some examples, the trained machine learning model or data representing the trained machine learning model can be stored in an external storage device (not shown) or transmitted to another computer (not shown) for use by other computers (not shown) to perform the above reference. Figures 1 to 3 In some examples, the trained machine learning model or data representing the trained machine learning model may be stored in the memory 604 or another local storage device of the device 600, for example, for the device 600 to implement the above reference. Figures 1 to 3 A method 300 of any of the described examples.

[0087] The apparatus 600 may be implemented as a processing system and / or a computer. Figures 1 to 5 The methods of any of the described examples are computer-implemented methods, and these methods may be implemented by the apparatus 600 .

[0088] Although in some of the above examples, the imaging data is differential imaging data representing the difference between the first medical imaging data and the second medical imaging data, it should be understood that in other examples, the imaging data may instead represent the first medical imaging data and the second medical imaging data. For example, obtaining the imaging data may include concatenating the first medical imaging data and the second medical imaging data together. The method may then include: instead of inputting the differential imaging data into the first trained machine learning model, inputting the concatenated imaging data into the first trained machine learning model. By means of training using a further loss function, the first trained machine learning model is still able to generate an image feature vector representing the difference between the first medical imaging data and the second medical imaging data.

[0089] The same applies to the training imaging data, that is, instead of the training method 400, 400a using training difference imaging data representing the difference between the first training medical imaging data and the second training medical imaging data, the method may use training imaging data representing the first training medical imaging data and the second training medical imaging data. For example, each training imaging data set may include concatenating the first training medical imaging data and the second training medical imaging data together.

[0090] According to a first aspect of the present framework, a computer-implemented method for generating natural language text describing changes between first medical imaging data and second medical imaging data is provided, the method comprising: obtaining imaging data representing the first medical imaging data and the second medical imaging data or representing the difference between the first medical imaging data and the second medical imaging data; inputting the imaging data into a first trained machine learning model to generate an image feature vector representing the difference between the first medical imaging data and the second medical imaging data; and inputting data representing the image feature vector into a second trained machine learning model to generate natural language text describing changes between the first medical imaging data and the second medical imaging data.

[0091] Optionally, the first trained machine learning model has been trained to generate an image feature vector based on an input of given differential imaging data representing a difference between given first medical imaging data and given second medical imaging data.

[0092] Optionally, the second trained machine learning model has been trained to generate natural language text describing changes between given first medical imaging data and given second medical imaging data based on an input of given data representing image feature vectors.

[0093] Optionally, the method includes generating a second medical text report associated with the second medical imaging data based on the first medical text report associated with the first medical imaging data and the natural language text describing the change.

[0094] Optionally, generating the second medical text report includes inputting the first medical text report and the natural language text describing the changes into a natural language processing model.

[0095] Optionally, the natural language processing model is a trained natural language processing machine learning model.

[0096] Optionally, the trained natural language processing machine model is a trained large language model.

[0097] Optionally, the trained large language model has been trained to generate a second medical text report based on input given the first medical text report, given natural language text describing the change, and a text prompt including instructions to generate the second medical text report.

[0098] Optionally, the method includes transmitting the natural language text and / or the second medical text report describing the change to a storage device for storage at the storage device.

[0099] Optionally, the method includes transmitting the natural language text and / or the second medical text report describing the change to a display device for display on the display device.

[0100] Optionally, the first medical imaging data is of the patient and has been captured at a first time, and the second medical imaging data is of the patient and has been captured at a second, later time.

[0101] Optionally, the first medical imaging data and the second medical imaging data are for a specific region of the patient.

[0102] Optionally, the change indicates the occurrence, progression, regression or disappearance of a medical abnormality.

[0103] Optionally, the method includes: obtaining first medical imaging data and second medical imaging data, the first medical imaging data including multiple first intensity values, and the second medical imaging data including multiple corresponding second intensity values; and for each of the multiple first intensity values, comparing the first intensity value with the corresponding second intensity value to obtain a differential intensity value, wherein the imaging data includes the obtained differential intensity value.

[0104] Optionally, comparing the first intensity value to the second intensity value comprises performing a subtraction operation using the first intensity value and the second intensity value.

[0105] Optionally, the method comprises registering the first medical imaging data with the second medical imaging data by converting the plurality of first intensity values ​​and the plurality of second intensity values ​​into a coordinate system before comparing the first intensity values ​​with the second intensity values.

[0106] Optionally, generating an image feature vector includes: inputting imaging data into a first group of trained machine learning models of a first trained machine learning model to generate one or more attention maps; and inputting data representing the one or more attention maps into a second group of trained machine learning models of the first trained machine learning model to generate an image feature vector.

[0107] Optionally, the method comprises generating data representing the one or more attention maps by applying a dimensionality reduction operation to each of the one or more attention maps.

[0108] Optionally, the one or more attention maps include a plurality of attention maps, and each attention map relates to a different segment of the imaging data.

[0109] According to a second aspect of the present framework, a computer-implemented training method for training a machine learning model is provided for generating natural language text describing changes between first medical imaging data and second medical imaging data, the method comprising: providing a machine learning model, the machine learning model comprising: (a) a first machine learning model, configured to generate an image feature vector based on an input of given imaging data representing a given first medical imaging data and a given second medical imaging data or representing the difference between the given first medical imaging data and the given second medical imaging data, and (b) a second machine learning model, configured to generate a natural language text describing changes between the given first medical imaging data and the given second medical imaging data based on an input of data representing the image feature vector; providing training data comprising a plurality of training imaging data sets, each training imaging data set representing a first training medical imaging data and a second training medical imaging data or representing the difference between the first training medical imaging data and the second training medical imaging data, for each training imaging data set, the training data further comprising a ground truth natural language text describing changes between the first training medical imaging data and the second training medical imaging data; and training the machine learning model based on the training data so as to minimize a loss function between the natural language text generated by the machine learning model for the training imaging data set and the corresponding ground truth natural language text generated for the training imaging data set.

[0110] Optionally, providing training data includes generating ground truth natural language text for each training imaging data set by inputting a first training medical text report associated with the first training medical imaging data and a second training medical text report associated with the second training medical imaging data into a third trained machine learning model.

[0111] Optionally, the third trained machine learning model is a trained large language model.

[0112] Optionally, the trained large language model has been trained to generate ground truth natural language text based on input of a given first training medical text report associated with given first training medical imaging data, a given second training medical text report associated with given second training medical imaging data, and a text prompt comprising instructions to generate ground truth natural language text.

[0113] Optionally, for each training imaging data set, the training data includes training data representing a first training medical text report and a second training medical text report; the machine learning model includes (c) a text encoder machine learning model configured to generate a text feature vector based on input representing a given first medical text report associated with a given first medical imaging data and a given second medical text report associated with a given second medical imaging data; and the method includes training the machine learning model based on the training data so as to minimize a further loss function between the following feature vectors: the text feature vectors generated by the text encoder machine learning model for the first training medical text report and the second training medical text report and the image feature vector generated by the first machine learning model for the corresponding training imaging data set.

[0114] Optionally, the further loss function is a contrastive loss function.

[0115] Optionally, minimizing the contrast loss function includes: reducing the distance in a common embedding space between: an image feature vector generated for a given set in multiple training imaging datasets and a text feature vector generated for the given set; and / or increasing the distance in a common embedding space between: an image feature vector generated for a first set in multiple training imaging datasets and one of the text feature vectors generated for the first set and an image feature vector generated for a second set in multiple training imaging datasets and one of the text feature vectors generated for the second set.

[0116] According to a third aspect of the present framework, an apparatus is provided, the apparatus being configured to perform the method according to the first aspect and / or the second aspect.

[0117] Optionally, the apparatus comprises a storage device and / or a display device.

[0118] According to a fourth aspect of the present framework, there is provided a computer program which, when executed by a computer, causes the computer to perform the method according to the first aspect and / or the second aspect.

[0119] The foregoing examples have been provided for the purpose of explanation only and are in no way to be construed as limiting the present framework disclosed herein. Although the present invention has been described with reference to various embodiments, it should be understood that the words used herein are words of description and illustration, rather than words of limitation. Further, although the present invention has been described herein with reference to specific means, materials and embodiments, the present invention is not intended to be limited to the details disclosed herein, but rather the present invention extends to all functionally equivalent structures, methods and uses, such as structures, methods and uses within the scope of the appended claims. Those skilled in the art who benefit from the teachings of this specification may make many modifications thereto, and changes may be made without departing from the scope and spirit of the present invention in its aspects.

Claims

1. A computer-implemented method for generating natural language text describing changes between first medical imaging data and second medical imaging data, the method comprising: obtaining imaging data representing the first medical imaging data and the second medical imaging data or representing a difference between the first medical imaging data and the second medical imaging data; inputting the imaging data into a first trained machine learning model to generate an image feature vector representing a difference between the first medical imaging data and the second medical imaging data; as well as Data representing the image feature vectors is input into a second trained machine learning model to generate natural language text describing changes between the first medical imaging data and the second medical imaging data.

2. The method according to claim 1, comprising: A second medical text report associated with the second medical imaging data is generated based on the first medical text report associated with the first medical imaging data and the natural language text describing the change. 3 . The method of claim 1 , wherein the first medical imaging data is of a patient and has been captured at a first time, and the second medical imaging data is of a patient and has been captured at a second, later time.

4. The method of claim 1, wherein the change represents the onset, progression, regression, or disappearance of a medical abnormality.

5. The method according to claim 1, comprising: obtaining first medical imaging data and second medical imaging data, the first medical imaging data comprising a plurality of first intensity values ​​and the second medical imaging data comprising a plurality of corresponding second intensity values; as well as for each of the plurality of first intensity values, comparing the first intensity value with a corresponding second intensity value to obtain a differential intensity value, The imaging data includes the obtained differential intensity values. The method of claim 5 , wherein comparing the first intensity value to the second intensity value comprises performing a subtraction operation using the first intensity value and the second intensity value.

7. The method according to claim 1, wherein generating an image feature vector comprises: inputting the imaging data into a first set of trained machine learning models of a first trained machine learning model to generate one or more attention maps; as well as Inputting data representing the one or more attention maps into a second component of the first trained machine learning model to generate an image feature vector.

8. The method of claim 7, wherein the method comprises generating data representing the one or more attention maps by applying a dimensionality reduction operation to each of the one or more attention maps.

9. The method of claim 7, wherein the one or more attention maps comprises a plurality of attention maps, and each attention map relates to a different segment of the imaging data.

10. A computer-implemented training method for training a machine learning model for generating natural language text describing changes between first medical imaging data and second medical imaging data, the method comprising: A machine learning model is provided, wherein the machine learning model comprises: (a) a first machine learning model configured to generate an image feature vector based on an input of given imaging data representing given first medical imaging data and given second medical imaging data or representing a difference between given first medical imaging data and given second medical imaging data, and (b) a second machine learning model configured to generate natural language text describing changes between given first medical imaging data and given second medical imaging data based on input of data representing said image feature vectors; providing training data comprising a plurality of training imaging data sets, each training imaging data set representing first training medical imaging data and second training medical imaging data or representing a difference between the first training medical imaging data and the second training medical imaging data, the training data further comprising, for each training imaging data set, ground truth natural language text describing a change between the first training medical imaging data and the second training medical imaging data; and The machine learning model is trained based on the training data so as to minimize a loss function between natural language text generated by the machine learning model for the training imaging dataset and corresponding ground truth natural language text generated for the training imaging dataset.

11. The method of claim 10, wherein providing training data comprises generating ground truth natural language text for each training imaging data set by inputting a first training medical text report associated with the first training medical imaging data and a second training medical text report associated with the second training medical imaging data into a third trained machine learning model.

12. The method according to claim 11, wherein: For each training imaging data set, the training data includes training data representing a first training medical text report and a second training medical text report; The machine learning model includes: (c) a text encoder machine learning model configured to generate, based on input given data representing: a given first medical text report associated with the given first medical imaging data and a given second medical text report associated with the given second medical imaging data, text feature vector; and The method includes training the machine learning model based on the training data.

13. The method of claim 12, wherein training the machine learning model based on the training data comprises training the machine learning model to minimize a further loss function between the following feature vectors: a text feature vector generated by the text encoder machine learning model for the first training medical text report and the second training medical text report, and The first machine learning model generates an image feature vector for the corresponding training imaging dataset.

14. A system for generating natural language text describing changes between first medical imaging data and second medical imaging data: a non-transitory memory device for storing computer readable program code; as well as a processor in communication with the non-transitory memory device, the processor operating with the computer readable program code to perform steps including obtaining imaging data representing the first medical imaging data and the second medical imaging data or representing a difference between the first medical imaging data and the second medical imaging data, inputting the imaging data into a first trained machine learning model to generate an image feature vector representing a difference between the first medical imaging data and the second medical imaging data; as well as Data representing the image feature vectors is input into a second trained machine learning model to generate natural language text describing changes between the first medical imaging data and the second medical imaging data.

15. The system of claim 14, wherein the processor operates with the computer readable program code to generate a second medical text report associated with the second medical imaging data based on a first medical text report associated with the first medical imaging data and a natural language text describing the change.

16. The system of claim 14, wherein the first medical imaging data is for a patient and has been captured at a first time, and the second medical imaging data is for a patient and has been captured at a second later time.

17. The system of claim 14, wherein the change indicates the onset, progression, regression, or disappearance of a medical abnormality.

18. The system of claim 14, wherein the processor operates with the computer readable program code to further perform additional steps, including obtaining first medical imaging data and second medical imaging data, the first medical imaging data comprising a plurality of first intensity values ​​and the second medical imaging data comprising a plurality of corresponding second intensity values; and for each of the plurality of first intensity values, comparing the first intensity value with a corresponding second intensity value to obtain a differential intensity value, The imaging data includes the obtained differential intensity values.

19. The system of claim 18, wherein comparing the first intensity value to the second intensity value comprises performing a subtraction operation using the first intensity value and the second intensity value.

20. The system of claim 14, wherein generating an image feature vector comprises: inputting the imaging data into a first set of trained machine learning models of a first trained machine learning model to generate one or more attention maps; as well as Inputting data representing the one or more attention maps into a second component of the first trained machine learning model to generate an image feature vector.

Images