Determining data representative of changes between medical imaging datasets

Abstract

A computer-implemented framework for determining data representative of a change between first medical imaging data and second medical imaging data is disclosed. Imaging data representing a difference between the first medical imaging data and the second medical imaging data is obtained. The imaging data is input into a trained image processing machine learning model to generate an image feature vector. A plurality of textual feature vectors are obtained, each textual feature vector representing natural language text describing a respective change in medical imaging data. For each of the plurality of textual feature vectors, a similarity metric indicative of a similarity to the image feature vector is determined. And selecting a text feature vector. Data representing a change between the first medical imaging data and the second medical imaging data is determined based on the selected textual feature vector.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to European patent application No. 23191390.6 filed on August 14, 2023, the contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to determining data representing changes between two medical imaging data sets. Background Art

[0003] Subsequent medical images of a patient may be taken after an initial baseline medical image is taken, for example, to assess any changes that have occurred in the patient between the time of the baseline medical image and the time of the subsequent medical image. For example, when a patient is diagnosed with a disease using a baseline medical image, follow-up medical imaging is often recommended to monitor the progression of the disease. After the follow-up images are taken, a radiologist may review the follow-up images and the baseline images and write a report describing any progression of the disease. However, interpretation of the follow-up images is a time-consuming task.

[0004] In addition, when comparing a baseline image taken at diagnosis with a subsequent image taken later, the image may include artifacts that are not related to disease progression. For example, artifacts may be visible where the illumination or imaging angle or other imaging parameters used to capture the baseline image differ from those used to capture the subsequent image. Other unrelated changes may be apparent in the image. This may make interpretation of the subsequent and baseline images difficult.

[0005] It would be desirable to provide a computer implemented method for interpreting subsequent medical images or at least for providing assistance in the interpretation of subsequent medical images. Summary of the invention

[0006] A computer-implemented framework for determining data representing a change between first medical imaging data and second medical imaging data is described herein. Imaging data representing the first medical imaging data and the second medical imaging data or the difference between the first medical imaging data and the second medical imaging data is obtained. The imaging data is input into a trained image processing machine learning model to generate an image feature vector representing the difference between the first medical imaging data and the second medical imaging data. A plurality of text feature vectors are obtained, each of which represents a natural language text describing a corresponding change in the medical imaging data. For each of the plurality of text feature vectors, a similarity measure indicating the similarity between the image feature vector and the text feature vector is determined. Based on the determined similarity measure, a text feature vector is selected from the plurality of text feature vectors. Based on the selected text feature vector, data representing the change between the first medical imaging data and the second medical imaging data is determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] A more complete understanding of the present disclosure and its many attendant aspects will be readily obtained as the disclosure and its many attendant aspects become better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings.

[0008] Figure 1 is a flow chart schematically illustrating an example method for determining data representing a change between first medical imaging data and second medical imaging data;

[0009] Figure 2a is a flow chart schematically illustrating an example method for obtaining differential imaging data from first medical imaging data and second medical imaging data;

[0010] Figure 2b is an illustration of the difficulties associated with identifying medical abnormalities from differential imaging data obtained by visual inspection according to one example;

[0011] Figure 3 is schematically shown in more detail according to Figure 1 a flow chart of an example method of the method shown;

[0012] Figure 4 is a schematic diagram showing the process of training a machine learning model to perform Figure 1 a flow chart of an example method of the method shown;

[0013] Figure 5 is a diagram schematically illustrating an apparatus according to an example. DETAILED DESCRIPTION

[0014] Independent of the use of the grammatical term, individuals with both male and female identities are also included in the term.

[0015] refer to Figure 1 , a computer-implemented method 100 for determining data representing changes between first medical imaging data 302 and second medical imaging data 304 is shown.

[0016] In summary, the method 100 includes: in step 102, obtaining imaging data representing first medical imaging data 302 and second medical imaging data 304, or representing the difference between the first medical imaging data 302 and the second medical imaging data 304 (see Figure 2a ); In step 104, the imaging data is input into the trained image processing machine learning model 340 to generate an image feature vector Zi representing the difference between the first medical imaging data 302 and the second medical imaging data 304 (e.g., Figure 3In step 106, a plurality of text feature vectors Zt are obtained, each of which represents a natural language text describing a corresponding change in the medical imaging data (e.g., Figure 3 In step 108, a similarity metric 352 (eg, Figure 3 In step 110, based on the determined similarity measure 352, a text feature vector 380 (eg, Figure 3 380); and in step 112, based on the selected text feature vector 380, determining data representing a change between the first medical imaging data 302 and the second medical imaging data 304 (eg, Figure 3 natural language text 382 or output image data 322 describing the changes).

[0017] Thus, data representing changes between the first medical imaging data 302 and the second medical imaging data 304 are generated.

[0018] By generating an image feature vector based on the imaging data using a trained image processing machine learning model, an image feature vector encoding features related to the change between the first medical imaging data 302 and the second medical imaging data 304 is generated. Thus, for example, the progression of a patient's disease depicted in the first medical imaging data 302 and the second medical imaging data 304 can be extracted and encoded into the image feature vector. Based on the image feature vector, a text feature vector representing a natural language text describing the change 382 in the medical imaging data is selected. Since the selection is performed by determining the similarity between the image feature vector and the text feature vector Zt, the selected text feature vector represents a natural language text 382 that can accurately describe the difference between the first medical imaging data 302 and the second medical imaging data 304. For example, the selected text feature vector can represent text that accurately describes the progression of a patient's disease indicated by the second medical imaging data compared to the first medical imaging data, such as the degree of progression. The text feature vector is used to determine data representing the change between the first medical imaging data 302 and the second medical imaging data 304. Thus, basing the determination of the data representing the change on the selected text feature vector allows the data representing the change to accurately represent the difference between the first medical imaging data 302 and the second medical imaging data 304. In an example, the data representing the change may include natural language text 382 corresponding to the selected text feature vector or data derived therefrom, the result of determining whether the natural language text 382 corresponding to the selected text feature vector represents a significant change 384, and / or output image data 322 representing the change between the first medical imaging data 302 and the second medical imaging data 304. Thus, the data representing the change between the first medical imaging data 302 and the second medical imaging data 304 may allow for automated and / or assisted interpretation of subsequent images. For example, radiologists may use the data representing the change to guide their interpretation and / or reporting of subsequent images.

[0019] The medical professional may use the data representing the changes in a variety of ways. For example, the medical professional may recommend a course of treatment, or determine that further follow-up imaging is needed.

[0020] Reference now Figure 3 The example method 300 is described in detail. Figure 3 The method 300 is Figure 1 A specific example of method 100 includes Figure 1 The method 100 comprises steps.

[0021] As described above, the method 100 includes, in step 102, obtaining imaging data representing the first medical imaging data 302 and the second medical imaging data 304, or representing the difference between the first medical imaging data 302 and the second medical imaging data 304. Figure 3 In the example of , the imaging data includes differential imaging data 306 representing a difference between the first medical imaging data 302 and the second medical imaging data 304 .

[0022] Each of the first medical imaging data 302 and the second medical imaging data 304 may include an array of elements, each element having a value. The elements may be pixels, each pixel having at least one value. The at least one value may correspond to or otherwise represent an output signal of a medical imaging technique used to generate the medical imaging data. For example, for magnetic resonance imaging, the value of an element (e.g., a pixel) may correspond to or represent the rate at which an excited nucleus returns to a state of equilibrium in a region corresponding to the element. As another example, in SPECT imaging, the value of an element may correspond to or represent the blood flow in a capillary represented by a given pixel. As another example, in CT imaging, the value of an element may correspond to or represent the amount of X-ray attenuation. In some examples, each element may have only one value. However, in other examples, each element may have multiple values ​​or be otherwise associated with multiple values. For example, multiple values ​​of a given element may represent the values ​​of corresponding multiple signal channels. For example, each signal channel may represent different medical imaging signals or properties of an imaging object. In some examples, at least one value may include an element (e.g., pixel) intensity value. For example, an output signal from medical imaging may be mapped to a pixel intensity value, such as a value within a defined intensity value range. For example, for a grayscale image, the intensity value may correspond to a value in the range of 0 to 255, e.g., where 0 represents a "black" pixel and 255 represents a "white" pixel. As another example, such as in the case of USHORT medical image data, the intensity value may correspond to a value in the range of 0 to 65536. As another example, in a color image (e.g., where different colors represent different properties of the imaged object), each pixel may have three intensity values, e.g., one each for the red, green, and blue channels. It will be appreciated that other values ​​may be used.

[0023] In some examples, the first medical imaging data 302 is for a patient and has been captured at a first time, and the second medical imaging data 304 is for the patient and has been captured at a second, later time. For example, the first medical imaging data 302 may form part of a baseline study of the patient. Figure 3As shown, a first medical text report 312 can be written for the baseline study. The second medical imaging data 304 can form part of a follow-up study of the patient. The follow-up study can be intended to track the progress of the medical abnormality 308, and / or identify the occurrence or progress of the medical abnormality 308 after the baseline study. In this example, the first medical imaging data 302 and the second medical imaging data 304 are directed to a specific area of ​​the patient. For example, the first medical imaging data 302 and the second medical imaging data 304 can be used for the abdominal area of ​​the patient, or for the limbs of the patient. In other examples, the first medical imaging data 302 and the second medical imaging data 304 can be directed to the whole body of the patient; that is, they can represent a whole-body image of the patient. The above-mentioned properties of the first medical imaging data 302 and the second medical imaging data 304 apply to any two medical imaging data sets described as "first" and "second" as described herein.

[0024] Determining data representative of a change, where the first medical imaging data 302 is for the patient and has been captured at a first time, and the second medical imaging data 304 is for the patient and has been captured at a second, later time, enables computer-implemented interpretation of subsequent medical images. This, in turn, enables monitoring of the progression of the patient's disease without necessarily requiring a human to analyze the first medical imaging data 302 and the second medical imaging data 304. Alternatively or additionally, this may enable computer-implemented assistance in the interpretation and / or reporting of subsequent medical images or monitoring of the progression of the patient's disease by a medical professional.

[0025] In some examples, the method 300 includes obtaining first medical imaging data 302 and second medical imaging data 304. For example, the first medical imaging data 302 and the second medical imaging data 304 may be obtained from a memory device such as a memory device (see, e.g., Figure 5 The first medical imaging data 302 may include a plurality of first intensity values, and the second medical imaging data 304 may include a plurality of corresponding second intensity values. That is, each first intensity value in the first medical imaging data 302 corresponds to a second intensity value in the second medical imaging data 304. Each first intensity value may be associated with a corresponding pixel position, and each second intensity value may be associated with one of the pixel positions to provide such a correspondence.

[0026] In this example, the method 300 includes image preprocessing 332 of the first medical imaging data 302 and the second medical imaging data 304. For example, the image preprocessing 332 may include image registration. Image registration generally involves transforming a plurality of first intensity values ​​and a plurality of second intensity values ​​into a coordinate system. Image registration may include associating each first intensity value with its corresponding second intensity value, and corresponding pixel locations may be defined during image registration.

[0027] In some examples, method 300 includes, for each of the plurality of first intensity values, comparing the first intensity value to a corresponding second intensity value to obtain a differential intensity value. In these examples, differential imaging data 306 includes the obtained differential intensity value. For example, Figure 2a As shown, comparing the first intensity value to the second intensity value may include performing a subtraction operation using the first intensity value and the second intensity value. For example, comparing the first intensity value to the second intensity value may include subtracting the second intensity value from the first intensity value, or vice versa. Subtracting the second intensity value from the first intensity value, or vice versa, may allow for a computationally simple method to obtain differential imaging data 306. Figure 2a and 2b In the illustrated differential imaging data 306, regions where the intensity values ​​of the second medical imaging data 304 are lower than the intensity values ​​of the first medical imaging data 302 are shown in light gray, and regions where the intensity values ​​of the second medical imaging data 304 are higher than the intensity values ​​of the first medical imaging data 302 are shown in dark gray. A medical abnormality 308 is shown on each of the first medical imaging data 302, the second medical imaging data 304, and the differential medical imaging data. Between the time when the first medical imaging data 302 was captured and the time when the second medical imaging data 304 was captured, the medical abnormality 308 has undergone a subtle change in appearance.

[0028] Figure 2b Two copies of differential medical imaging data are shown, with the location of the abnormality indicated on the second copy. Figure 2b As can be appreciated from the differential imaging data 306 shown, it may be difficult to identify the medical abnormality 308 from only a visual inspection of the differential imaging data 306. In fact, it may also be difficult for a computer vision system to identify the medical abnormality 308 using only the differential imaging data 306. Some examples described herein allow for determining data representing changes between the first medical imaging data 302 and the second medical imaging data 304.

[0029] As described above, the method 100 includes, at step 104, inputting imaging data into the trained image processing machine learning model 340 to generate an image feature vector representing the difference between the first medical imaging data 302 and the second medical imaging data 304. In this example, the imaging data is differential imaging data 306, and the image feature vector represents the differential imaging data 306.

[0030] In this example, the trained image processing machine learning model 340 has been trained to generate an image feature vector representing the difference between the given first medical imaging data and the given second medical imaging data based on the given imaging data. In this example, the given imaging data represents the difference between the given first medical imaging data and the given second medical imaging data. However, in other examples, the given imaging data may represent the given first medical imaging data and the given second medical imaging data. For example, the given imaging data may include the concatenation of the given first medical imaging data and the given second medical imaging data. Figure 4 The training process is described in more detail.Any of the machine learning models described here may include a neural network.

[0031] The image feature vector can accurately represent the difference between the first medical imaging data 302 and the second medical imaging data 304. Figure 4 An example manner in which the image processing machine learning model 440 may be trained to generate image feature vectors is explained.

[0032] As described above, the method 100 includes obtaining a plurality of text feature vectors Zt at step 106, each text feature vector representing a natural language text describing a corresponding change in the medical imaging data.

[0033] Two exemplary methods for generating text feature vectors representing natural language texts that describe corresponding changes in medical imaging data are provided here. In some examples, one of the exemplary methods can be used to generate text feature vectors. In other examples, a first method can be used to generate one or more such text feature vectors, and a second method is used to generate another one or more such text feature vectors.

[0034] The first method includes inputting a first medical text report 312 associated with the first medical imaging data 302 and a text prompt 316 into a trained large language model 352 , the text prompt 316 including instructions for generating natural language text describing possible changes to the first medical imaging data 302 based on the first medical text report 312 .

[0035] The trained large language model 352 may be ChatGPT or GPT-4. In a first approach, the trained large language model 352 has been trained to (among other things) generate natural language text describing possible variations based on a given first medical text report associated with a given first medical imaging data and an input of a text prompt 316, the text prompt 316 including instructions to generate natural language text describing possible variations of the given first medical imaging data based on the given first medical text report.

[0036] In this example, the first medical text report 312 includes at least one sentence, such as "there is a low-density lesion in the right lobe of the liver." More generally, a medical text report "associated with" specific medical imaging data means that the medical text report includes at least one sentence describing the medical imaging data. The medical text report may be written by a radiologist who reviewed the medical imaging data.

[0037] As described above, in the first method, the text prompt 316 includes instructions for generating natural language text describing possible changes of the first medical imaging data 302 based on the first medical text report 312. In this example, the "possible changes" of the specific imaging data include changes that may occur on the imaging data. In this example, the natural language text describing the possible changes of the first medical imaging data 302 includes at least one sentence, such as "the size of the low-density lesion in the right lobe of the liver increases". In any case, the change (as referred to anywhere in this article) can represent the occurrence or progression of the medical abnormality 308. One or more text prompts 316 can be used to generate multiple natural language text sets describing the corresponding possible changes of the first medical imaging data 302. The same text prompt 316 can be used with different first medical text reports (for example, for different patients) to generate a natural language text set describing the possible changes of each such first medical text report. The text prompt 316 can be referred to as a prompt template. For example, the first text prompt 316 can be: “Below is a radiology report describing the patient’s medical images. Identify a possible theoretical change in the patient's disease and describe that change. [First Medical Text Report 312]" And the trained large language model 352 can then generate natural language text describing the first possible variation.

[0038] After generating the first possible change, in the same conversation with the large language model, a second text prompt 316 may be: "Now determine another possible theoretical change in the patient's disease, and describe that change."

[0039] Other suitable prompts may be used, such as: "Identify 10 possible theoretical variations of the patient's disease reported in the following radiology report, and describe each variation [first medical text report 312]". This may result in the output of 10 natural language text sets that describe the 10 possible theoretical variations of the patient's disease reported in the first medical text report. It should be understood that 10 is an example, and in principle any number may be used. This is an example of how a text prompt 316 can generate multiple natural language text sets.

[0040] As another example, other suitable text prompt(s) could be of the form “Generate a sentence that would represent a {significant} / {mild} {worsening} / {improvement} of the patient's medical condition reported in the following radiology report [first medical text report 312]”. For example, there may be four separate text prompts corresponding to "significantly worsened," "significantly improved," "mildly worsened," and "mildly improved," each prompt producing a corresponding set of natural language text.

[0041] As another example, there may be a text prompt 316 that includes instructions to generate four sets of natural language text. For example, a text prompt 316 may be "Generate four sentences representing respectively marked worsening, mild worsening, marked improvement, and mild improvement of the patient's medical condition reported in the following radiology report [first medical text report 312]". In either case, a plurality of natural language text sets are generated that respectively describe different possible variations of the patient's disease as described in the first medical text report 312 .

[0042] In the first approach, the possible change does not necessarily describe the actual change that occurred between the two medical imaging datasets. Instead, it only describes the change that may have occurred.

[0043] By using a first method to "guess" possible changes based on the first medical text report 312, a text feature vector (as further described below) can be generated, the content of which is relevant to the first medical text report 312. The first medical text report 312 generally relates to a specific medical abnormality, such as the liver lesion described above. In an example in which the first medical imaging data 302 and the second medical imaging data 304 relate to the same patient, an important factor to be considered in the second medical imaging data 304 is the progression of the medical abnormality 308 described in the first medical text report 312. Therefore, generating natural language text describing possible changes based on the first medical text report 312 can help ensure that the natural language text 382 describes the relevant changes of the first medical imaging data 302. In addition, since the selected text feature vector 380 is selected from a plurality of text feature vectors Zt based on a similarity measure 352 with the image feature vector, a text feature vector representing text describing the most applicable possible change can be identified.

[0044] In the second method, each of the plurality of text feature vectors represents a natural language text describing a change between the corresponding other first medical imaging data and the corresponding other second medical imaging data. In this example, a given set of other first medical imaging data 302 and other second medical imaging data 304 is for another patient, the patient being different from the patient associated with the first medical imaging data 302 and the second medical imaging data 304, and the patient being different from another patient associated with another set of other first medical imaging data sets and other second medical imaging data.

[0045] In a second method, each of the plurality of text feature vectors may be generated before obtaining the imaging data 302, 304. For example, each of the plurality of text feature vectors may be generated at the end of a training phase (see below). Figure 4 described in more detail). Multiple text feature vectors can be stored in a memory device and provide a "dictionary" of text feature vectors, each text feature vector representing a corresponding natural language text describing a corresponding different change in the medical imaging data. In this example, to generate each of the multiple text feature vectors, the second method includes inputting a corresponding first medical text report 372 associated with the corresponding other first medical imaging data, a corresponding second medical text report 374 associated with the corresponding other second medical imaging data, and a text prompt 376 into the trained large language model 352, the text prompt 376 including instructions for generating natural language text describing the changes between the corresponding other first medical imaging data and the corresponding other second medical imaging data based on the corresponding other first medical text report 372 and the corresponding other second medical text report 374.

[0046] In this example, the trained large language model 352 used in the second method is the same model as the trained large language model 352 used in the first method. In the second method, the trained large language model 352 has been trained to (among other things) generate natural language text describing the change based on a given first medical text report associated with the given first medical imaging data, a given second medical text report associated with the given second medical imaging data, and an input of a text prompt 376, the text prompt 376 including instructions to generate natural language text describing the change between the given first medical imaging data and the given second medical imaging data based on the given first medical text report and the given second medical text report.

[0047] In this example, in the second approach, the text hint 376 may be the same for all text feature vectors generated. An example text hint 376 is: “Write the following description of any differences between the baseline and follow-up radiology reports. Here is the baseline radiology report. [Corresponding First Medical Text Report 372] Here is the follow-up radiology report. [Corresponding Second Medical Text Report 374]", and "The following radiology reports each describe two medical images of the same patient taken at different times. This is the first radiology report. [Corresponding first medical text report 372] This is the Second Radiology Report. [Corresponding Second Medical Text Report 374] Write a description of any changes that occurred between the time the medical images were taken”.

[0048] As described above, in the second method, before obtaining imaging data, a natural language text set describing corresponding changes can be generated for many such first medical text reports 372 and second medical text reports 374 to form a dictionary of changes that may occur between the first medical imaging data set and the second medical imaging data set.

[0049] In the second method, the natural language text describing the changes can describe the actual changes that occurred between the other first medical imaging data and the other second medical imaging data. This allows the generation of text feature vectors, each of which is associated with a real-world example of a change in the medical imaging data, so that each text feature vector represents a change that may have occurred between the first medical imaging data 302 and the second medical imaging data 304. Since the selected text feature vector 380 is selected based on the similarity measure 352 with the image feature vector Zi, a text feature vector can be identified that represents text that accurately describes the changes between the first medical imaging data 302 and the second medical imaging data 304. In addition, compared to the first method, each time the method 300 is used to determine data representing changes between a given set of first medical imaging data and second medical imaging data, it is not necessary to generate a new set of natural language text describing possible changes.

[0050] In this example, in both the first method and the second method, generating the text feature vector includes using a trained text encoder 350 and generating the text feature vector based on the corresponding generated natural language text describing the changes. In the first method, the text feature vectors each relate to different corresponding possible changes, while in the second method, the text feature vectors relate to the actual changes between the medical imaging data sets. The trained text encoder 350 has been trained to generate the text feature vector based on the natural language text describing the changes.

[0051] In the exemplary second method described above, a large language model is used to generate a set of natural language texts, each of which describes the changes between a corresponding first medical text report 372 associated with the corresponding other first medical imaging data and a corresponding second medical text report 374 associated with the corresponding other second medical imaging data. The trained text encoder 350 is then used to generate a corresponding text feature vector for each of these natural language text sets. However, in some examples, it is not necessary to use a large language model. For example, a variation of the second method may include using a trained text encoder 350 to generate a text feature vector representing a natural language text describing the changes between the corresponding other first medical imaging data and the corresponding other second medical imaging data based on the corresponding first medical text report 372 and the corresponding second medical text report 374. For example, the first corresponding medical text report 372 and the second corresponding medical text report 374 can be spliced ​​and input into the trained text encoder 350, and the trained text encoder can output a text feature vector representing a natural language text describing the changes. As will be referred to in Figure 4As described, in such an example, the trained text encoder 350 may be trained to generate a text feature vector that is aligned with the image feature vector in the embedding space and may therefore represent changes between the first medical imaging data 302 and the second medical imaging data 304 .

[0052] As described above, the method 100 includes, at step 108 , determining a similarity measure 352 indicating a similarity between the image feature vector and the text feature vector for each of the plurality of text feature vectors Zt.

[0053] In this example, the image feature vector and the plurality of text feature vectors Zt are represented in a common embedding space, and the similarity measure 352 is a distance measure in the common embedding space between the image feature vector and the text feature vector. Any distance measure described herein may be, for example, a Euclidean distance or a cosine distance. By representing the image feature vector and the plurality of text feature vectors Zt in a common embedding space, the distance measure may be used to identify text feature vectors that are closely aligned with the image feature vector in terms of their content. This enables the selection of text feature vectors that represent text that accurately describes the changes between the first medical imaging data 302 and the second medical imaging data 304.

[0054] As described above, the method 100 includes, at step 110 , selecting a text feature vector 380 from a plurality of text feature vectors Zt based on the determined similarity measure 352 .

[0055] In this example, selecting a text feature vector 380 includes selecting a text feature vector corresponding to the lowest distance metric among the distance metrics from a plurality of text feature vectors Zt. That is, each distance metric has a value, and selecting a text feature vector includes selecting a text feature vector 380 having the lowest distance metric between the text feature vector and the image feature vector.

[0056] As described above, the method 100 includes, at step 112 , determining data representing changes between the first medical imaging data 302 and the second medical imaging data 304 based on the selected text feature vector 380 .

[0057] We have described here several examples of data representing a change between the first medical imaging data 302 and the second medical imaging data 304. The data representing the change may include data according to any combination of the described examples.

[0058] In a first example, the data representing the change includes the natural language text 382 represented by the selected text feature vector 380 or data derived therefrom. For example, determining the data representing the change may include determining the natural language text 382 represented by the selected text feature vector 380. The data representing the change may then include the natural language text 382 represented by the selected text feature itself, and / or data derived therefrom, such as, for example, keywords from the natural language text 382 or a rewrite of the natural language text. This may assist a radiologist in interpreting the second medical imaging data 304. For example, where the first medical imaging data 302 is associated with a patient and the second medical imaging data 304 is associated with the same patient, the natural language text 382 (or data, such as text derived therefrom) representing the change may alert a radiologist to medically relevant changes that may have occurred between those medical imaging data sets.

[0059] In a second example, determining data representing a change includes determining whether the natural language text 382 represented by the selected text feature vector 380 describes a significant change 384. Significant change 384 may refer to, for example, a medically significant change, such as the growth of a lesion. Alternatively, significant change 384 may refer to a large or noticeable change, as opposed to a minor change, such as a substantial growth of a lesion. Determining whether the natural language text 382 represented by the selected text feature vector 380 describes a significant change 384 may include performing a text mining operation on the natural language text 382 represented by the selected feature vector. Text mining may seek to determine whether the natural language text 382 includes keywords that may be associated with a significant change 384, such as "growth" or "increase". In the example where the natural language text 382 describing the change is "increase in size of a low-density lesion in the right lobe of the liver", the text mining operation may identify the word "increase" and determine that there has been a significant change 384.

[0060] Alternatively, the trained large language model 352 can be used to determine whether the natural language text 382 describes a significant change 384. For example, the trained large language model 352 can be provided with natural language text 382, ​​which is accompanied by a text prompt, such as "Does this describe a significant change? Provide a yes or no answer." The "yes" or "no" output of the trained large language model 352 can be used to determine that the natural language text 382 describes a significant or insignificant change 384, respectively. As another example, each generated text feature vector (e.g., generated according to the first or second method of generating a text feature vector as described above) can be stored in association with a label indicating whether the natural language text represented by the text feature vector describes a significant change 384. In this case, whether the natural language text 382 describes a significant change 384 is determined based on the label.

[0061] In a second example, the data representing the change includes an indication of whether the natural language text 382 represented by the selected text feature vector 380 describes a significant change 384. For example, the data representing the change may represent text that describes, for example, "there are significant changes." Alternatively, the data representing the change may include a 1-bit indicator of whether the natural language text 382 represented by the selected text feature vector 380 describes a significant change 384.

[0062] The second example enables a medical professional, for example, to be alerted to the fact that a significant change 384 has been determined to exist between the first medical imaging data 302 and the second medical imaging data 304. This may enable the medical professional to confirm the information and determine whether further follow-up imaging or treatment is required in response to the indication of whether the change is significant.

[0063] In a third example, the method 300 includes outputting output image data 322 representing a change between the first medical imaging data 302 and the second medical imaging data 304 through a trained image processing machine learning model 340 and based on the imaging data and a selected text feature vector 380, wherein the data representing the change includes the output image data 322.

[0064] For example, the method 300 may include generating a plurality of image datasets using a trained image processing machine learning model 340, each image dataset representing a change between a first medical imaging data 302 and a second medical imaging data 304, and each image dataset represented by a corresponding feature vector. Each image dataset may include an attention map. In this example, the trained image processing machine learning model 340 has been trained to generate one or more attention maps 324 based on the input of a given imaging data. By using the following reference Figure 4 The described loss function 470 trains the trained image processing machine learning model 340, which is capable of producing an attention map that indicates features of the imaging data that represent information related to the medical text report.

[0065] Each attention map can indicate the degree to which a particular portion or segment of the differential imaging data 306 contributes to the encoding or a particular portion of the encoding performed by the trained image processing machine learning model 340. For example, each attention map can relate to a different segment or different context of the imaging data. A segment may include multiple subgroups of differential intensity values. The trained image processing machine learning model 340 can segment the differential imaging data 306 into the attention maps 324 based on the contribution of each particular segment or context to the encoding. For example, the trained image processing machine learning model 340 can be configured to segment the differential imaging data 306 based on the location of possible anomalies that appear in the differential imaging data 306.

[0066] The method 300 may then include determining, for each of the plurality of image datasets (e.g., the attention map 324), using the image selector 360, a further similarity measure indicating a similarity between the corresponding feature vector representing the image and the selected text feature vector 380. In a third example, the corresponding feature vector and the selected text feature vector 380 are represented in a common embedding space, and the further similarity measure is a further distance measure in the common embedding space between the corresponding feature vector and the selected text feature vector 380.

[0067] In a third example, output image data 322 is selected from a plurality of image data sets based on the determined further similarity metric. In the third example, selecting the output image data 322 includes selecting, by the image selector 360, from the feature vector a feature vector corresponding to a further distance metric that is the lowest in the further distance metric. That is, each further distance metric has a value, and selecting the output image data 322 includes selecting the output image data 322 represented by the feature vector having the lowest further distance metric between the feature vector and the text feature vector. In an example, the output image data 322 may include an attention map corresponding to the feature vector, or data derived from the attention map. The output image data 322 may be generated by inputting the attention map to the image selector 360, rather than including an attention map corresponding to the feature vector.

[0068] In a third example, the output image data 322 provides a visual illustration of one or more portions of the medical imaging data that are associated with the changes described by the natural language text 382 represented by the selected text feature vector. In the case where the imaging data is differential imaging data 306, the output image data 322 may provide a visual illustration of the associated differences between the first medical imaging data 302 and the second medical imaging data 304, which may be used to assist a radiologist in interpreting and / or reporting the second medical imaging data 304.

[0069] In a fourth example, the data representing the change may include a saliency map or output mask data 326. For example, in the fourth example, the method 300 may include generating output image data 322 according to the third example, wherein the output image data 322 includes a plurality of output intensity values, each having a corresponding pixel location. In some examples, the output image data 322 generated according to the third example (e.g., the attention map) may have a relatively lower resolution, for example, compared to the second medical imaging data 304. In these cases, the method may include upsampling the output image data 322 generated according to the third example, for example, to produce output image data 322 having the same size as the second medical imaging data 304. Nevertheless, the upsampled output image data 322 includes a plurality of output intensity values, each having a corresponding pixel location. In the fourth example, the method 300 includes determining whether one or more of the plurality of output intensity values ​​satisfies a condition; and based on the determination, outputting output mask image data 326 representing the pixel locations of the output intensity values ​​that satisfy the condition. In the fourth example, the condition is a threshold. The threshold may include a specified intensity value.

[0070] By applying such a masking function to the output image data 322, a mask can be generated that indicates only pixel locations associated with changes described by the natural language text 382 corresponding to the selected text feature vector. Specifically, the output mask data 326 can indicate portions of the attention map that indicate medically significant aspects of the second medical imaging data 304, including those aspects that may represent medically significant changes. This can help radiologists quickly identify portions of the second medical imaging data 304 that are associated with medical abnormalities.

[0071] exist Figures 1 to 3 In the example of FIG. 3 , the trained image processing machine learning model 340 has been based on the current reference Figure 4 The training method 400 described in detail is trained. Figure 4 In the example of , the trained text encoder 350 has also been trained according to the training method 400. The trained machine learning models may include the trained image processing machine learning model 340 and the trained natural language processing machine learning model. Here, the trained natural language processing model may include the trained text encoder 350, and optionally the trained large language model 352. The trained large language model 352 (e.g., ChatGPT) itself may have been trained in a separate process and can be used in the process of the training method 400 in its trained form.

[0072] The training method 400 includes providing a machine learning model. The machine learning model includes (a) an image processing machine learning model 440, which is configured to generate an image feature vector representing the difference between given first medical imaging data and given second medical imaging data based on given imaging data representing the given first medical imaging data and given second medical imaging data or representing the difference between the given first medical imaging data and given second medical imaging data. In this example, the given imaging data is given differential imaging data, which represents the difference between the given first medical imaging data and the given second medical imaging data. The machine learning model also includes (b) a natural language processing machine learning model, which is configured to generate a text feature vector based on given text data representing a given first medical text report associated with the given first medical imaging data and a given second medical text report associated with the given second medical imaging data or representing a given natural language text describing the changes between the given first medical imaging data and the given second medical imaging data. In the natural language processing machine learning model ( Figure 4 In the previous example (not shown in FIG), the text feature vector is generated based on given text data representing a given first medical text report and a given second medical text report. As described above, the given first medical text report and the given second medical text report can be spliced ​​and input into the natural language processing machine learning model. In the latter example (as shown in FIG. Figure 4 ), the trained large language model 352 with appropriate prompts can be used to generate a given natural language text from a given first medical text report and a given second medical text report in the manner described above in relation to the first other medical text report and the second other medical text report. Then, the text encoder 450 is used to generate a given natural language text from a given first medical text report and a given second medical text report in the manner described above with reference to Figure 3 The text feature vector is generated in the manner described by the trained text encoder 350 .

[0073] The training method 400 includes providing training data. The training data includes a plurality of training imaging data sets, each of which represents a first training medical imaging data set 402 and a second training medical imaging data set 404, or represents a difference between the first training medical imaging data set 402 and the second training medical imaging data set 404. In this example, the training imaging data is training differential imaging data 306 representing a difference between the first training medical imaging data set 402 and the second training medical imaging data set 404. In other examples, the training imaging data may include a concatenation of the first training medical imaging data set 402 and the second training medical imaging data set 404.

[0074] For each training imaging data set, the training data also includes text data representing a first training medical text report 412 associated with the first training medical imaging data 402 and a second training medical text report 412 associated with the second training medical imaging data 404, or representing natural language text describing changes between the first training medical imaging data 402 and the second training medical imaging data 404. The natural language text describing changes between the first training medical imaging data 402 and the second training medical imaging data 404 can be generated using the first medical text report 312 and the second medical text report in the same manner as natural language text describing changes between other first medical imaging data 302 and other second medical imaging data 304 can be generated using the corresponding first medical text report and the corresponding second medical text report (i.e., using the trained large language model 352).

[0075] In this example, for each training imaging data set, multiple training imaging data sets include text data representing a corresponding first medical text report and a corresponding second medical text report. For example, the corresponding first medical text report 372 and the corresponding second medical text report 374 can be spliced ​​and input into the text encoder 450 to generate a text feature vector, as described in the "variation of the second method" above. In other examples, for each training imaging data set, multiple training imaging data sets include text data representing a natural language text describing the change between the corresponding other first medical imaging data and the corresponding other second medical imaging data. For example, the corresponding first medical text report 372 and the corresponding second medical text report 374 can be input with a text prompt such as a text prompt 376 to generate a natural language text describing the change, and the natural language text can be input into the text encoder 450 to generate a text feature vector, as described above in the second method.

[0076] Compared with using different medical text reports in the training and inference processes, using the same medical text report in the training process (training text encoder 450) and the inference process (generating multiple text feature vectors Zt) can allow the number of medical text reports that need to be acquired to be reduced. In addition, the above-mentioned dictionary including text feature vectors can be established during the training method 400, or determined and stored before performing the inference process (method 300). This means that the text feature vectors do not have to be newly generated during the execution of the inference process, which can help provide a computationally efficient inference process.

[0077] The training method 400 includes training a machine learning model based on training data. Training the machine learning model includes training the machine learning model to minimize a loss function 470 between an image feature vector generated by an image processing machine learning model 440 for a training imaging data set and a corresponding text feature vector generated by a natural language processing machine learning model for the training imaging data set. As described above, in this example, the training imaging data is training differential imaging data.

[0078] In this example, the loss function 470 is a function of an input image feature vector generated by the image processing machine learning model 440 for a training imaging dataset and an input corresponding text feature vector generated for the same training imaging dataset. Training the machine learning model to minimize the loss function 470 includes, for each training imaging dataset, minimizing the loss function 470 between the generated image feature vector and the generated text feature vector.

[0079] In this example, minimizing the loss function 470 includes reducing the distance between the image feature vectors generated for a given set of multiple training differential imaging datasets and the text feature vectors generated for the given set in the common embedding space. That is, the loss function 470 may tend to attract the image feature vectors to the text feature vectors generated for the same training differential imaging dataset in the common embedding space.

[0080] The loss function 470 generally has the effect that when performing Figures 1 to 3 During the method 300 (i.e., during inference), the image feature vector generated by the trained image processing machine learning model 340 captures features that are considered relevant when writing natural language text describing the changes 382 between the first medical imaging data 302 and the second medical imaging data 304. This is because, by virtue of training using the loss function 470, the first machine learning model is trained to generate image feature vectors that are close to the text feature vectors in the common embedding space.

[0081] In some examples, the loss function 470 is a contrastive loss function 470. In addition to the above properties of the loss function 470, minimizing the contrastive loss function 470 also includes increasing the distance between one of the image feature vectors or text feature vectors generated for one of the multiple training differential imaging datasets and one of the image feature vectors or text feature vectors generated for another of the multiple training differential imaging datasets in the common embedding space. That is, the contrastive loss function 470 tends to separate feature vectors belonging to different sets from each other. The contrastive loss function 470 generally has a functional form: Where S(x, y) is a function whose value decreases as the distance between x and y increases, 1d is a first set of the plurality of training data sets, and 2d is a second set of the plurality of training data sets different from the first set. Zi is an image feature vector, and Zt is a text feature vector. Thus, for example, S(Zi 2d , Zt 1d ) refers to the distance between the image feature vector generated by the image processing machine learning model 440 for the second episode and the text feature vector generated by the text encoder 450 for the first episode different from the second episode.

[0082] Different training data sets correspond to different variations of medical imaging data, such as for different patients. The contrastive loss function provides that the text and / or image feature vectors representing these different variations are spaced apart in a common embedding space, but the image feature vectors and text feature vectors of any one training data set are close together. This helps to ensure that, at inference time, the image feature vector representing the difference between the first medical imaging data 302 and the second medical imaging data 304 will be close to the text feature vector representing the natural language text that accurately describes the variation between the first medical imaging data 302 and the second medical imaging data 304, but away from the text feature vector representing the natural language text that describes the different variation. Therefore, using a contrastive loss function during training can help allow data that accurately represents the variation between the first medical imaging data and the second medical imaging data to be determined at inference time.

[0083] refer to Figure 5 , shows an apparatus 600 according to an example. The apparatus 600 includes an input interface 606, an output interface 608, and a processor 602 in communication with a non-transitory memory device 604. The processor 602 and the non-transitory memory device 604 may be configured to execute the above reference Figures 1 to 4 The memory device 604 may store instructions or computer readable program codes that, when executed by the processor 602, cause the processor 602 to perform the above-referenced method 100, 300, or training method 400. Figures 1 to 3 Any of the example methods 100, 300 described above, and / or according to the above reference Figure 4 Any of the example training methods 400 described herein. The instructions may be stored on any computer-readable medium (eg, any non-transitory computer-readable medium).

[0084] For example, the input interface 606 can receive imaging data, and the processor 602 can implement the above reference Figures 1 to 3The method 100, 300 described above can be used to determine data representing a change, and the processor 602 can output the data representing the change through the output interface 608. In some examples, the data representing the change can be transmitted to a structured storage device (not shown) so that the output data is stored in the structured storage device. In some examples, the data representing the change is transmitted to a display device (not shown) such as a user terminal to allow the user to view the output data. In some examples, the output data can be stored in the memory 604 alternatively or additionally.

[0085] As another example, alternatively or additionally, the input interface 606 may receive training data, and the processor 608 may implement training of the machine learning model, as described above with reference to Figure 4 As described, the processor 602 can output the trained machine learning model, the trained image processing machine learning model 340, the trained text encoder 350, or data representing any of these trained models via the output interface 608. In some examples, any trained model or data representing them can be stored in an external storage device (not shown) or transmitted to another computer (not shown) for use by another computer (not shown) to perform the above reference. Figures 1 to 3 In some examples, any trained models or data representing them may be stored in the memory 604 or another local storage device of the device 600, for example, for the device 600 to implement the above reference. Figures 1 to 3 Any of the example methods 100, 300 described above may be used.

[0086] The apparatus 600 may be implemented as a processing system and / or a computer. Figures 1 to 4 The methods 100 , 300 , 400 of any of the described examples are computer-implemented methods, and these methods 100 , 300 , 400 may be implemented by the apparatus 600 .

[0087] Although in some of the above examples, the imaging data is the differential imaging data 306 representing the difference between the first medical imaging data 302 and the second medical imaging data 304, it should be understood that in other examples, the imaging data may instead represent the first medical imaging data 302 and the second medical imaging data 304. For example, obtaining the imaging data may include stitching the first medical imaging data 302 and the second medical imaging data 304. The method 300 may then include, instead of inputting the differential imaging data 306 into the trained image processing machine learning model 340, inputting the stitched imaging data into the trained image processing machine learning model 340. The image processing machine learning model 440 is still able to generate an image feature vector representing the difference between the first medical imaging data 302 and the second medical imaging data 304 by virtue of training using the loss function 470.

[0088] The same applies to the training imaging data; that is, instead of the training method 400 using training difference imaging data representing the difference between the first training medical imaging data 402 and the second training medical imaging data 404, the training method 400 may use training imaging data representing the first training medical imaging data 402 and the second training medical imaging data 404. For example, each training imaging data set may include a concatenation of the first training medical imaging data 402 and the second training medical imaging data 404.

[0089] The above examples should be understood as illustrative examples of the present invention. It should be understood that any feature described with respect to any one example can be used alone or in combination with other features described, and can also be used in combination with one or more features of any other example or any combination of any other example. In addition, equivalents and modifications not described above may also be adopted without departing from the scope of the present invention as defined by the appended claims.

[0090] According to a first aspect of the present framework, a computer-implemented method for determining data representing a change between first medical imaging data and second medical imaging data is provided, the method comprising: obtaining imaging data representing the first medical imaging data and the second medical imaging data or the difference between the first medical imaging data and the second medical imaging data; inputting the imaging data into a trained image processing machine learning model to generate an image feature vector representing the difference between the first medical imaging data and the second medical imaging data; obtaining a plurality of text feature vectors, each text feature vector representing a natural language text describing a corresponding change in the medical imaging data; for each of the plurality of text feature vectors, determining a similarity measure indicating the similarity between the image feature vector and the text feature vector; selecting a text feature vector from the plurality of text feature vectors based on the determined similarity measure; and determining data representing the change between the first medical imaging data and the second medical imaging data based on the selected text feature vector.

[0091] Optionally, the trained image processing machine learning model has been trained to generate an image feature vector representing the difference between given first medical imaging data and given second medical imaging data based on given imaging data.

[0092] Optionally, the given imaging data represents given first medical imaging data and given second medical imaging data.

[0093] Optionally, the given imaging data represents a difference between given first medical imaging data and given second medical imaging data.

[0094] Optionally, the first medical imaging data is of the patient and has been captured at a first time, and the second medical imaging data is of the patient and has been captured at a second, later time.

[0095] Optionally, the first medical imaging data and the second medical imaging data are directed to a specific region of the patient.

[0096] Optionally, the change indicates the onset or progression of a medical abnormality.

[0097] Optionally, the method comprises storing data representing the changes in a storage device.

[0098] Optionally, the method comprises outputting data representing the change to a user terminal.

[0099] Optionally, determining data representing the change comprises determining natural language text represented by the selected text feature vector.

[0100] Optionally, the data representing the change comprises natural language text represented by the selected text feature vector or data derived therefrom.

[0101] Optionally, determining the data representative of the changes comprises determining whether the natural language text represented by the selected text feature vector describes a significant change, and the data representative of the changes comprises an indication of whether the natural language text represented by the selected text feature vector describes a significant change.

[0102] Optionally, the method includes outputting output image data representing changes between the first medical imaging data and the second medical imaging data through a trained image processing machine learning model and based on the imaging data and the selected text feature vector, wherein the data representing the changes includes the output image data.

[0103] Optionally, the output image data includes attention map data.

[0104] Optionally, the method includes: generating multiple image datasets using a trained image processing machine learning model, each image dataset representing changes between first medical imaging data and second medical imaging data, and each image dataset being represented by a corresponding feature vector; and determining, for each of the multiple image datasets, a further similarity measure indicating the similarity between the corresponding feature vector representing the image and the selected text feature vector, wherein output image data is selected from the multiple image datasets based on the determined further similarity measure.

[0105] Optionally, each of the multiple image datasets includes corresponding attention map data.

[0106] Optionally, the output image data includes multiple output intensity values, each output intensity value has a corresponding pixel position, and the method includes: determining whether one or more of the multiple output intensity values ​​meets a condition; and based on the determination, outputting output mask image data representing the pixel positions of the output intensity values ​​that meet the condition.

[0107] Optionally, the condition is a threshold.

[0108] Optionally, the method includes: obtaining first medical imaging data and second medical imaging data, the first medical imaging data including multiple first intensity values, and the second medical imaging data including multiple corresponding second intensity values; and for each of the multiple first intensity values, comparing the first intensity value with the corresponding second intensity value to obtain a differential intensity value, wherein the imaging data includes the obtained differential intensity value.

[0109] Optionally, comparing the first intensity value to the second intensity value comprises performing a subtraction operation using the first intensity value and the second intensity value.

[0110] Optionally, the method comprises registering the first medical imaging data with the second medical imaging data by transforming the plurality of first intensity values ​​and the plurality of second intensity values ​​into a coordinate system before comparing the first intensity value with the second intensity value.

[0111] Optionally, the image feature vector and the plurality of text feature vectors are represented in a common embedding space, and the similarity measure is a distance measure between the image feature vector and the text feature vector in the common embedding space.

[0112] Optionally, selecting the text feature vector includes selecting a text feature vector corresponding to a lowest distance metric among the distance metrics from the plurality of text feature vectors.

[0113] Optionally, the method includes generating each of one or more of the plurality of text feature vectors by: inputting into a trained large language model: a first medical text report associated with first medical imaging data, and a text prompt comprising instructions for generating natural language text describing possible variations of the first medical imaging data based on the first medical text report; and generating a text feature vector using a trained text encoder and based on the corresponding generated natural language text describing the possible variations.

[0114] Optionally, the trained large language model has been trained to generate natural language text describing possible changes based on an input of a given first medical text report and a text prompt associated with a given first medical imaging data, the text prompt comprising instructions for generating natural language text describing possible changes to the given first medical imaging data based on the given first medical text report; and the trained text encoder has been trained to generate a text feature vector based on the natural language text describing possible changes.

[0115] Optionally, each of the plurality of text feature vectors represents a natural language text describing a change between the corresponding other first medical imaging data and the corresponding other second medical imaging data.

[0116] Optionally, the data representing the changes include natural language text represented by the selected text feature vectors.

[0117] Optionally, the method includes generating each of one or more of the multiple text feature vectors by: inputting the following into a trained large language model: a corresponding first medical text report associated with the corresponding other first medical imaging data, a corresponding second medical text report associated with the corresponding other second medical imaging data, and a text prompt, the text prompt including instructions for generating natural language text describing changes between the corresponding other first medical imaging data and the corresponding other second medical imaging data based on the corresponding other first medical text report and the corresponding other second medical text report; and generating a text feature vector using a trained text encoder and based on the corresponding generated natural language text describing the change.

[0118] Optionally, for each text feature vector, the corresponding other first medical imaging data is for the corresponding patient and has been captured at a first time, and the corresponding other second medical imaging data is for the corresponding patient and has been captured at a second, later time.

[0119] Optionally, the trained large language model has been trained to generate natural language text describing changes based on an input of a given first medical text report associated with a given first medical imaging data, a given second medical text report associated with a given second medical imaging data, and a text prompt, wherein the text prompt includes instructions for generating natural language text describing changes between the given first medical imaging data and the given second medical imaging data based on the given first medical text report and the given second medical text report; and the trained text encoder has been trained to generate a text feature vector based on the natural language text describing the changes.

[0120] Optionally, the trained machine learning model includes a trained image processing machine learning model and a trained natural language processing machine learning model, and the trained image processing machine learning model has been trained according to a training method, the training method comprising: providing a machine learning model, the machine learning model comprising: (a) an image processing machine learning model, which is configured to generate an image feature vector based on given imaging data representing given first medical imaging data and given second medical imaging data or representing the difference between the given first medical imaging data and the given second medical imaging data, the image feature vector representing the difference between the given first medical imaging data and the given second medical imaging data, and (b) a natural language processing machine learning model, which is configured to generate an image feature vector based on given imaging data representing a given first medical imaging data and a given second medical imaging data associated with the given first medical imaging data and a given second medical text report associated with the given second medical imaging data or representing a description of the given first medical imaging data and the given second medical text report. generating a text feature vector based on given text data of a given natural language text describing the changes between the first training medical imaging data and the second training medical imaging data; providing training data comprising a plurality of training imaging data sets, each training imaging data set representing first training medical imaging data and second training medical imaging data, or representing the difference between the first training medical imaging data and the second training medical imaging data, and for each training imaging data set, the training data further comprising: text data representing a first training medical text report associated with the first training medical imaging data and a second training medical text report associated with the second training medical imaging data, or representing a natural language text describing the changes between the first training medical imaging data and the second training medical imaging data; and training a machine learning model based on the training data so as to minimize a loss function between an image feature vector generated by the image processing machine learning model for the training imaging data set and a corresponding text feature vector generated by the natural language processing machine learning model for the training imaging data set.

[0121] Optionally, for each training imaging data set, the plurality of training imaging data sets include text data representing a corresponding first medical text report and a corresponding second medical text report or representing natural language text describing changes between corresponding other first medical imaging data and corresponding other second medical imaging data.

[0122] Optionally, the loss function is a contrastive loss function.

[0123] Optionally, minimizing the contrast loss function includes: reducing the distance in a common embedding space between: an image feature vector generated for a given set of multiple training imaging datasets, and a text feature vector generated for the given set; and / or increasing the distance in a common embedding space between: an image feature vector generated for a first set of multiple training imaging datasets and one of the text feature vectors generated for the first set, and an image feature vector generated for a second set of multiple training imaging datasets and one of the text feature vectors generated for the second set.

[0124] According to a second aspect of the present framework, there is provided an apparatus configured to perform the method according to the first aspect.

[0125] Optionally, the apparatus includes a storage device and / or a user terminal.

[0126] According to a third aspect of the present framework, there is provided a computer program which, when executed by a computer, causes the computer to perform the method according to the first aspect.

[0127] The foregoing examples are provided for the purpose of explanation only and are in no way to be construed as limitations on the present framework disclosed herein. Although the present invention has been described with reference to various embodiments, it should be understood that the words that have been used herein are descriptive and illustrative, rather than restrictive. In addition, although the present invention has been described herein with reference to specific components, materials, and embodiments, the present invention is not intended to be limited to the details disclosed herein, but rather, the present invention extends to all functionally equivalent structures, methods, and uses, such as within the scope of the appended claims. A person skilled in the art who has the benefit of the teachings of this specification may make various modifications thereto, and may make changes without departing from the scope and spirit of the present invention in its aspects.

Claims

1. A computer-implemented method for determining data representing a change between first medical imaging data and second medical imaging data, the method comprising: obtaining imaging data representing the first medical imaging data and the second medical imaging data or representing a difference between the first medical imaging data and the second medical imaging data; inputting the imaging data into a trained image processing machine learning model to generate an image feature vector representing a difference between the first medical imaging data and the second medical imaging data; obtaining a plurality of text feature vectors, each text feature vector representing a natural language text describing a corresponding change in the medical imaging data; for each of the plurality of text feature vectors, determining a similarity measure indicating a degree of similarity between the image feature vector and the text feature vector; selecting a text feature vector from a plurality of text feature vectors based on the determined similarity measure; and Based on the selected text feature vectors, data representing changes between the first medical imaging data and the second medical imaging data are determined.

2. The method of claim 1, wherein the data representing the change comprises natural language text represented by the selected text feature vector or data derived therefrom.

3. A method according to claim 1, wherein determining the data representing the change includes determining whether the natural language text represented by the selected text feature vector describes a significant change, and the data representing the change includes an indication of whether the natural language text represented by the selected text feature vector describes a significant change.

4. The method according to claim 1 includes outputting output image data representing the changes between the first medical imaging data and the second medical imaging data through a trained image processing machine learning model and based on the imaging data and the selected text feature vector, wherein the data representing the changes includes the output image data.

5. The method according to claim 4, comprising: generating a plurality of image datasets using the trained image processing machine learning model, each image dataset representing a change between the first medical imaging data and the second medical imaging data, and each image dataset being represented by a corresponding feature vector; as well as A further similarity measure is determined for each of the plurality of image datasets indicating a similarity between a corresponding feature vector representing the image and the selected text feature vector, wherein output image data is selected from the plurality of image datasets based on the determined further similarity measure.

6. The method according to claim 1, comprising: obtaining first medical imaging data and second medical imaging data, the first medical imaging data comprising a plurality of first intensity values ​​and the second medical imaging data comprising a plurality of corresponding second intensity values; as well as For each of the plurality of first intensity values, the first intensity value is compared to a corresponding second intensity value to obtain a differential intensity value, wherein the imaging data includes the obtained differential intensity value. The method of claim 6 , wherein comparing the first intensity value to the second intensity value comprises performing a subtraction operation using the first intensity value and the second intensity value.

8. The method of claim 1, wherein the image feature vector and the plurality of text feature vectors are represented in a common embedding space, and the similarity measure is a distance measure between the image feature vector and the text feature vector in the common embedding space.

9. The method of claim 1, comprising generating each of one or more of the plurality of text feature vectors by: The following is input into the trained large language model: a first medical text report associated with the first medical imaging data, and a text prompt including instructions to generate natural language text describing a possible change to the first medical imaging data based on the first medical text report; and A text feature vector is generated using the trained text encoder and based on the corresponding generated natural language text describing the possible variations.

10. The method according to claim 1, wherein: Each of the plurality of text feature vectors represents a natural language text describing a change between corresponding other first medical imaging data and corresponding other second medical imaging data.

11. The method of claim 10, comprising generating each of one or more of the plurality of text feature vectors by: The following is input into the trained large language model: a corresponding first medical text report associated with the corresponding other first medical imaging data, a corresponding second medical text report associated with the corresponding other second medical imaging data, and a text prompt including instructions for generating a natural language text describing a change between the corresponding other first medical imaging data and the corresponding other second medical imaging data based on the corresponding first medical text report and the corresponding second medical text report; and A text feature vector is generated using the trained text encoder and based on the corresponding generated natural language text describing the change.

12. The method according to claim 1, wherein: The trained machine learning model includes a trained image processing machine learning model and a trained natural language processing machine learning model, and the trained image processing machine learning model has been trained according to a training method, the training method comprising: A machine learning model is provided, wherein the machine learning model comprises: (a) an image processing machine learning model configured to generate an image feature vector representing a difference between given first medical imaging data and given second medical imaging data based on given imaging data representing the given first medical imaging data and given second medical imaging data or representing a difference between the given first medical imaging data and given second medical imaging data, and (b) a natural language processing machine learning model configured to generate a text feature vector based on given text data representing a given first medical text report associated with given first medical imaging data and a given second medical text report associated with given second medical imaging data or representing a given natural language text describing a change between the given first medical imaging data and the given second medical imaging data; Providing training data including a plurality of training imaging data sets, each training imaging data set representing first training medical imaging data and second training medical imaging data, or representing a difference between the first training medical imaging data and the second training medical imaging data, and for each training imaging data set, the training data further comprising: text data representing a first training medical text report associated with the first training medical imaging data and a second training medical text report associated with the second training medical imaging data, or representing a natural language text describing a change between the first training medical imaging data and the second training medical imaging data; and The machine learning model is trained based on the training data so as to minimize a loss function between an image feature vector generated by the image processing machine learning model for the training imaging data set and a corresponding text feature vector generated by the natural language processing machine learning model for the training imaging data set.

13. The method according to claim 12, wherein: The plurality of training imaging data sets include, for each training imaging data set, text data representing a corresponding first medical text report and a corresponding second medical text report.

14. The method according to claim 12, wherein: The plurality of training imaging data sets include, for each training imaging data set, text data representing natural language text describing changes between corresponding other first medical imaging data and corresponding other second medical imaging data.

15. A system for determining data representing a change between first medical imaging data and second medical imaging data, comprising: a non-transitory memory device for storing computer readable program code; as well as a processor in communication with the memory device, the processor being operable with the computer readable program code to perform steps comprising: obtaining imaging data representing the first medical imaging data and the second medical imaging data or a difference between the first medical imaging data and the second medical imaging data, inputting the imaging data into a trained image processing machine learning model to generate an image feature vector representing a difference between the first medical imaging data and the second medical imaging data, obtaining a plurality of text feature vectors, each text feature vector representing a natural language text describing a corresponding change in the medical imaging data, For each of the plurality of text feature vectors, determining a similarity measure indicating a degree of similarity between the image feature vector and the text feature vector, selecting a text feature vector from the plurality of text feature vectors based on the determined similarity measure, and Based on the selected text feature vectors, data representing changes between the first medical imaging data and the second medical imaging data are determined.

16. The system of claim 15, wherein the data representing the variation comprises natural language text represented by the selected text feature vector or data derived therefrom.

17. A system according to claim 15, wherein determining the data representing the change includes determining whether the natural language text represented by the selected text feature vector describes a significant change, and the data representing the change includes an indication of whether the natural language text represented by the selected text feature vector describes a significant change.

18. The system of claim 15, comprising outputting output image data representing changes between the first medical imaging data and the second medical imaging data through a trained image processing machine learning model and based on the imaging data and the selected text feature vector, wherein the data representing the changes includes the output image data.

19. The system according to claim 18, comprising: generating a plurality of image datasets using the trained image processing machine learning model, each image dataset representing a change between the first medical imaging data and the second medical imaging data, and each image dataset being represented by a corresponding feature vector; as well as For each of the plurality of image datasets, a further similarity measure is determined indicating a similarity between a corresponding feature vector representing the image and the selected text feature vector, wherein output image data is selected from the plurality of image datasets based on the determined further similarity measure.

20. One or more non-transitory computer-readable media containing instructions executable by a machine to perform operations comprising: obtaining imaging data representing the first medical imaging data and the second medical imaging data or representing a difference between the first medical imaging data and the second medical imaging data; inputting the imaging data into a trained image processing machine learning model to generate an image feature vector representing a difference between the first medical imaging data and the second medical imaging data; obtaining a plurality of text feature vectors, each text feature vector representing a natural language text describing a corresponding change in the medical imaging data; for each of the plurality of text feature vectors, determining a similarity measure indicating a degree of similarity between the image feature vector and the text feature vector; selecting a text feature vector from a plurality of text feature vectors based on the determined similarity measure; and Based on the selected text feature vectors, data representing changes between the first medical imaging data and the second medical imaging data are determined.

Images