Compensating for differences in medical images

Abstract

A computer-implemented method for compensating for differences in medical images is provided. The method comprises receiving (S110) image data comprising a time series of medical images (1101... i). The time series includes one or more medical images generated by a first type of imaging modality (120) and one or more medical images generated by a second type of imaging modality (130, 130 '). The imaging modality of the first type is different from the imaging modality of the second type. In one aspect, the method comprises generating (S120a, S120b) a time sequence of normalized medical images (1401.. i) from the time sequence (1101.. i) of medical images, and outputting (S130a, S130a ', S130b) the time sequence of normalized medical images (1401.. i) and / or one or more measurements derived therefrom. In another aspect, the method comprises generating (S120a, S120b) one or more normalized measurements (1501... j) representing a region of interest (1101... i) in the time series of medical images from the time series of medical images (1101... i), and outputting the normalized measurements (1501... j).

Description

Technical Field

[0001] The present disclosure relates to compensating for discrepancies in medical images. A computer-implemented method, computer program product, and system are disclosed. Background Art

[0002] Clinical investigations often involve the acquisition of medical images of a subject. Images may be acquired at different points in time, i.e., as a time series of images. The images may be used to identify temporal changes in a region of interest, and thus assess the progression of a medical condition in the subject. However, during the course of a clinical investigation, images may be acquired using different types of imaging modalities. For example, factors such as the availability of an imaging system, the need to perform additional investigations of the subject's medical condition, etc. may result in the acquisition of medical images of a region of interest using different types of imaging modalities over time. Images generated by different types of imaging modalities have very different appearances. Therefore, it is difficult to accurately identify temporal changes in a region of interest based on these images. This hinders the assessment of the progression of the subject's medical condition.

[0003] As an example, a clinical investigation may be performed by generating projection X-ray images of a region of interest in a subject at different points in time. However, due to the availability of imaging systems, or due to the need to perform additional investigations, some of the images generated during the course of the clinical investigation may be generated using a computed tomography "CT" imaging system or a magnetic resonance "MR" imaging system. Such images are very different in appearance from X-ray projection images, and there is also no direct correspondence between their image intensities and those in the X-ray projection images. Therefore, an assessment of the progression of a medical condition in a subject may be performed using only images generated by a single type of imaging modality, even if data exists at additional time points from different types of imaging modalities. This represents a missed opportunity, as the data at the additional time points may otherwise provide valuable information for assessing the progression of the medical condition.

[0004] In addition to the challenges presented by images acquired using different types of imaging modalities during the course of a clinical investigation, there may also be differences in the way the images are acquired. For example, there may be differences in the posture of the subject, or differences in the viewing angle of the medical imaging system, or differences in the ionizing radiation dose used to acquire the image. Such factors may exist even when images are acquired from a single type of imaging modality. These factors exacerbate the challenge of identifying temporal changes in the region of interest and may even lead to an erroneous diagnosis of the subject's condition. Some of these factors can be addressed by applying broad device imaging protocols. For example, some clinical settings allow the subject to be positioned in an optimal and standardized manner relative to the imaging system. Protocols can also be set up in which imaging systems from a common modality use standardized radiation doses. The application of broad device imaging protocols can reduce the amount of image-to-image variability caused by differences in the way images are acquired. However, the use of such protocols limits flexibility. In some clinical settings, it is also impractical or even impossible to position an object with limited mobility in an optimal and standardized manner relative to the imaging system. For example, in a critical care environment, it may be impractical or even impossible to acquire images of an object with a desired posture, inhalation state, or an object in a desired position relative to the medical imaging system. Therefore, differences in the way images are acquired may also hamper the assessment of the progression of a medical condition in a subject. In particular, the fact that these confounding factors are not constant over time and that they vary between each image makes it challenging to assess longitudinal changes in a subject's condition. Therefore, clinicians interpreting such images may intuitively compensate for such differences, which can lead to incorrect diagnoses.

[0005] Therefore, there is a need for improvements that facilitate identifying changes in time series of medical images. Summary of the invention

[0006] According to one aspect of the present disclosure, a computer-implemented method for compensating for differences in medical images is provided. The method comprises:

[0007] receiving image data comprising a temporal sequence of medical images, the temporal sequence comprising one or more medical images generated by a first type of imaging modality and one or more medical images generated by a second type of imaging modality, the first type of imaging modality being different from the second type of imaging modality;

[0008] generating a normalized time series of medical images or one or more normalized measurements representing a region of interest in the time series of medical images from the time series of medical images; and

[0009] The time series of normalized medical images and / or one or more measured values ​​derived from the time series of normalized medical images are output, or one or more normalized measured values ​​are output, respectively.

[0010] In the above method, the time series of normalized medical images, the one or more measurements derived from the time series of normalized medical images, and the one or more normalized measurements each provide compensation for differences in the type of imaging modality used to acquire the images in the time series. By providing such compensation, a more reliable comparison between images can be made, which allows a more reliable assessment of longitudinal changes in a medical condition in a subject.

[0011] Other aspects, features and advantages of the present disclosure will become apparent from the following description of examples with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart illustrating an example of a method of compensating for discrepancies in medical images according to some aspects of the present disclosure.

[0013] Figure 2 is a schematic diagram illustrating an example of a system 200 for compensating for discrepancies in medical images according to some aspects of the present disclosure.

[0014] Figure 3 is a schematic diagram illustrating a first example of a method of compensating for discrepancies in a medical image according to some aspects of the present disclosure.

[0015] Figure 4 FIG. 2 is a diagram illustrating projecting a volumetric medical image 1103 onto a virtual detector 240 using a virtual source 230v according to some aspects of the present disclosure. v A schematic diagram of an example of operations on .

[0016] Figure 5 An example of an atlas image 170 for a lung region is shown in accordance with some aspects of the present disclosure.

[0017] Figure 6 An example of an atlas image 170 for a thorax is shown in accordance with some aspects of the present disclosure.

[0018] Figure 7 An example of the results of applying a feature detector to a medical image 1401 to identify a plurality of landmarks in the medical image is shown in accordance with some aspects of the present disclosure.

[0019] Figure 8 An example of warping two 2D medical images in a time series 1401 , 1402 to a reference image is shown according to some aspects of the present disclosure.

[0020] Fig. 9 An example of warping two 2D medical images in a time series 1401 , 1402 to a reference image and adjusting the intensity of the medical images according to some aspects of the present disclosure is shown.

[0021] Fig.10 An example of suppressing image features outside of a lung region of interest in a medical image 1401 is shown in accordance with aspects of the present disclosure.

[0022] Fig.11 An example of warping two normalized medical images 1401 , 1402 to a reference image and adjusting image intensity values ​​in the images and suppressing image features outside a lung region of interest according to some aspects of the present disclosure is shown.

[0023] Fig.12 is a schematic diagram illustrating a second example of a method of compensating for discrepancies in a medical image according to some aspects of the present disclosure. DETAILED DESCRIPTION

[0024] Examples of the present disclosure are provided with reference to the following description and accompanying drawings. In this specification, many specific details of certain examples are set forth for the purpose of explanation. References in the specification to "example," "implementation," or similar language mean that the features, structures, or characteristics described in conjunction with the example are at least included in the example. It should also be understood that features described with respect to one example may also be used in another example, and that not all features are necessarily repeated in each example for the sake of brevity. For example, features described with respect to a computer-implemented method may be implemented in a computer program product and implemented in a corresponding manner in a system.

[0025] In the following description, reference is made to computer-implemented methods of processing image data associated with a time series of medical images. In some examples, reference is made to medical images representing lungs of a subject. However, it should be understood that the lungs are used only as an example, and the methods disclosed herein may alternatively be used with medical images representing any portion of an anatomical structure.

[0026] Note that the computer-implemented methods disclosed herein may be provided as a non-transitory computer-readable storage medium including computer-readable instructions stored thereon, which, when executed by at least one processor, cause the at least one processor to perform the method. In other words, the computer-implemented methods may be implemented in a computer program product. The computer program product may be provided by dedicated hardware or hardware capable of running software in association with appropriate software. When provided by a processor, the functionality of the method features may be provided by a single dedicated processor or by a single shared processor or by a plurality of separate processors, some of which may be shared. One or more of the functionality of the method features may be provided, for example, by a processor shared within a networked processing architecture such as a client / server architecture, a peer-to-peer architecture, the Internet, or a cloud.

[0027] The explicit use of the term "processor" or "controller" should not be interpreted as exclusively referring to hardware capable of running software, and may implicitly include, but is not limited to, digital signal processor "DSP" hardware, read-only memory "ROM" for storing software, random access memory "RAM", non-volatile storage devices, etc. In addition, examples of the present disclosure may take the form of a computer program product accessible from a computer-usable storage medium or a computer-readable storage medium, which provides program code for use by or in conjunction with a computer or any instruction execution system. For the purposes of this specification, a computer-usable storage medium or a computer-readable storage medium may be any device capable of containing, storing, transmitting, propagating or transmitting a program for use by or in conjunction with an instruction execution system, device or apparatus. The medium may be an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system or device or a propagation medium. Examples of computer-readable media include semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory "RAM", read-only memory "ROM", rigid magnetic disks and optical disks. Current examples of optical disks include compact disk-read only memory "CD-ROM", compact disk-read / write "CD-R / W", Blu-ray TM and DVD.

[0028] As noted above, there is a need for improvements that facilitate identifying changes in time series of medical images.

[0029] Figure 1 is a flow chart illustrating an example of a method of compensating for discrepancies in medical images according to some aspects of the present disclosure. Figure 2 is a schematic diagram illustrating an example of a system 200 for compensating for discrepancies in medical images according to some aspects of the present disclosure. Figure 1 The operations described in the method shown can also be performed in Figure 2 The system 200 shown in FIG. 1 is implemented in the system 200, and vice versa. Figure 1, the computer-implemented method of compensating for differences in medical images, comprising:

[0030] Receiving S110 includes medical image 110 1…i a time series of image data, the time series comprising one or more medical images generated by a first type of imaging modality 120 and one or more medical images generated by a second type of imaging modality 130, 130', the first type of imaging modality being different from the second type of imaging modality;

[0031] According to the medical image 110 1…i The time series generates S120a, S120b the normalized medical image 140 1…i A time series, or representation of the medical image 110 1…i One or more normalized measurements of a region of interest in a time series 150 1…j ;as well as

[0032] The normalized medical images 140 are outputted respectively at S130a, S130a', and S130b. 1…i time series and / or from the normalized medical image 140 1…i The one or more measured values ​​derived from the time series, or the one or more normalized measured values ​​150 are outputted 1…j .

[0033] In the above method, the time series of normalized medical images, the one or more measurements derived from the time series of normalized medical images, and the one or more normalized measurements each provide compensation for differences in the type of imaging modality used to acquire the images in the time series. By providing such compensation, a more reliable comparison between images can be made, which allows a more reliable assessment of longitudinal changes in a medical condition in a subject.

[0034] Figure 3 is a schematic diagram illustrating a first example of a method of compensating for differences in medical images according to some aspects of the present disclosure. Figure 1 and Figure 3 In operation S110, image data is received. The image data includes a medical image 110 1…i The received medical image 110 1…i A time series is formed in the sense that the images are acquired at different points in time. The images may be acquired periodically (i.e., at regular intervals) or intermittently (i.e., at irregular intervals). A time series of medical images may be acquired over a period of time, such as over a period of minutes, hours, days, weeks, or longer. Medical image 110 1…iA time series of images may represent a so-called longitudinal study of a subject. 1…i Any anatomical region may be represented. For example, the medical image region of interest may be the lungs, heart, liver, kidneys, etc. As an example, the medical image 110 received in operation S110 1…i The time series of can represent the chest of the object, such as Figure 3 For example, a time series of medical images may represent daily images of a subject's chest acquired as part of a longitudinal study in the subject to assess the progression of Covid 19 in the subject's lungs.

[0035] The medical image 110 received in operation S110 1…i The time series includes one or more medical images generated by a first type of imaging modality 120 and one or more medical images generated by a second type of imaging modality 130, 130'. The first type of imaging modality is different from the second type of imaging modality. In general, the first imaging modality and the second imaging modality can be any type of imaging modality. As some examples, the first type of imaging modality 120 and the second type of imaging modality 130, 130' can be selected from the following group: computed tomography, magnetic resonance, ultrasound, and projection X-ray. In view of their ability to acquire image data for reconstructing volumetric images (i.e., 3D images), computed tomography, magnetic resonance, and ultrasound imaging modalities can be referred to as volumetric imaging modalities. In view of its ability to acquire image data for reconstructing projection images (i.e., 2D images), the projection X-ray "PXR" imaging modality can be referred to as a projection imaging modality. A 2D medical image 110 can be generated 1…i Examples of current projection X-ray imaging systems include the Mobile Diagnost M50 (which is a mobile X-ray imaging system), the Digital Diagnost C90 (which is a ceiling-mounted X-ray imaging system), and the Azurion 7 (which includes a source-detector arrangement mounted to a C-arm), all of which are sold by Philips Healthcare of Best, the Netherlands.

[0036] As an example, the medical image 110 received in operation S110 1…i Can include Figure 3 The projection X-ray images generated by the projection X-ray imaging system labeled "PXR" in Figure 3 A volume image generated by a CT imaging system labeled "CT" in FIG. Figure 2 The projection X-ray imaging system 220 shown in FIG. Figure 3, the projection X-ray imaging system 220 is a so-called mobile X-ray imaging system, which is often used in intensive care facilities in order to obtain images of subjects with restricted mobility. Figure 2 The projection X-ray imaging system 220 shown includes an X-ray source 230 and a wirelessly coupled X-ray detector 240. The versatility of this type of X-ray imaging system facilitates imaging of subjects in complex settings where the subjects have limited mobility, such as Figure 2 The bed rest setting shown in . Figure 3 As shown in the example of FIG. 1 , a medical image 110 in a time series is generated using a projection X-ray imaging system PXR. 1…i In the example shown, these images are generated every day or every few days. However, Figure 3 As shown in the example of FIG. 1 , a time series 110 is generated by a CT imaging modality (ie, a different type of imaging modality). 1…i Given the availability of imaging systems, the CT imaging modality may have been used to perform additional clinical investigations on the subject, or the CT imaging modality may be used instead of a projection X-ray imaging system. As described above, since CT images are very different in appearance from X-ray projection images and there is no direct correspondence between their image intensities and those in X-ray projection images, it is challenging to assess the condition of a subject based on images acquired from different imaging modalities.

[0037] return Figure 1 In the method shown, generally speaking, in operation S110, a medical image 110 in a time series is received. 1…i The medical images 110 may be received via any form of data communication, including, for example, wired, optical, and wireless communications. By way of some examples, when wired or optical communications are used, the communication may occur via signals transmitted over electrical or optical cables, and when wireless communications are used, the communication may be, for example, via RF or optical signals. For example, the medical images 110 may be received in a time series by one or more processors. 1…i , such as Figure 2 For example, one or more processors 210 may be provided by a medical imaging system such as Figure 2 The medical image 110 is received from a projection X-ray imaging system 220 shown in FIG. 1 or from another medical imaging system or from another source such as a computer readable storage medium, the Internet, or the cloud. 1…i .

[0038] Continue to refer Figure 1 In operation S120a, according to the medical image 110 1…i Generate a normalized medical image from a time series of 1…itime series. Alternatively or in combination Figure 1 In operation S130a, the normalized medical image 140 is output. 1…i time series.

[0039] Outputting S130a the normalized medical image 140 may be performed in various ways. 1…i The time series operations include displaying, printing and storing the normalized medical image 140 1…i The output images may be displayed on a display. For example, the output images may be displayed on a display of a tablet computer, a laptop computer, or a mobile phone. Alternatively, the output images may be displayed on a monitor, such as Figure 2 As some examples, the normalized medical image 140 1…i The time series of images may be displayed side by side, or displayed as a time series, or as an overlay image. In another example, the normalized medical image 140 may be displayed by outputting an image from a first type of imaging modality together with a normalized image generated from an image generated by a second type of imaging modality, and overlaying the image generated by the second imaging modality on the normalized image generated from the image generated by the second type of imaging modality. 1…i The images generated by the second imaging modality are also registered to the normalized image generated from the images generated by the second type of imaging modality. This enables clinical findings from the second type of imaging modality to be provided to the user, which may not be apparent from the images from the first type of imaging modality, in order to assist the user in analyzing the medical condition in the subject.

[0040] In operation S130a′, the normalized medical image 140 is output. 1…i Therefore, instead of outputting a normalized medical image 140 1…i In addition to outputting these images, the normalized medical image 140 may be used to generate a time series of 1…i One or more measurements may be derived from a time series of images and the one or more measurements may be output. Typically, the measurements may be output for one or more images in the time series. For example, a single measurement may be derived from a single image in the time series and the single measurement may be output, or a single measurement may be derived from multiple images in the time series and the single measurement may be output. In the latter case, the single measurement may represent changes over time between the multiple images. Alternatively, a measurement may be output for each of the multiple images.

[0041] In this regard, various measurements may be derived from the image and output. Typically, the measurements may represent image intensity or diagnostic metrics. For example, the measurements may represent a normalized medical image 140. 1…i The intensity, volume or shape of a region of interest in a time series of images of the lungs may be measured by, for example, measuring the intensity, volume or shape of a region of interest in a time series of images of the lungs. An example of a diagnostic metric is a lung perfusion index. Examples of lung perfusion indices include: the volume of a pneumothorax, i.e., the volume of a "collapsed lung", or the amount of pleural effusion, i.e., the amount of "fluid surrounding the lungs", or the amount of pulmonary edema, i.e., the amount of "fluid in the lungs", or the stage of pneumonia. Continuing with the lung example, the perfusion index may be measured by, for example, measuring the perfusion index of a region of interest in a time series of images of the lungs. 1…i The lungs are contoured in the time series of FIG. 140 , pixels in the lungs are assigned as air or water based on their intensity values, and the normalized medical image 140 is used to calculate the lung contours. 1…i The volume of the corresponding air and water regions is estimated for each of the multiple images in the time series, thereby determining a measurement of the lung perfusion index. The value of the lung perfusion index can then be output for each of the multiple images in the time series. Alternatively, a single measurement value can be derived from the multiple images in the time series and output by, for example, determining individual measurement values ​​for the multiple images in the time series and expressing the individual measurement values ​​as a rate of change of the lung perfusion index. The normalized medical image 140 can also be determined in other ways. 1…i In one example, the normalized medical image 140 is determined by applying statistical analysis to the region of interest. 1…i In this example, refer to Figure 1 The described method may also include:

[0042] For normalized medical images 140 1…i performing statistical analysis on image intensity values ​​within the region of interest in ; and

[0043] The output S130a′ is obtained from the normalized medical image 140 1…i A time series of one or more measurements derived from a graph, including:

[0044] The results of the statistical analysis are output.

[0045] Various statistical analysis methods can be applied to the normalized medical images 140 1…i An example implementation of such statistical analysis is based on analyzing the normalized medical image 140 1…i In this example, the normalized medical image 140 may be analyzed based on a local patch-based analysis of the pixel values ​​in the image. 1…iStatistics are generated for each patch in the . This may include determining the mean and standard deviation of the pixel values ​​within the patch, assuming, for example, a Gaussian distribution. When comparing the normalized medical images 140 1…i When corresponding patches of, for example, two images A and B are viewed in a video, the Gaussian statistics of the patch of image A can be used to estimate what the measured intensity values ​​in the patch from image B are likely to be. The likelihood can be thresholded to provide a prediction of changes within the patch. The likelihood can also be visualized as an overlay of images A and B to allow a reviewing clinician to analyze detected changes in the images over time. Thus, this visualization can allow a reviewing clinician to assess whether the detected changes are statistically relevant, or whether, for example, they simply result from a flawed normalization process.

[0046] In a related example, the normalized medical image 140 is output. 1…i The operation of the result of the statistical analysis may include displaying the spatial map of the statistical analysis as a normalized medical image 140 1…i Providing the results of the statistical analysis in this manner can facilitate clinicians in providing efficient analysis of the progression of a medical condition in a subject.

[0047] By way of some other examples, the output is from the normalized medical image 140 1…i The one or more measurements derived from the time series may include outputting a numerical value of magnitude, such as an absolute value or a percentage, or outputting the measurement(s) graphically. In one example, the measurements are graphically represented as normalized medical images 140 in the time series by representing increases in lung volume between consecutive images in the sequence in green and decreases in lung volume between consecutive images in the sequence in red. 1…i The superposition of one or more of them.

[0048] By outputting the normalized medical image 140 1…i time series and / or from the normalized medical image 140 1…i A time series of one or more measurements derived from Figure 1 The method shown facilitates the reviewing clinician to make more reliable comparisons between regions of interest in the images. This in turn allows the clinician to make a more accurate assessment of longitudinal changes in the subject's condition.

[0049] Typically, the normalized medical image 140 1…i The time series of may represent images generated by a common type of imaging modality. In a first set of examples, the common type of imaging modality is a projection imaging modality, and the normalized medical image 140 is provided by projecting a volumetric medical image. 1…iIn the second set of examples, the common type of imaging modality is a projection imaging modality, and a normalized medical image 140 is provided by inputting the medical image into the neural network NN1. 1…i Typically, the common type of imaging modality may be the first type of imaging modality or the second type of imaging modality, or an imaging modality of a type different from the first type of imaging modality and the second type of imaging modality.

[0050] Refer to the above reference Figure 1 In the first set of examples, the generating operation includes generating a medical image 110 according to the medical image 110. 1…i The time series of the normalized medical image 140 is generated S120a. 1…i The output operation includes outputting S130a, S130a' of the normalized medical image 140 1…i time series and / or from the normalized medical image 140 1…i In addition, the normalized medical image 140 1…i The time series of represents images generated by a common type of imaging modality.

[0051] Therefore, in the first set of examples, the normalized medical image 140 is output 1…i time series, such as Figure 3 shown.

[0052] Continuing with the first set of examples, in one example, the first type of imaging modality 120 is a projection imaging modality, and the second type of imaging modality 130, 130' is a volumetric imaging modality; and the common type of imaging modality is a projection imaging modality. In this example, by providing a time series of normalized medical images as a time series 110 generated by the first type of imaging modality 120 1…i The normalized medical image 140 is generated by combining one or more medical images and one or more projection images. 1…i , and wherein the one or more projected images are provided by projecting one or more medical images generated by the second type of imaging modality 130, 130', such that the one or more projected images correspond to one or more of the medical images generated by the first type of imaging modality 120. Thus, the normalized medical image 140 1…i The temporal sequence includes projection images generated by a first type of imaging modality and projection images generated by projecting a volumetric image generated by a second type of imaging modality.

[0053] In this example, the normalized medical image 140 1…iThe temporal sequence of is thus projection images, and the volumetric image generated by the second type of imaging modality 130, 130' is projected so as to provide a simulated or "virtual" projection image corresponding to one or more of the projection images generated by the first imaging modality. Figure 3 , and wherein a projection image generated by a projection X-ray imaging system labeled “PXR” provides a normalized medical image 140 in the lower portion of the figure. 1…i and providing other “virtual” projection images in the normalized time series by projecting a volumetric image generated by a CT imaging system labeled “CT”.

[0054] Typically, in the normalized medical image 140 1…i In a time series of , the volume image can be replaced or enhanced by a projection image generated from the volume image at the corresponding time point. Figure 3 The operation of generating a simulated or "virtual" projection image from a volume image is shown in FIG. In doing so, the output normalized medical image 140 1…i A time series or output from the normalized medical image 140 1…i One or more measurements derived from a time series of the subject may be used to identify temporal changes in a region of interest and thereby assess the progression of a medical condition in a subject.

[0055] In one example, projecting one or more medical images generated by the second type of imaging modality 130, 130' such that the one or more projected images correspond to one or more medical images generated by the first type of imaging modality 120 includes using a virtual source 230 v Projecting one or more medical images generated by the second type of imaging modality onto the virtual detector 240 v Reference Figure 3 and Figure 4 This example is described. In another example described later, the neural network NN1 is used to generate a projection image.

[0056] refer to Figure 3 In this example, the first imaging modality 120 is a projection modality. For example, the first imaging modality 120 may be Figure 3 The projected X-ray modality labeled “PXR” in FIG. 1 and its time series 110 at times corresponding to day 0, day 2, day 7, and day 12 1…i In this example, the second imaging modality is a volume imaging modality. For example, the second imaging modality may be a Figure 3 The CT imaging modality is labeled “CT” in FIG, and it generates volumetric CT images at times corresponding to day 5, day 10, and day 14. Figure 4 is a diagram illustrating the use of virtual source 230 according to some aspects of the present disclosure. v Projecting the volumetric medical image 1103 onto the virtual detector 240 v A schematic diagram of an example of an operation on . Figure 4 The example volume image generated on day 3 (ie, the CT image 1103 representing the chest) is projected onto the virtual detector 240 v 1403 to generate a normalized medical image 1403. The normalized medical image 1403 is a projection image corresponding to the projection X-ray image generated by the first imaging modality 120 (i.e., the projection X-ray imaging modality). The normalized medical image 1403 is then included in the normalized medical image 1403 at the time corresponding to the generation of the volumetric image (i.e., day 5). 1…i in the time series.

[0057] The operation of projecting the volume image 1103 onto the virtual detector may be performed mathematically using a ray tracing operation. Figure 4 In this operation, the density value represented by the volume image 1103 is projected along the path from the virtual X-ray source 230v through the volume image 1103 to the virtual X-ray detector 240 v The paths of virtual rays (such as the thick dashed lines) on the virtual X-ray detector 240 are integrated to obtain the v The integrated density value may also be adjusted to compensate for X-ray scatter. By performing the ray tracing operation and calculating the integrated density value across the virtual X-ray detector 240 v The integrated density values ​​at multiple positions of the virtual source 230 are used to generate the projection image 1403. These operations are performed mathematically, that is, using the virtual source 230 v and Virtual Detector 240 v .

[0058] The operation of projecting the volumetric image 1103 onto the virtual detector can be performed based on the known relative positioning between the virtual source, the virtual detector, and the object represented in the one or more medical images generated by the second type of imaging modality 120. This provides (one or more) projection images corresponding to one or more of the medical images generated by the first type of imaging modality 120. When the projection images are generated using the first type of imaging modality, the relative positioning of the virtual source and the virtual detector can be known from the relative positions of the actual source and the actual detector of the first imaging modality relative to the object. These relative positions can be known by performing imaging using standard clinical imaging protocols. The protocol can define parameters such as the spacing between the actual X-ray source and the actual detector, the position of the object relative to the detector, the image resolution, etc. For example, if a so-called "PA" projection image is generated by the first imaging modality with the object standing and facing an upright X-ray detector and the object having a specific distance from each of the X-ray source and the X-ray detector, the same relative positions will be used to generate projection images using the virtual source, the virtual detector, and the object in the (one or more) volumetric images generated by the second type of imaging modality. Alternatively, the relative position may be measured when generating the projection image using the first imaging modality.For example, the relative position may be measured using a (depth) camera that captures the actual X-ray source, the actual X-ray detector and the object when generating the projection image.

[0059] Alternatively or additionally, the operation of projecting the volumetric image 1103 onto the virtual detector may include adjusting the relative positioning between the virtual source, the virtual detector, and the one or more medical images generated by the second type of imaging modality 130, 130' so that the shape of one or more anatomical features in the projected one or more images corresponds to the shape of one or more corresponding anatomical features in the one or more medical images generated by the first type of imaging modality 120. In this case, the shape correspondence provides that the (one or more) projected images correspond to one or more of the medical images generated by the first type of imaging modality 120. In this case, an optimization process is performed in which the relative positioning between the virtual source, the virtual detector, and the one or more medical images generated by the second type of imaging modality 130, 130' is iteratively adjusted until the difference between the shape of the one or more anatomical features in the (one or more) projected images and the shape of the one or more corresponding anatomical features in the one or more medical images generated by the first type of imaging modality 120 is less than a predetermined threshold. The starting point for the optimization may be a rough relative position expected for the image type and the region of interest in the image. Alternative starting points for the optimization may be expected relative positions from standard imaging protocols, or measured relative positions measured using a (depth) camera.In this example, the shape of the anatomical feature in the image may be determined by applying a known image segmentation algorithm to the corresponding image.

[0060] In these examples where a volumetric image is projected onto a virtual detector, the use of the virtual source 230 may additionally be performed based on known operating parameters of the projection imaging modality (ie, the first type of imaging modality). v Projecting one or more medical images generated by the second type of imaging modality onto the virtual detector 240 v For example, if the first imaging modality is a projection X-ray imaging modality, the X-ray source may be additionally configured based on one or more operating parameters (such as the kV p The projection operation is performed using parameters such as energy, X-ray dose, exposure time, sensitivity of the X-ray detector, etc. By additionally using such parameters, an improved correspondence can be achieved between the projection image 1403 and one or more medical images generated by the first type of imaging modality 120.

[0061] As described above, in addition to the challenges of acquiring images in a time series using different types of imaging modalities during the course of a clinical investigation, acquiring the time series 110 1…i There may also be differences in the way medical images are viewed in Figure 3 For example, there may be differences in the subject's posture, or differences in the viewing angle of the medical imaging system, or differences in the ionizing radiation dose used to acquire the images. Such factors may exist even when the images are acquired from a single type of imaging modality. These factors exacerbate the challenge of identifying temporal changes in the region of interest and may even lead to an erroneous diagnosis of the subject's condition.

[0062] The normalized medical image 140 may be 1…i The method may further improve the identification of temporal changes in the region of interest and thereby improve the assessment of the progression of the medical condition in the subject.

[0063] refer to Figure 1 In one example, the method shown in FIG. 120a generates a normalized medical image 140. 1…i The time series operation includes making the normalized medical image 140 1…i One or more images in the time sequence of warping, so that the warped one or more images 140 ′ 1,2 The shape of one or more anatomical features 160 in the image 170 corresponds to the shape of one or more anatomical features 160 in the reference image 170 .

[0064] In this example, the effect of the warping is to provide an image in which anatomical features have similar shapes. This enables the normalized medical image 140 to be 1…iIn this example, the received medical image 110 1…i A time series of images or a normalized medical image 140 1…i The time series images or atlas images provide reference images 170. By way of some examples, Figure 5 An atlas image 170 for a lung field is shown according to some aspects of the present disclosure. L example, and Figure 6 An atlas image 170 for the thorax is shown according to some aspects of the present disclosure. R In this example, for example, the warping operation may include making the normalized medical image 140 1…i Warping one or more images in the time series so that the shape of the lungs or the shape of the thorax in the warped one or more images corresponds to the shape of the lungs or the thorax in the time series Figure 5 and Figure 6 The shape of the lungs or the shape of the thorax in the atlas image shown in FIG.

[0065] Figure 5 and Figure 6 The atlas image 170 shown in L，R The probability of a structure belonging to the lung boundary and the ribs is indicated by their intensity values, respectively. For example, Figure 2 The one or more processors 210 shown in FIG. 1 receive the atlas image 170 L，R The atlas image may be received from a database of atlas images. 1…i The time series of the imaged object and the atlas image 170 L，R The atlas image 170 is selected from the database based on the similarity between the object and the reference object represented in FIG. For example, the atlas image can be selected based on the similarity between the age, gender, and body shape of the object and the reference object. L，R A reference shape is provided for an anatomical feature 160 (i.e., a lung) or a rib. The atlas image can represent a preferred viewing angle of the anatomical feature in a reference subject. The atlas image can be acquired while the reference subject maintains a preferred posture. By way of example, if the anatomical feature is a lung, the atlas image of the lung can be a so-called posteroanterior "PA" chest view, which represents the lungs, bony thorax, mediastinum, and great vessels. Such an atlas image can be acquired with the subject standing and facing an upright X-ray detector in a specified inspiratory state, with the top of the X-ray detector a specified distance above the shoulder joint, the chin lifted out of the image field, the shoulder rotated forward to allow the scapula to move laterally away from the lung field, and at specified operating settings of the X-ray imaging system (e.g., X-ray kVp energy and exposure time).

[0066] In the warping operation, the normalized medical image 1401…i The time series is warped to the atlas image 170 L，R The warping operation may be performed using various known transforms. One example of a suitable transform is an affine transform. Other examples of suitable transforms include B-splines, thin plate splines, and radial basis functions. The warping operation may be performed based on a plurality of corresponding landmarks 180 represented in the warped image and the reference image 170. i…j The warping operation can be performed using a neural network. If the warping is performed by a neural network, the neural network can use such landmarks explicitly or implicitly. The landmarks can be provided by anatomical features or by fiducial markers. 1…j By way of some examples, anatomical features (such as bones, e.g., ribs, shoulder blades, etc.) or organ contours (e.g., contours of the lungs, diaphragm, heart shadow, etc.) may be used as anatomical landmarks. Such features are more visible in the normalized medical image 140 by virtue of their X-ray attenuation. 1…i Fiducial markers are identifiable in a time series of images. Fiducial markers formed of X-ray attenuating material are also identifiable in medical images, and these markers can also be used as landmarks. Fiducial landmarks can be located on the body surface or inside the body and are located at known reference positions. In the latter case, the fiducial markers may have been implanted to be used, for example, as a surgical guide. Implanted devices such as pacemakers can also be used as fiducial markers. Fiducial markers can also be provided by interventional devices inserted into the body.

[0067] Various techniques may be used to identify the normalized medical image 140 1…i Landmarks in the time series 180 i…j and corresponding landmarks in the reference image 170. In one example, a feature detector is used to identify the normalized medical image 140 1…i The identified landmarks are then mapped to corresponding anatomical landmarks marked in the reference image 170. In this example, the reference image 170 includes a plurality of marked anatomical landmarks; and the reference image 170 includes a plurality of marked anatomical landmarks; Figure 1 The described methods also include:

[0068] Applying feature detectors to normalized medical images 140 1…i to identify a plurality of landmarks in the medical image corresponding to the labeled anatomical landmarks; and

[0069] Therein, the mapping is determined using the identified corresponding landmarks.

[0070] In this example, applying feature detectors to medical images in time series can be performed using edge detectors or model-based segmentation or neural networks. By way of example, Figure 7An example of the results of applying a feature detector to a medical image 1401 to identify a plurality of landmarks in the medical image is shown in accordance with some aspects of the present disclosure. Figure 7 The medical image 1401 shown in FIG. 1 represents a chest and includes bony regions such as ribs and a spine as well as contours of a heart shadow and lungs. In this example, the anatomical region used in the warping operation is the lungs, and the feature detector has identified contours representing the right lung, the right side of the heart shadow, the right side of the diaphragm, the left lung, the left side of the heart shadow, and the left side of the diaphragm in the medical image 1401 as landmarks 180 . 1…6 These landmarks correspond to landmarks in the atlas image, such as Figure 5 Atlas image of the lung fields shown in 170 L The landmarks 180 that have been identified in the medical image 1401 1…6 Then it is mapped to its atlas image 170 L , so as to warp the medical image 1401 to the atlas image in a warping operation.

[0071] The result of the warping operation is to provide a warped medical image 140' 1…i , wherein the shape of one or more anatomical features 160 corresponds to the shape of one or more anatomical features 160 in reference image 170. The medical image 140′ may be warped using, for example 1…i Correspondence is measured by comparing the shape of the anatomical feature(s) in the image with a distance measurement between corresponding landmarks in the reference image 170. Alternatively, the medical image 140′ may be warped by computationally 1…i The correspondence is measured by the value of the Dice coefficient between the segmentation mask in and the segmentation mask of the reference image 170. When the calculated value of such a measure is within a predetermined range, the shapes can be considered to correspond to each other. The effect of the warping operation is Figure 8 It is shown in Figure 8 An example of warping of two 2D medical images 1401, 1402 in a time series to a reference image is shown in accordance with some aspects of the present disclosure. Figure 8 The two 2D medical images 1401, 1402 shown on the left side of FIG. 1 are projection X-ray images generated by the same type of imaging modality, a projection X-ray "PXR" imaging system. Although images 1401 and 1402 represent the same anatomical region in a subject (i.e., the lungs), the images were acquired with the subject in different postures. The difference in the subject's posture confounds the identification of temporal changes in the medical condition of the lungs between images 1401 and 1402. Figure 8The warped images 140'1 and 140'2 on the right side of FIG. 1 have been warped so that the subject's lungs correspond to the lung fields in the atlas image, as described above. As shown in the warped images 140'1 and 140'2, the warping operation has the effect of compensating for differences in the subject's pose when the images 1401 and 1402 were acquired. Thus, the warping operation allows for a more reliable identification of temporal changes in the subject's lungs, and therefore allows for a more reliable assessment of the progression of a medical condition in the subject.

[0072] As described above, the normalized medical image 140 may also be 1…i other operations are performed on the time series in order to compensate for the differences in the way the medical images were acquired. These operations may be performed instead of or in addition to the warping operations. The inventors have observed that not only does the difference in the shape of the anatomical region between the images hinder the assessment of temporal changes in the medical images, but the assessment of such changes may also be hindered by differences in the intensity scale of the images. More specifically, it is the differences in the intensity scale that arise from differences in the way the medical images in the time series were acquired that may hinder their comparison. By way of example, the intensity at any point in a 2D X-ray image depends on the value of the X-ray energy "kVp" used to acquire the image, the exposure time and the sensitivity of the X-ray detector. Reference Figure 3 , since the images in the time series are collected over a period of time 110 1…i , the images in the time series may be generated by a projection X-ray imaging system "PXR" with different settings, or indeed they may be generated by different X-ray imaging systems. In contrast to computed tomography images, and where a Hounsfield unit value may be assigned to each image voxel, in 2D projection X-ray images there is no absolute attenuation scale corresponding to the pixel intensity values.

[0073] In one example, generating S120a a normalized medical image 140 1…i Time series operations include:

[0074] Adjusting the normalized medical image 140 based on the intensity at one or more locations in the reference image 170 1…i The intensity of the images in the time series;

[0075] and / or

[0076] Using image style transfer algorithms to adjust normalized medical images 140 1…i The intensity of the images in the time series.

[0077] These adjustments are provided to compensate for image-to-image variability caused by differences in the way medical images are acquired.

[0078] refer to Fig. 9 The example is described, Fig. 9 An example of warping two 2D medical images 1401, 1402 in a time series to a reference image and adjusting the intensity of the medical images according to some aspects of the present disclosure is shown. In this example, a warping operation is performed on 2D projection X-ray "PXR" medical images 1401 and 1402 to provide warped images 140'1 and 140'2, as described above.

[0079] In this example, the position may be defined manually or automatically in the reference image 170. By way of example, if Fig. 9 Normalized medical image 140 in 1…i The time series represents the lungs, Figure 5 The locations in the reference image 170 shown may be defined within a lung field or within the diaphragm. The locations may be predefined in the reference image 170 or defined based on user input provided via a user input device in conjunction with the displayed image. Techniques such as windowing and histogram normalization are then used to adjust the position of the image in the reference image 170. Fig. 9 The intensity values ​​of the time series of the normalized medical images 140'1 and 140'2 at the center of Fig. 9 The right side of provides intensity adjusted images 140"1 and 140"2.

[0080] Alternatively or additionally, an image style transfer algorithm may be used to provide the intensity adjusted images 140"1 and 140"2. Various image style transfer transforms are known for this purpose. For example, these may be based on, for example, a Laplacian pyramid or a (convolutional) neural network. An example of an image style transfer algorithm is disclosed in document WO2013 / 042018A1. The document discloses a technique for transforming a slave image, which involves generating a color or grayscale transform based on a master image and a slave image. The transform is used to optically adapt the slave image to the master image. The technique may be used to adjust the medical images 140"1 and 140"2 in a time series. 1…i The intensity of the normalized medical image 140' 1…i The adjusted intensity values ​​in the time series have a similar appearance or style. 1…i as a slave image and using a reference image such as Figure 5 and Figure 6 ) or medical images 110 in a time series 1…i One of them is used as the main image to apply this document.

[0081] In a related example, the normalized medical image 140 is adjusted based on the intensity at one or more locations in the reference image 170. 1…iOperations on the intensity of images in a time series can include:

[0082] calculating an average intensity value within the portion of the reference image 170; and

[0083] Using the average intensity value to normalize the medical image 140 1…i The intensity values ​​in each medical image in the time series are normalized.

[0084] In this example, the reference image 170 is used to identify each medical image 140. 1…i The portion of the image whose intensity value is normalized in the medical image 110 may be selected as a portion of the reference image, wherein the image intensity in the reference image is expected to be 1…i For example, if the region of interest is the lungs, the selected portion of the reference image may correspond to a portion of the spine, since the attenuation of the spine is not expected to change over time. Alternatively, if the region of interest is the lungs, the selected portion may correspond to a portion of the medical image 110 that is not expected to be present in the reference image. 1…i For example, if the condition of the subject being investigated is pulmonary edema, the collection of fluid over time may be expected to result in temporal changes in image intensity in the lower portion of the lung, while the upper portion of the lung may be expected to remain free of disease and therefore have a stable image intensity. In this case, the upper portion of the lung in reference image 170 may be used as a reference image for the medical image 140. 1…i The part where the intensity values ​​in are normalized.

[0085] The portion of the lung in the reference image 170 may be identified automatically or based on user input received via a user input device in conjunction with the display device. The portion of the reference image 170 may represent a combination of air in the lung and ribs surrounding the lung. The average intensity of the pixel values ​​in the portion of the reference image may then be calculated. This average may then be used to compare the medical image 140 to the lung. 1…i The intensity values ​​of the time series are normalized. Figure 5 For example, this may include: identifying a corresponding portion of the lung that is free of disease, such as the top of the lung in the lung field toward the atlas image; calculating an average of intensity values ​​in the corresponding portion that is free of disease; calculating a scaling factor as a ratio of the average intensity value in the disease-free region of the medical image 1401 to the average intensity value in the portion of the lung in the atlas image; and scaling the medical image 1401 using the scaling factor. 1…i The image intensity values ​​in the time series are scaled to provide the adjusted intensity values ​​in the intensity-adjusted medical images 140″1, 140″2. Then, the other medical images 140 in the time series are 1…iThese operations are repeated. Since the lung portion in the atlas image is used to identify each medical image 140 in the time series 1…i With normalization, medical images are scaled to a common reference intensity, which facilitates more reliable comparisons between image intensities in other parts of the image.

[0086] Additional portions of the reference image 170 may also be used to perform the intensity normalization described above. For example, a second region of interest may be identified within the diaphragm or within a bone region of the reference image and the second region of interest is also used to perform intensity normalization. 1…i Normalizing the intensities in the image may facilitate more accurate mapping of intensity values ​​and, therefore, more reliable identification of changes in regions of interest in the image.

[0087] In this example, it is also possible to analyze the normalized medical image 140. 1…i The value of the quality metric can be calculated and output based on the adjusted intensity values ​​in the reference image. 1…i The value of the quality metric is calculated based on the quality of the mapping between the corresponding parts of the reference image or the landmarks used in the warping operation. If the variance of the image intensity values ​​in the part of the reference image is high, the value of the quality metric is relatively low, and vice versa. The value of the output quality metric is reviewed by a clinician to reassure the accuracy of the method.

[0088] The inventors also observed that after the above warping operation, the normalized sequence medical image 140′ 1…i Some areas of the anatomical region outside the warping region may be warped into unnatural shapes. This may confuse the reviewing physician. Figure 1 The method described may also include:

[0089] Receiving a received medical image 110 1…i Input of the region of interest in the time series; and

[0090] In the normalized medical image 140 1…i One or more image features outside or within a region of interest are suppressed in a time series of images.

[0091] By suppressing one or more image features according to this example, presenting confounding information to the reviewing clinician is avoided. Fig.10 The example is described, Fig.10 An example of suppressing image features outside of a lung region of interest in a medical image 1401 is shown in accordance with aspects of the present disclosure. Fig.10a shows images from a time series of received medical images 1101 . Fig.10 b shows a region of interest in image 1101 by a dotted line. The region of interest may be automatically defined in image 1101. For example, a segmentation algorithm or a feature detector may be used to automatically define the lung region of interest. Alternatively, the region of interest may be manually defined based on user input received via a user input device in conjunction with a display device displaying image 1101. Fig.10 In c, the outline of the region of interest is shown, and this is used to define a binary mask that is applied to image 1101 in order to suppress features outside the lung region of interest. Fig.10 d shows a normalized medical image 1401, in which features outside the lung region of interest have been suppressed. Fig.10 In the images shown in , confounding features outside the lung region of interest are omitted.

[0092] exist Fig.10 In the example shown, image features outside the lung region of interest have been suppressed by mapping their pixel values ​​to black level pixel intensity values. However, these features may alternatively be suppressed in different ways. For example, their pixel values ​​may be mapped to pixel intensity values ​​different from the black level pixel intensity values, or their pixel values ​​may be gradually adjusted to a predetermined value so as to "feather out" from the region of interest. The operation of suppressing image features outside the region of interest may also be performed using a reference image (such as an atlas image).

[0093] For example, in this example, refer to Figure 1 The method described may also include:

[0094] receiving input identifying a region of interest in a reference image 170;

[0095] Mapping the contour of the region of interest from the reference image 170 to the normalized medical image 140 1…i Each of the

[0096] Adjusting the normalized medical image outside the mapping contour 140 1…i to suppress one or more image features outside the region of interest.

[0097] In this example, various example regions of interest may be predefined in a reference image and used to suppress one or more image features outside the region of interest. Alternatively, the region of interest may be defined by segmenting the reference image. In this example, since contours from the reference image are used to suppress one or more image features outside the region of interest, reliable contours are used.

[0098] The normalized medical image 140 may be 1…i Another operation that is performed in the time series of images to compensate for differences in the way medical images are acquired is to remove foreign objects from the images. Foreign objects, such as implanted devices, can be identified in the images based on their X-ray attenuation values. Feature detectors or neural networks can be used for this purpose. The image intensity values ​​of foreign objects are set to a default level, such as a black level or a white level, to avoid their presence distracting the reviewing clinician.

[0099] By way of another example, Fig.11 An example of warping two normalized medical images 1401 , 1402 to a reference image and adjusting image intensity values ​​in the images and suppressing image features outside a lung region of interest is shown in accordance with some aspects of the present disclosure. Fig.11 The effect of the combined operations of warping, adjusting image intensity values, and suppressing features outside the region of interest is shown, and is a further illustration of the amount of variability introduced by differences in the way medical images are acquired during a longitudinal study of subjects, which can be achieved according to the methods described herein.

[0100] As described above, instead of providing a normalized medical image 140 by projecting a volumetric medical image, 1…i In the second set of examples, the normalized medical image 140 is provided by inputting the medical image into the neural network NN1. 1…i One or more images in the time series of . Figure 1 and Figure 3 The method shown describes a second set of examples.

[0101] In a second set of examples, the first type of imaging modality 120 is a projection imaging modality and the second type of imaging modality 130, 130' is a volumetric imaging modality. The common type of imaging modality is a projection imaging modality. By providing a time series of normalized medical images as a time series 110 generated by the first type of imaging modality 120 1…i The normalized medical image 140 is generated by combining one or more medical images and one or more projection images. 1…i One or more projection images are provided by inputting one or more medical images generated by the second type of imaging modality 130, 130' into the neural network NN1. The neural network NN1 is trained to generate a projection image corresponding to the first type of imaging modality 120 for each of the input images.

[0102] The neural network NN1 can be provided by various architectures, including, for example, convolutional neural networks CNN, Cycle GAN, or transformer architectures. Typically, the training of a neural network involves inputting a training data set into the neural network and iteratively adjusting the parameters of the neural network until the trained neural network provides accurate outputs. Training is typically performed using a graphics processing unit "GPU" or a dedicated neural processor (such as a neural processing unit "NPU" or a tensor processing unit "TPU"). Training typically uses a centralized approach, in which a cloud-based or host-based neural processor is used to train the neural network. After training it using the training data set, the trained neural network can be deployed to a device for analyzing new input data during reasoning. The processing requirements during reasoning are significantly less than the processing requirements required during training, allowing the neural network to be deployed to various systems, such as laptops, tablet computers, mobile phones, and the like. For example, inference can be performed by a central processing unit "CPU", GPU, NPU, TPU, on a server, or in the cloud.

[0103] therefore, Figure 3 The process of training neural network NN1 shown includes adjusting its parameters. Parameters (or more specifically weights and biases) control the operation of activation functions in the neural network. In supervised learning, the training process automatically adjusts weights and biases so that when presented with input data, the neural network accurately provides the corresponding expected output data. To this end, the value of the loss function or error is calculated based on the difference between the predicted output data and the expected output data. The value of the loss function can be calculated using functions such as negative log-likelihood loss, mean square error or Huber loss or cross entropy loss. During training, the value of the loss function is usually minimized, and training is terminated when the value of the loss function meets the stopping criteria. Sometimes, training is terminated when the value of the loss function meets one or more of a plurality of criteria.

[0104] Various methods are known to solve the loss minimization problem, such as gradient descent, quasi-Newton method, etc. Various algorithms have been developed to implement these methods and their variants, including but not limited to: stochastic gradient descent "SGD", batch gradient descent, mini-batch gradient descent, Gauss-Newton, Levenberg Marquardt, impulse, Adam, Nadam, Adagrad, Adadelta, RMSProp and Amainx "optimizer". These algorithms use the chain rule to calculate the derivatives of the loss function with respect to the model parameters. This process is called back propagation because the derivatives are calculated starting at the last layer or output layer and moving toward the first layer or input layer. These derivatives tell the algorithm how the model parameters must be adjusted in order to minimize the error function. That is, the adjustments made to the model parameters start from the output layer and work backwards in the network until the input layer is reached. In the first training iteration, the initial weights and biases are usually randomized. Then, the neural network predicts the output data, which is also random. Back propagation is then used to adjust the weights and biases. The training process is iteratively performed by adjusting the weights and biases in each iteration. When the error or difference between the predicted output data and the expected output data is within an acceptable range for the training data or some validation data, the training is terminated. Subsequently, the neural network can be deployed and the trained neural network uses the trained values ​​of its parameters to make predictions for new input data. If the training process is successful, the trained neural network accurately predicts the expected output data from the new input data.

[0105] In one example, the neural network NN1 is trained to generate a projection image corresponding to the first type of imaging modality 120 for each of the input images by:

[0106] receiving training data, the training data comprising a plurality of volumetric training images representing the region of interest, the volumetric training images being generated by an imaging modality 130, 130' of the second type;

[0107] receiving live data comprising, for each of the volumetric training images, a corresponding live projection image generated by a first type of imaging modality 120;

[0108] Inputting the training data into the neural network NN1; and

[0109] For each of multiple input volume training images:

[0110] Using the neural network NN1 to predict the corresponding projection image;

[0111] Adjusting the parameters of the neural network NN2 based on the difference between the predicted projection image and the actual projection image; and

[0112] The predicting and the adjusting are repeated until a stopping criterion is met.

[0113] By way of example, the first type of imaging modality may be a projection X-ray imaging modality, and the second type of imaging modality may be a CT imaging modality. In this example, the training data may include a volumetric image of a chest generated by a CT imaging modality. The training data may include hundreds or thousands or more such images. The live data includes a projection image of a chest region generated by an X-ray projection imaging modality. The live data may be provided by a pseudo X-ray projection image generated from a volumetric training image of a chest generated by a CT imaging modality, wherein the volumetric image is projected to provide a pseudo X-ray projection image. The projection may be performed according to the above-described techniques. For example, the volumetric image may be projected from a predefined orientation and using predefined values ​​for a dose of an X-ray source, an exposure time, and a sensitivity of an X-ray detector to provide a live pseudo X-ray projection image. Alternatively, the live projection image may be provided by a projection imaging system (in this example, a projection X-ray imaging system). The adjustment of the parameters of the neural network NN1 may be performed according to the above-described back-propagation technique. The resulting trained neural network NN1 is trained so that when a new CT image is provided, it outputs a corresponding projection image. Similarly, the neural network NN1 can be trained to predict X-ray projection images based on images generated by an imaging modality different from the CT imaging modality. For example, alternatively, the first imaging modality can be an MRI imaging modality, and the neural network NN1 is trained to predict the corresponding X-ray projection images. In this case, the corresponding live data can be provided by a projection X-ray imaging system. The training of the neural network NN1 is then performed in a similar manner.

[0114] A normalized medical image 140 has been provided 1…i The time series of 110 is generated by the first type of imaging modality 120 as a time series 110 1…i and one or more projection images generated by the neural network NN1; warping the medical image 140 1…i , and similarly adjust the normalized medical image 140 1…i The intensity of the image in the time series can then be used to normalize the medical image 140 1…i The time series performs suppression of one or more image features outside the region of interest in order to further reduce the impact of differences in the way the images were acquired.

[0115] In the third set of examples, following Figure 1 In the third set of examples, a medical image 110 is generated. 1…iOne or more normalized measurements of a region of interest in a time series 150 1…j , and output one or more normalized measurement values ​​150 1…j Therefore, compared with the first and second groups of examples, there is no need to generate the time-normalized medical image 140. 1…i The time series operation and, conversely, directly based on the medical image 110 1…i One or more normalized measurements are generated from the time series 150 1…j .

[0116] Refer to the above reference Figure 1 The method described herein comprises generating an operation according to a medical image 110 1…i The time series generation S120b represents the medical image 110 1…i One or more normalized measurements of a region of interest in a time series 150 1…i The output operation includes outputting S130b one or more normalized measurement values ​​150 1…i The one or more normalized measurements representing the region of interest are generated by: 1…i :

[0117] Medical image 110 1…i The time series of is input into the neural network NN2; and

[0118] The neural network NN2 is trained to generate a representation for the input image 110 1…i The normalized measurement value of the region of interest in each image is 150 1…i .

[0119] refer to Fig.12 Describe the third set of examples, Fig.12 is a schematic diagram illustrating a second example of a method of compensating for discrepancies in a medical image according to some aspects of the present disclosure.

[0120] refer to Fig.12 , for example, the first imaging modality 120 may be a projection X-ray imaging modality labeled “PXR”. The second imaging modality may be a CT imaging modality 130 labeled “CT”, or it may be a 3D ultrasound imaging modality 130′ labeled “US”. Typically, the normalized measurements predicted by the second neural network NN2 may represent image intensity or a diagnostic metric, such as a lung perfusion index. For example, the measurements may represent a normalized medical image 140′ 1…i The intensity, volume or shape of a region of interest in a time series of . An example of a diagnostic metric is the lung perfusion index. In the example shown, the normalized measurement value 150 1…iis the lung perfusion index. Fig.12 In the example shown, a medical image 110 generated by a projection X-ray imaging modality “PXR” or a CT imaging modality “CT” or a 3D ultrasound imaging modality “US” is formed. 1…i The time series is input into the neural network NN2 to generate the normalized measurement value 150 1…i , that is, the value of the lung perfusion index.

[0121] The second neural network NN2 may be provided by various architectures, including, for example, a convolutional neural network CNN or a transformer architecture. In one example, the neural network NN2 is trained to generate normalized measurements 150 representing a region of interest for each of the input images by: 1…i :

[0122] receiving training data, the training data comprising a plurality of training images representing the region of interest, the training images comprising a plurality of images generated by the first type of imaging modality 120 and a plurality of images generated by the second type of imaging modality 130, 130';

[0123] receiving ground truth data, the ground truth data comprising, for each of the training images, corresponding ground truth values ​​representing normalized measurements of the region of interest;

[0124] Inputting the training data into the neural network NN2; and

[0125] For each of multiple input training images:

[0126] using the neural network NN2 to predict a value representing a normalized measurement of a region of interest;

[0127] adjusting the parameters of the neural network NN2 based on the difference between the predicted values ​​of the normalized measurements and the corresponding actual values; and

[0128] The predicting and the adjusting are repeated until a stopping criterion is met.

[0129] The second neural network may be trained to generate normalized measurements using the techniques described above 150 1…i , the normalized measured value 150 1…i For the input image 110 1…i The region of interest for each image.

[0130] By way of example, the first type of imaging modality may be a projection X-ray imaging modality, and the second type of imaging modality may be a CT imaging modality. In this example, the training data may include X-ray projection images of a chest generated by an X-ray projection imaging modality, and volumetric images of a chest generated by a CT imaging modality. The training data may include hundreds or thousands or more such images. For example, the normalized measurement value predicted by the second neural network NN2 may be a lung perfusion index. The live data may be provided by an expert evaluation of the images in the training data, and thus include live values ​​of the lung perfusion index. Adjustment of the parameters of the second neural network NN2 may be performed according to the above-described back-propagation technique. The resulting trained second neural network NN2 is trained such that when a new image is provided from the first imaging modality (e.g., from a CT imaging modality) or from a second imaging modality (e.g., an X-ray projection imaging modality), the second neural network NN2 outputs a predicted value of the lung perfusion index. Similarly, the second neural network NN2 may be trained to predict the lung perfusion index based on images from another type of imaging modality (such as, for example, an MRI imaging modality). The training of the second neural network NN2 is then performed in a similar manner. In general, the second neural network NN2 can be trained to generate normalized measurements 150 for medical images from a single type of imaging modality or for medical images from multiple types of imaging modalities. 1…i Independent neural networks can also be trained to generate normalized measurements for different types of imaging modalities 150 1…i .

[0131] In another example, a computer program product is provided. The computer program product includes instructions that, when executed by one or more processors, cause the one or more processors to perform a method of compensating for differences in a medical image, the method comprising:

[0132] Receiving S110 includes medical image 110 1…i a time series of image data, the time series comprising one or more medical images generated by a first type of imaging modality 120 and one or more medical images generated by a second type of imaging modality 130, 130', the first type of imaging modality being different from the second type of imaging modality;

[0133] According to medical images 110 1…i The time series generates S120a, S120b the normalized medical image 140 1…i A time series or representation of a medical image 110 1…i One or more normalized measurements of a region of interest in a time series 150 1…j ;as well as

[0134] The normalized medical images 140 are outputted respectively at S130a, S130a', and S130b. 1…i time series and / or from the normalized medical image 140 1…i One or more measured values ​​derived from a time series, or one or more normalized measured values ​​150 1…j .

[0135] In another example, a system 200 for compensating for differences in medical images is provided. The system includes one or more processors 210 configured to:

[0136] Receiving S110 includes medical image 110 1…i a time series of medical images, the time series comprising one or more medical images generated by a first type of imaging modality 120 and one or more medical images generated by a second type of imaging modality 130, 130', the first type of imaging modality being different from the second type of imaging modality;

[0137] According to medical images 110 1…i The time series generates S120a, S120b the normalized medical image 140 1…i A time series or representation of a medical image 110 1…i One or more normalized measurements of a region of interest in a time series 150 1…j ;as well as

[0138] The normalized medical images 140 are outputted respectively at S130a, S130a', and S130b. 1…i time series and / or from the normalized medical image 140 1…i One or more measured values ​​derived from a time series, or one or more normalized measured values ​​150 1…j .

[0139] Figure 2 An example of the system 200 is shown in FIG. Note that the system 200 may also include one or more of the following: a plurality of medical imaging systems for generating the image data received in operation S110, such as, for example Figure 2 The projection X-ray imaging system 220 and the CT imaging system or the MRI imaging system shown in FIG. 1 ; a display device, such as a monitor 250 , a tablet computer, etc., for displaying output data, such as the normalized medical image 140 1…i time series, from normalized medical images 140 1…i One or more measured values ​​derived from a time series, one or more normalized measured values ​​150 1…jetc.; patient bed 260; and, user input device ( Figure 2 ), which is used to receive user input related to the operations performed by the system, such as keyboard, mouse, touch screen, etc.

[0140] An enumerated list of examples of the present disclosure is provided below:

[0141] Example 1. A computer-implemented method for compensating for discrepancies in a medical image, the method comprising:

[0142] Receive (S110) a medical image (110 1…i ), the time series comprising one or more medical images generated by a first type of imaging modality (120) and one or more medical images generated by a second type of imaging modality (130, 130'), the first type of imaging modality being different from the second type of imaging modality;

[0143] According to medical images (110 1…i ) time series generation (S120a, S120b) of a normalized medical image (140 1…i ) time series, or represents a medical image (110 1…i ) of the region of interest in the time series (150 1…j );as well as

[0144] The normalized medical images (140) are outputted respectively (S130a, S130a', S130b). 1…i ) of the time series and / or from the normalized medical image (140 1…i ), or outputting the one or more normalized measured values ​​(150 1…j ).

[0145] Example 2. The computer-implemented method of Example 1, wherein the generating comprises generating a medical image (110 1··i ) time series generates (S120a) a normalized medical image (140 1··i ) time series; and

[0146] The output includes outputting (S130a, S130a') the normalized medical image (140 1··i ) time series and / or from a normalized medical image (140 1··i ); and

[0147] Wherein, the normalized medical image (1401··i ) represent images generated by a common type of imaging modality.

[0148] Example 3. A computer-implemented method according to Example 2, wherein the first type of imaging modality (120) is a projection imaging modality and the second type of imaging modality (130, 130') is a volumetric imaging modality; and wherein the common type of imaging modality is the projection imaging modality; and

[0149] wherein the normalized time series of medical images is provided as a time series (110) generated by the first type of imaging modality (120) 1··i ) and one or more projection images to generate the normalized medical image (140 1··i ), and wherein the one or more projection images are provided by projecting the one or more medical images generated by the second type of imaging modality (130, 130'), so that the one or more projection images correspond to one or more of the medical images generated by the first type of imaging modality (120).

[0150] Example 4. A computer-implemented method according to Example 3, wherein the projecting includes projecting the one or more medical images generated by the second type of imaging modality onto a virtual detector (240v) using a virtual source (230v).

[0151] Example 5. A computer-implemented method according to Example 3 or Example 4, wherein the projection is based on a known relative positioning (120) between the virtual source, the virtual detector, and an object represented in the one or more medical images generated by the second type of imaging modality; and / or

[0152] Wherein, the projection includes adjusting the relative positioning between the virtual source, the virtual detector and the one or more medical images generated by the second type of imaging modality (130, 130') so that the shape of one or more anatomical features in the projected one or more images corresponds to the shape of one or more corresponding anatomical features in the one or more medical images generated by the first type of imaging modality (120).

[0153] Example 6. The computer-implemented method of any one of Examples 2-5, wherein said generating (S120a) a normalized medical image (S140 1··i ) time series include:

[0154] Warping the normalized medical image (140 1··i) in a time sequence such that a shape of one or more anatomical features (160) in the warped one or more images (140'1,2) corresponds to a shape of one or more anatomical features (160) in a reference image (170).

[0155] Example 7. The computer-implemented method of Example 6, wherein the warping is based on a plurality of corresponding landmarks (180) represented in both the warped image and the reference image (170). i··j ) between them.

[0156] Embodiment 8: The computer-implemented method according to any one of Examples 2-7, wherein said generating (S120a) a normalized medical image (S140 1··i ) time series include:

[0157] The normalized medical image (140) is adjusted based on the intensity at one or more locations in the reference image (170). 1··i )’s intensity in the time series of images;

[0158] and / or

[0159] Using an image style transfer algorithm to adjust the normalized medical image (140 1··i )’s time series.

[0160] Embodiment 9: The computer-implemented method according to any one of embodiments 2-8, wherein the method further comprises:

[0161] Receiving a medical image received by a definition (110 1··i )’s time series; and

[0162] In the normalized medical image (140 1··i ) in a time series of suppressing one or more image features outside the region of interest or within the region of interest.

[0163] Embodiment 10. The computer-implemented method of Example 9, wherein the region of interest is defined in the reference image (170), and wherein the method further comprises:

[0164] Mapping the contour of the region of interest from the reference image to the normalized medical image (140 1··i ) time series; and

[0165] Adjusting the normalized medical image outside the mapping contour or within the mapping contour, respectively (140 1··i) in a time series to suppress the one or more image features outside the region of interest.

[0166] Embodiment 11. A computer-implemented method according to any one of Examples 6-10, wherein the reference image (170) is provided by: 1··i ) of a time series, or from the normalized medical image (140 1··i ) time series images, or atlas images.

[0167] Embodiment 12. A computer-implemented method according to embodiment 2, wherein the first type of imaging modality (120) is a projection imaging modality and the second type of imaging modality (130, 130') is a volumetric imaging modality; and wherein the common type of imaging modality is the projection imaging modality; and

[0168] wherein the normalized time series of medical images is provided as a time series of medical images from the time series (110) generated by the first type of imaging modality (120) 1··i ) and one or more projection images to generate the normalized medical image (140 1··i ), and wherein the one or more projection images are provided by inputting the one or more medical images generated by the second type of imaging modality (130, 130') into a neural network (NN1); and

[0169] The neural network (NN1) is trained to generate a projection image corresponding to the first type of imaging modality (120) for each of the input images.

[0170] Example 13. A computer-implemented method according to Example 12, wherein the neural network (NN1) is trained to generate a projection image corresponding to the first type of imaging modality (120) for each of the input images by:

[0171] receiving training data, the training data comprising a plurality of volumetric training images representing the region of interest, the volumetric training images being generated by an imaging modality (130, 130') of the second type;

[0172] receiving live data, the live data comprising, for each of the volumetric training images, a corresponding live projection image generated by an imaging modality (120) of the first type;

[0173] inputting the training data into the neural network (NN1); and

[0174] For each of multiple input volume training images:

[0175] Predicting a corresponding projection image using the neural network (NN1);

[0176] Adjusting the parameters of the neural network according to the difference between the predicted projection image and the actual projection image;

[0177] The predicting and the adjusting are repeated until a stopping criterion is met.

[0178] Example 14. A computer-implemented method according to Example 1, wherein the generating comprises: 1··i ) generates a time series (S120b) representing the medical image (110 1··i ) of the region of interest in the time series (150 1··i );and

[0179] The output includes outputting (S130b) the one or more normalized measurement values ​​(150 1··i );as well as

[0180] The one or more normalized measurement values ​​representing the region of interest are generated by the following operations (150): 1··i ):

[0181] The medical image (110 1··i ) is input into a neural network (NN2), and in response to the input, the neural network (NN2) is used to generate a representation for the input image (110 1··i ) of each of the regions of interest (150 1··i );as well as

[0182] wherein the neural network (NN2) is trained to generate the normalized measurement value (150) representing the region of interest for each of the input images (1101··i) using training data 1··i ), the training data comprises a plurality of training images representing the region of interest, the training images comprising a plurality of images generated by the first type of imaging modality (120) and a plurality of images generated by the second type of imaging modality (130, 130'), and for each training image, the corresponding actual value of the normalized measurement value represents the region of interest.

[0183] Example 15. A computer-implemented method according to Example 14, wherein the neural network (NN2) is trained to generate the normalized measurement value (150) representing the region of interest for each of the input images by: 1··i ):

[0184] receiving the training data; and

[0185] For each of the multiple training images in the training data:

[0186] Inputting the training image into the neural network;

[0187] predicting, using the neural network (NN2), a value representing the normalized measurement of the region of interest;

[0188] adjusting parameters of the neural network (NN2) based on differences between predicted values ​​of the normalized measurements and corresponding actual values; and

[0189] The predicting and the adjusting are repeated until a stopping criterion is met.

[0190] The above examples should be understood as illustrating the present invention, rather than being restrictive. Further examples are also contemplated. For example, the examples described about the computer-implemented method may also be provided by a computer program product or by a computer-readable storage medium or by a system 200 in a corresponding manner. It should be understood that the features described about any one example may be used alone or in combination with the features described with other features, and may be used in combination with one or more features of another example or a combination of other examples. In addition, the above equivalents and modifications may also be adopted without departing from the scope of the present invention defined in the appended claims. In the claims, the word "including" does not exclude other elements or operations, and the indefinite article "one" or "an" does not exclude multiple. The fact that certain features are described in mutually different dependent claims does not indicate that the combination of these features cannot be used to advantage. Any figure mark in the claim should not be interpreted as limiting its scope.

Claims

1. A computer-implemented method of compensating for discrepancies in a medical image, the method comprising: Receive (S110) a medical image (110 1··i ), the time series comprising one or more medical images generated by a first type of imaging modality (120) and one or more medical images generated by a second type of imaging modality (130, 130'), the first type of imaging modality being different from the second type of imaging modality; According to the medical image (110 1··i ) generates (S120a, S120b) a normalized medical image (140) representing an image generated by a common type of imaging modality 1··i ) time series; as well as Output (S130a, S130a', S130b) the normalized medical image (140 1··i ) of the time series and / or from the normalized medical image (140 1··i ) is a time series derived from one or more measurements.

2. The computer-implemented method of claim 1 , wherein: The first type of imaging modality (120) is a projection imaging modality and the second type of imaging modality (130, 130') is a volumetric imaging modality; and wherein the common type of imaging modality is the projection imaging modality; and wherein the normalized time series of medical images is provided as the time series (110) generated by the first type of imaging modality (120) 1··i ) and one or more projection images to generate the normalized medical image (140 1··i ), and wherein the one or more projection images are provided by projecting the one or more medical images generated by the second type of imaging modality (130, 130'), so that the one or more projection images correspond to one or more of the medical images generated by the first type of imaging modality (120).

3. The computer-implemented method of claim 2, wherein: The projection includes using a virtual source (230 v ) projecting the one or more medical images generated by the second type of imaging modality onto a virtual detector (240 v )superior.

4. A computer-implemented method according to claim 2 or claim 3, wherein: The projection is based on a known relative positioning of the virtual source, the virtual detector, and an object represented in the one or more medical images generated by the second type of imaging modality (120); and / or The projection includes adjusting the relative positioning between the virtual source, the virtual detector and the one or more medical images generated by the second type of imaging modality (130, 130'), so that the shape of one or more anatomical features in the projected one or more images corresponds to the shape of one or more corresponding anatomical features in the one or more medical images generated by the first type of imaging modality (120).

5. The computer-implemented method of any one of claims 1 to 4, wherein: The generating (S120a) of a normalized medical image (140 1··i ) time series include: Warping the normalized medical image (140 1··i ) in a time sequence, so that the warped one or more images (140' 1,2 ) corresponds to the shape of the one or more anatomical features (160) in the reference image (170).

6. The computer-implemented method of claim 5, wherein: The warping is based on a plurality of corresponding landmarks (180) represented in both the warped image and the reference image (170). i··j ) between them.

7. The computer-implemented method of any one of claims 1 to 6, wherein: The generating (S120a) of a normalized medical image (140 1··i ) time series include: The normalized medical image (140) is adjusted based on the intensity at one or more locations in the reference image (170). 1··i )’s intensity in the time series of images; and / or Using an image style transfer algorithm to adjust the normalized medical image (140 1··i ) in a time series.

8. The computer-implemented method of any one of claims 1 to 7, wherein: The method further comprises: Receiving a received medical image (110 1··i )’s time series; and Suppressing the normalized medical image (140 1··i ) in a time series of one or more image features outside or within the region of interest.

9. The computer-implemented method of claim 8, wherein: The region of interest is defined in the reference image (170), and wherein the method further comprises: Mapping the contour of the region of interest from the reference image to the normalized medical image (140 1··i ) in a time series; and The normalized medical image is adjusted outside the mapped contour or within the mapped contour, respectively (140 1··i ) in a time series to suppress the one or more image features outside the region of interest.

10. The computer-implemented method of any one of claims 5 to 9, wherein: The reference image (170) is provided by: a received medical image (110 1··i ) of a time series; or from the normalized medical image (140 1··i ) time series image; or atlas image.

11. The computer-implemented method of claim 1 , wherein: The first type of imaging modality (120) is a projection imaging modality and the second type of imaging modality (130, 130') is a volumetric imaging modality; and wherein the common type of imaging modality is the projection imaging modality; and wherein the normalized time series of medical images is provided as the time series (110) generated by the first type of imaging modality (120) 1··i ) and one or more projection images to generate the normalized medical image (140 1··i ), and wherein the one or more projection images are provided by inputting the one or more medical images generated by the second type of imaging modality (130, 130') into a neural network (NN1), and in response to the input, the one or more projection images are generated using the neural network (NN1); and The neural network (NN1) is trained to generate a projection image corresponding to the first type of imaging modality (120) for each of the input images using training data, the training data comprising a plurality of volume training images representing the region of interest, the volume training images being generated by the second type of imaging modality (130, 130'), and for each volume training image, a corresponding live projection image being generated by the first type of imaging modality (120).

12. The computer-implemented method of claim 11, wherein: The neural network (NN1) is trained to generate a projection image corresponding to the first type of imaging modality (120) for each of the input images by: receiving the training data; and For each of the plurality of volumetric training images in the training data: inputting the volumetric training image into the neural network; Predicting a corresponding projection image using the neural network (NN1); adjusting the parameters of the neural network (NN1) according to the difference between the predicted projection image and the actual projection image; and The predicting and the adjusting are repeated until a stopping criterion is met.

13. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform a method of compensating for discrepancies in a medical image, the method comprising: Receive (S110) a medical image (110 1··i ), the time series comprising one or more medical images generated by a first type of imaging modality (120) and one or more medical images generated by a second type of imaging modality (130, 130'), the first type of imaging modality being different from the second type of imaging modality; According to the medical image (110 1··i ) generates (S120a, S120b) a normalized medical image (140) representing an image generated by a common type of imaging modality 1··i ) time series; as well as Output (S130a, S130a', S130b) the normalized medical image (140 1··i ) of the time series and / or from the normalized medical image (140 1··i ) is a time series derived from one or more measurements.

14. A system (200) for compensating for discrepancies in a medical image, the system comprising one or more processors (210), the one or more processors (210) being configured to: Receive (S110) a medical image (110 1··i ), the time series comprising one or more medical images generated by a first type of imaging modality (120) and one or more medical images generated by a second type of imaging modality (130, 130'), the first type of imaging modality being different from the second type of imaging modality; According to the medical image (110 1··i ) generates (S120a, S120b) a normalized medical image (140) representing an image generated by a common type of imaging modality 1··i ) time series; and Output (S130a, S130a', S130b) the normalized medical image (140 1··i ) of the time series and / or from the normalized medical image (140 1··i ) is a time series derived from one or more measurements.

Images