Compensation for Differences in Medical Images

Abstract

A computer-implemented method for compensating for differences in medical images is provided. The method has a step S110 of receiving image data including a time series 110 of medical images. The time series includes one or more medical images generated by a first type of imaging method 120 and one or more medical images generated by a second type of imaging method 130, 130'. The first type of imaging method is different from the second type of imaging method. In one aspect, the method includes steps S120a, S120b of generating a normalized time series 140 of medical images from the time series 110 of medical images, and steps S130a, S130a', S130b of outputting the normalized time series 140 of medical images and / or one or more measurement values derived from the normalized time series of the medical images. In another aspect, the method includes steps S120a, S120b of generating one or more normalized measurement values 150 representing regions of interest within the time series 110 of medical images from the time series 110 of medical images, and a step of outputting the normalized measurement values 150. 1…i The method has a step S110 of receiving image data including a time series 110 of medical images. The time series includes one or more medical images generated by a first type of imaging method 120 and one or more medical images generated by a second type of imaging method 130, 130'. The first type of imaging method is different from the second type of imaging method. In one aspect, the method includes steps S120a, S120b of generating a normalized time series 140 of medical images from the time series 110 of medical images, and steps S130a, S130a', S130b of outputting the normalized time series 140 of medical images and / or one or more measurement values derived from the normalized time series of the medical images. In another aspect, the method includes steps S120a, S120b of generating one or more normalized measurement values 150 representing regions of interest within the time series 110 of medical images from the time series 110 of medical images, and a step of outputting the normalized measurement values 150. 1…i from the time series 110 of medical images, a normalized time series 140 of medical images 1…i Steps S120a, S120b of generating, and steps S130a, S130a', S130b of outputting the normalized time series 140 of medical images and / or one or more measurement values derived from the normalized time series of the medical images. 1…i and / or one or more measurement values derived from the normalized time series of the medical images 1…i from the time series 110 of medical images, the time series 110 of medical images 1…i one or more normalized measurement values 150 representing regions of interest within 1…j Steps S120a, S120b of generating, and steps S130a, S130a', S130b of outputting the normalized time series 140 of medical images and / or one or more measurement values derived from the normalized time series of the medical images. 1…j the normalized measurement values 150

Description

Technical Field

[0001] The present invention relates to compensating for differences in medical images. A computer-implemented method, a computer program product, and a system are disclosed.

Background Art

[0002] Clinical examinations often involve the collection of medical images of a subject. These images can be acquired at different times, i.e., as a time series of images. These images can be used to identify temporal changes in the region of interest and, as a result, to evaluate the progression of the subject's medical condition. However, during the process of a clinical examination, the images can be acquired using different types of imaging modalities. For example, as a result of factors such as the availability of the imaging system and the need to perform additional examinations regarding the subject's medical condition, medical images of the region of interest may be acquired using different types of imaging modalities over time. Images generated by different types of imaging modalities can have very different appearances. Therefore, it is difficult to accurately identify temporal changes in the region of interest from such images. This hinders the evaluation of the progression of the subject's medical condition.

[0003] As an example, a clinical examination can be performed by generating projection X-ray images of the region of interest of a subject at different times. However, due to the availability of the imaging system or the need to perform additional examinations, some of the images generated during the process of the clinical examination can be generated using a computed tomography (CT) imaging system or a magnetic resonance (MR) imaging system. Such images look very different from X-ray projection images and there is no direct correspondence between their image intensities (luminances) and the image intensities of X-ray projection images. As a result, the evaluation of the progression of the subject's medical condition can be performed using only images generated by a single type of imaging modality even when there is additional time-point data from different types of imaging modalities. This misses an opportunity because the above additional time-point data may provide valuable information for evaluating the progression of the medical condition.

[0004] In addition to the difficulties caused by the fact that images are acquired using different types of imaging modalities during the course of a clinical examination, there can also be differences in the manner in which these images are acquired. For example, there can be differences in the posture of the subject, differences in the viewing angle of the medical imaging system, or differences in the amount of ionizing radiation used to acquire the image. Such factors can also exist even when the images are acquired from a single type of imaging modality. These factors exacerbate the difficulty of identifying temporal changes in the region of interest and can even lead to an incorrect diagnosis of the subject's condition. Some of these factors can be resolved by applying a facility-wide image diagnosis protocol. For example, some clinical settings allow for the positioning of the subject in an optimal and standardized manner with respect to the imaging system. A protocol can also be set up in which a common type of imaging system uses a standardized radiation dose. The application of a facility-wide imaging protocol can reduce the variability between images resulting from differences in the way the images are acquired. However, the use of such protocols limits flexibility. In some clinical environments, it may be unrealistic or even impossible to position a subject with restricted movement in an optimal and standardized manner with respect to the imaging system. For example, in an intensive care environment, it may be unrealistic or even impossible to acquire an image of the subject in a desired posture, in an inspiration state, or at a desired position with respect to the medical imaging system. As a result, differences in the way images are acquired can also impede the assessment of the progression of the subject's medical condition. In particular, these confounding factors are not constant over time and vary from image to image, making it difficult to assess long-term changes in the subject's condition. As a result, a clinician interpreting such images may intuitively compensate for such differences, which can lead to an incorrect diagnosis. SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0005] Therefore, there is a need for an improvement that facilitates the identification of changes in a time series of medical images. MEANS FOR SOLVING THE PROBLEMS

[0006] According to one aspect of the present disclosure, a computer-implemented method for compensating for differences in medical images is provided. The method includes: Receiving image data including a time series of medical images, the time series including one or more medical images generated by a first type of imaging method and one or more medical images generated by a second type of imaging method, wherein the first type of imaging method is different from the second type of imaging method; Generating a normalized time series of the medical images, or one or more normalized measurements representing regions of interest in the time series of the medical images, from the time series of the medical images; and Outputting, respectively, the normalized time series of the medical images and / or one or more measurements derived from the normalized time series of the medical images, or one or more normalized measurements; having.

[0007] In the above method, the normalized time series of the medical images, the one or more measurements derived from the normalized time series of the medical images, and the one or more normalized measurements each compensate for the difference in the type of imaging method used to acquire the images in the time series. By performing such compensation, a more reliable comparison between images can be made, thereby enabling a more reliable evaluation of the long-term changes in the subject's medical condition.

[0008] Further aspects, features, and advantages of the present disclosure will become apparent from the following description of examples made with reference to the accompanying drawings.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

[0010] Examples of the present disclosure are provided with reference to the following description and figures. In this description, for purposes of explanation, numerous specific details of particular examples are set forth. References to "example," "implementation," or similar terms in this specification mean that a feature, structure, or characteristic described in connection with the example is included in at least one example. It is also understood that features described in connection with one example may be used in other examples, and not all features necessarily overlap in each example for the sake of brevity. For example, features described in connection with a computer-implemented method may also be implemented in a corresponding manner in a computer program product and a system.

[0011] In the following description, a computer-implemented method involving the processing of image data related to a time series of medical images is referred to. In some examples, medical images representing a subject's lungs are referred to. However, the lungs are merely examples, and it should be understood that the methods disclosed herein can alternatively be used for medical images representing any part of an anatomical structure.

[0012] Note that the computer-implemented methods disclosed herein can also be provided as a non-transitory computer-readable storage medium storing computer-readable instructions that cause at least one processor to execute the method when executed by the at least one processor. In other words, the computer-implemented method can be implemented in a computer program product. The computer program product can be provided by dedicated hardware or by hardware capable of executing software in conjunction with appropriate software. When provided by a processor, the features of the method can be provided by a single dedicated processor, a single shared processor, or multiple individual processors that can be shared. One or more functions of the features of the method can be provided by a processor shared within a networked processing architecture such as, for example, a client / server architecture, a peer-to-peer architecture, the Internet, or the cloud.

[0013] The explicit use of the terms "processor" or "controller" should not be construed to refer only to hardware capable of executing software, but implicitly includes, but is not limited to, digital signal processor "DSP" hardware, read-only memory "ROM" for storing software, random access memory "RAM", non-volatile storage devices, etc. Further, examples of the present disclosure may take the form of a computer program product accessible from a computer-usable storage medium or a computer-readable storage medium, the computer program product providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable storage medium or a computer-readable storage medium can be any device that can store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or propagation medium. Examples of computer-readable media include semiconductor memory or solid-state memory, magnetic tape, removable computer disk, random access memory "RAM", read-only memory "ROM", rigid magnetic disk, and optical disk, etc. Current examples of optical disks include compact disk read-only memory "CD-ROM", compact disk read-write "CD-R / W", Blu-Ray (trademark), and DVD. As described above, there is a need for improvements that facilitate the identification of changes in the time series of medical images.

[0014] FIG. 1 is a flowchart showing an example of a method for compensating for differences in medical images according to some aspects of the present invention. FIG. 2 is a schematic diagram showing an example of a system 200 for compensating for differences in medical images according to some aspects of the present invention. The operations described in connection with the method shown in FIG. 1 can also be performed by the system 200 shown in FIG. 2, and vice versa. Referring to FIG. 1, the computer-implemented method for compensating for differences in medical images is as follows: Time series 110 of medical images1…i Step S110 of receiving image data including 1…i , wherein the time series includes one or more medical images generated by a first type of imaging method 120 and one or more medical images generated by second type of imaging methods 130, 130', and the first type of imaging method is different from the second type of imaging methods; Time series 110 of medical images 1…i From which a normalized time series 140 of medical images 1…i Or the time series 110 of medical images 1…i One or more normalized measurement values 150 representing regions of interest in 1…j Are generated in steps S120a, S120b; and Normalized time series 140 of medical images 1…i And / or the normalized time series 140 of medical images 1…i One or more measurement values derived therefrom, or one or more normalized measurement values 150 1…j Are each output in steps S130a, S130a', S130b; Are included.

[0015] In the above method, the normalized time series of medical images, one or more measurement values derived from the normalized time series of medical images, and one or more normalized measurement values each provide compensation for differences in the type of imaging method used to acquire the time series of images. By performing such compensation, a more reliable comparison between images can be made, thereby enabling a more reliable assessment of the long - term changes in the subject's medical condition.

[0016] FIG. 3 is a schematic diagram showing a first example of a method for compensating for differences in medical images according to some aspects of the present invention. Referring to FIGS. 1 and 3, image data is received in process S110. The image data includes a time series 110 of medical images 1…i Including. The received medical images 110 1…iforms a time series in the sense that each image is acquired at a different point in time. These images can be acquired periodically, i.e., at regular intervals, or intermittently, i.e., at irregular intervals. The time series of the medical images can be acquired over a period such as minutes, hours, days, weeks, or even longer periods. The time series 110 of medical images 1…i can represent a so-called long-term investigation of a subject. Generally, the medical image 110 1…i can represent any anatomical region. For example, the medical image of the region of interest can be the lungs, heart, liver, kidneys, etc. As an example, as shown in FIG. 3, the time series 110 of medical images received in process S110 1…i can represent the chest of a subject. The time series of the medical images can represent, for example, daily images of the chest of a subject acquired as part of a long-term investigation of the subject to evaluate the progression of Covid-19 in the lungs of the subject.

[0017] The time series 110 of medical images received in process S110 1…i includes one or more medical images generated by a first type of imaging method 120 and one or more medical images generated by second types of imaging methods 130, 130'. The first type of imaging method is different from the second type of imaging method. Generally, the first type of imaging method and the second type of imaging method can be any type of imaging method. As some examples, the first type of imaging method 120 and the second types of imaging methods 130, 130' can be selected from the group of computed tomography, magnetic resonance, ultrasound, and projection X-ray. Computed tomography, magnetic resonance, and ultrasound imaging methods can be referred to as volume imaging methods because they can acquire image data for reconstructing volume images, i.e., 3D images. The projection X-ray "PXR" imaging method can be referred to as a projection imaging method because of its ability to acquire image data for reconstructing projection images, i.e., 2D images. 2D medical image 110 1…iExamples of current projection X-ray imaging systems that can generate include the MobileDiagnost M50, a mobile X-ray imaging system, the DigitalDiagnost C90, a ceiling-mounted X-ray imaging system, and the Azurion 7, which includes a source-detector device attached to a C-arm, all of which are sold by Philips Healthcare, the best in the Netherlands.

[0018] As an example, the medical image 110 received in process S110 1…i may include a projection X-ray image generated by a projection X-ray imaging system labeled "PXR" in FIG. 3 and a volume image generated by a CT imaging system labeled "CT" in FIG. 3. The projection X-ray image shown in FIG. 3 may be, for example, generated by the projection X-ray imaging system 220 shown in FIG. 2, which is a so-called mobile X-ray imaging system often used in intensive care units to acquire images of subjects with restricted movement. The projection X-ray imaging system 220 shown in FIG. 2 includes an X-ray source 230 and a wirelessly connected X-ray detector 240. The versatility of this type of X-ray imaging system facilitates imaging of subjects in complex environments where the subject's movement is restricted, such as the bed-restrained state shown in FIG. 2. As shown in the example of FIG. 3, a part of the medical image 110 in the time series 1…i is generated using the projection X-ray imaging system PXR. These images are generated daily or, in the illustrated example, every few days. However, as shown in the example of FIG. 3, a part of the medical images in the time series 110 1…i is generated by a CT imaging method, that is, a different type of imaging method. The CT imaging method may be used to perform additional clinical examinations on the subject or may be used instead of the projection X-ray imaging system considering the availability of the imaging system. As outlined earlier, it can be difficult to evaluate the subject's condition from images acquired by different imaging methods. This is because CT images look very different from X-ray projection images and there is no direct correspondence between their image intensities (luminances) and those of X-ray projection images.

[0019] Returning to the method shown in FIG. 1, generally, the time series 110 of medical images received in process S110 1…i can be received via any form of data communication including, for example, wired, optical, and wireless communication. As some examples, when wired or optical communication is used, the communication is performed via signals transmitted on an electrical cable or an optical cable, while when wireless communication is used, the communication can be, for example, via RF or optical signals. The time series 110 of medical images 1…i can be received by one or more processors such as the one or more processors 210 shown in FIG. 2. The one or more processors 210 1…i can receive the medical image 110 from a medical imaging system such as the projection X-ray imaging system 220 shown in FIG. 2, or from another medical imaging system or other sources such as, for example, a computer-readable storage medium, the Internet, the cloud, etc.

[0020] Continuing to refer to FIG. 1, in process S120a, a normalized time series 140 of medical images 1…i is generated from the above time series 110 of medical images 1…i The subsequent processes S130a and S130a' shown in FIG. 1 can be executed alternatively or in combination. In process S130a, the normalized time series 140 of medical images 1…i is output.

[0021] The process S130a of outputting the time series 140 of medical images 1…i can be executed in various ways including processes of displaying, printing, and storing the normalized time series 140 of the medical images 1…i The output image can be displayed on a display. For example, the output image can be displayed on the display of a tablet, a laptop, or a mobile phone. As another example, the output image can be displayed on a monitor such as the monitor 250 shown in FIG. 2. As some examples, the normalized time series 140 of medical images 1…i can be displayed side by side, as a time series, or as an overlay image. In other examples, the normalized time series 140 of medical images1…i may output images from the first type of imaging modality together with a normalized image generated from images generated by the second type of imaging modality and display the images generated by the second type of imaging modality by overlaying the images generated by the second type of imaging modality on the normalized image generated from images generated by the second type of imaging modality. The images generated by the second type of imaging modality are also registered to the normalized image generated from images generated by the second type of imaging modality. This may provide a user with clinical findings from the second type of imaging modality that may not be apparent from the images from the first type of imaging modality to assist the user in analyzing the subject's medical condition.

[0022] In step S130a′, a normalized time series 140 of medical images is 1…i In this manner, a normalized time series of medical images 140 is output. 1…i , or in addition to outputting these images, one or more measurements may be converted into a normalized time series of medical images 140 1…i may be derived and output. In general, measurements may be output for one or more images in the time series. For example, a single measurement may be derived and output for each image in the time series, or a single measurement may be derived and output for multiple images in the time series. In the latter case, a single measurement may represent the change over time between multiple images. Alternatively, a measurement may be output for each of multiple images.

[0023] In this regard, various measurements can be derived from the images and output. In general, the measurements may represent image intensity or diagnostic measures. These measurements can be, for example, generated from a normalized time series 140' of medical images. 1…iIt may represent the intensity, volume, or shape of the region of interest. An example of a diagnostic measure is the lung perfusion index. Examples of the lung perfusion index include the volume of pneumothorax (i.e., the volume of the "collapsed lung"), the amount of pleural effusion (i.e., the amount of "fluid around the lung"), the amount of pulmonary edema (i.e., the amount of "fluid in the lung"), or the stage of pneumonia. Continuing with the example of the lung, the measured value of the lung perfusion index can be, for example, the normalized time series 140 of medical images 1…i draw the contour of the lung in 1…i , assign the pixels in the lung to air or water based on their intensity values (luminance values), and for each of the normalized time series 140 of medical images 1…i determine the volume of each of the air regions and water regions by estimating them. Then, for each of the plurality of images in the time series, a value of the lung perfusion index can be output. As another example, a single measurement value can be derived from the plurality of images in the time series, for example, by determining individual measurement values for the plurality of images in the time series, representing these individual measurement values as the rate of change of the lung perfusion index, and outputting them. The normalized time series 140 of medical images 1…i The measured value (or measured values) derived from can also be determined by other methods. In one example, the measured value derived from the normalized time series 140 of medical images 1…i is determined by applying statistical analysis to the region of interest. In this example, the method described with reference to FIG. 1 is: a process of performing statistical analysis on the image intensity values within the region of interest in the normalized medical image 140 1…i ; also has a process S130a' of outputting one or more measured values derived from the normalized time series 140 of medical images 1…i is: a process of outputting the result of the statistical analysis; has.

[0024] For the image intensity values within the region of interest in the normalized medical image 140 1…i various statistical analysis methods can be applied. One exemplary implementation of such statistical analysis is the normalized medical image 140 1…iIt is based on local patch-based analysis of pixel values in. In this example, statistics can be generated for patches in each of the normalized medical images 140 1…i For each patch in, this includes, for example, assuming a Gaussian distribution and determining the mean and standard deviation of the pixel values within the patch. For the normalized medical images 140 1…i When comparing corresponding patches of, for example, two images A and B in, the Gaussian statistics of the patch in image A can be used to estimate what the measured intensity values in the patch of image B might be. This likelihood can be thresholded to predict the variation within the patch. The likelihood can also be visualized as an overlay of images A and B to enable the clinician performing the examination to analyze the detected changes in the image over time. This visualization can, as a result, enable the clinician performing the examination to evaluate whether the detected changes are statistically related or, for example, whether these changes are only due to an incomplete normalization procedure.

[0025] In a related example, the process of outputting the results of a statistical analysis regarding the normalized medical images 140 1…i may include the process of displaying the spatial map of the statistical analysis as an overlay on one or more of the normalized medical images 140 1…i Providing the results of the statistical analysis in this manner can facilitate the clinician's efficient analysis of the progression of the subject's medical condition.

[0026] As some other examples, the process of outputting one or more measurement values derived from the normalized time series 140 of medical images may include the process of outputting the magnitude value as an absolute value or a percentage, or the process of graphically outputting the measurement value(s). In one example, the measurement value represents an increase in lung volume between consecutive images in the series in green, while a decrease in lung volume between consecutive images in the series is represented in red, such that it can be graphically represented as an overlay on one or more of the normalized medical images 140 1…i in the time series. 1…i

[0027] ​ Normalized time series 140 of medical images 1…i and / or Normalized time series 140 of medical images 1…i By outputting one or more measurement values derived from Normalized time series 140 of medical images, the method shown in FIG. 1 facilitates a more reliable comparison between regions of interest in the image for the clinician performing the examination. This in turn enables the clinician to more accurately evaluate the long-term changes in the subject's condition.

[0028] Generally, Normalized time series 140 of medical images 1…i may represent images generated by a general type of imaging modality. In a first set of examples, the general type of imaging modality is a projection imaging modality, while one or more of the images in Normalized time series 140 of medical images 1…i are provided by projecting volume medical images. In a second set of examples, the general type of imaging modality is a projection imaging modality, while one or more of the images in Normalized time series 140 of medical images 1…i are provided by inputting medical images into neural network NN1. The general type of imaging modality may typically be a first type of imaging modality, a second type of imaging modality, or a type of imaging modality different from the first and second types of imaging modalities.

[0029] Referring to the method described above in connection with FIG. 1, in a first set of examples, the generating process includes step S120a of generating Normalized time series 140 of medical images 1…i from time series 110 of medical images 1…i to Normalized time series 140 of medical images 1…i The outputting process includes steps S130a, S130a' of outputting one or more measurement values derived from Normalized time series 140 of medical images 1…i and / or Normalized time series 140 of medical images 1…i Normalized time series 140 of medical images represents images generated by a general type of imaging modality.

[0030] Thus, in the first example, Normalized time series 140 of medical images1…i can be output as shown in FIG. 3.

[0031] Continuing with the first example, in one example, the first type of imaging modality 120 is a projection imaging modality, the second types of imaging modalities 130, 130' are volume imaging modalities, and the general type of imaging modality is a projection imaging modality. In this example, the normalized time series 140 of medical images 1…i is the normalized time series of the medical images, the time series 110 generated by the first type of imaging modality 120 1…i from one or more medical images, and one or more medical images generated by the second types of imaging modalities 130, 130' are provided as a combination with one or more projected images provided by projecting the one or more projected images so as to correspond to the one or more medical images generated by the first type of imaging modality 120. Thus, the normalized time series 140 of medical images 1…i includes a projected image generated by the first type of imaging modality and a projected image generated by projecting a volume image generated by the second type of imaging modality.

[0032] In this example, the normalized time series 140 of medical images 1…i is thus a projected image, and the volume images generated by the second types of imaging modalities 130, 130' are projected to provide simulated or "virtual" projected images corresponding to one or more projected images generated by the first type of imaging modality. This example is shown in FIG. 3, in which some of the images within the normalized time series 140 of medical images at the bottom of the figure 1…i are provided by projected images generated by a projection X-ray imaging system labeled "PXR", while other "virtual" projected images in the normalized time series are provided by projecting volume images generated by a CT imaging system labeled "CT".

[0033] Generally, the normalized time series 140 of medical images 1…iIn this case, the volume image can be replaced or augmented by a projection image generated from the volume image at the corresponding time point. The process of generating a simulated or "virtual" projection image from the volume image is shown as process SIM in FIG. 3. In doing so, the normalized time series 140 of the output medical images 1…i , or one or more measurement values derived from the normalized time series 140 of the output medical images 1…i can be used to identify the temporal changes in the region of interest and, as a result, to evaluate the progression of the subject's medical condition.

[0034] In one example, the process of projecting one or more medical images generated by the second type of imaging modality 130, 130' so that these one or more projected images correspond to one or more medical images generated by the first type of imaging modality 120 is to project one or more medical images generated by the second type of imaging modality onto a virtual detector 240 v using a virtual source 230 v . This example will be described with reference to FIGS. 3 and 4. In other examples, as will be described later, a neural network NN1 is used to generate the projection image.

[0035] Referring to FIG. 3, in this example, the first imaging modality 120 is a projection modality. The first imaging modality 120 is, for example, a projection X-ray modality labeled "PXR" in FIG. 3, and generates projection X-ray images of the time series 1101…i at times corresponding to day 0, day 2, day 7, and day 12. In this example, the second imaging modality is a volume imaging modality. The second imaging modality is, for example, a CT imaging modality labeled "CT" in FIG. 3, and generates volume CT images at times corresponding to day 5, day 10, and day 14. FIG. 4 shows a volume medical image 1103 using a virtual source 230 v onto a virtual detector 240 vIt is a schematic diagram showing an example of the process of projecting onto. Referring to FIG. 4, an exemplary volume image generated on the 3rd day, that is, a CT image 1103 representing the chest, is a virtual detector 240 to generate a normalized medical image 1403 v is projected onto. The normalized medical image 1403 is a projection image corresponding to a projection X-ray image generated by a first imaging method 120, that is, a projection X-ray imaging method. The normalized medical image 1403 is then, at the time corresponding to the time when the volume image was generated, that is, on the 5th day, in a normalized time series 140 of medical images 1…i is included in.

[0036] The process of projecting the volume image 1103 onto the virtual detector can be mathematically executed using a ray tracing process. Referring to FIG. 4, in this process, the density values represented by the volume image 1103 pass through the volume image 1103 from a virtual X-ray source 230 v and are integrated along the path of virtual radiation such as thick dotted lines projected onto a virtual X-ray detector 240 v to provide the integrated density values at each position on the virtual X-ray detector 240 v The integrated density values can also be adjusted to compensate for X-ray scattering. The projection image 1403 is generated by performing this ray tracing process and calculating the integrated density values at a plurality of positions across the virtual X-ray detector 240 v These processes are performed mathematically, that is, using a virtual source 230 v and a virtual detector 240 v is used to execute.

[0037] The process of projecting the above-described volume image 1103 onto the virtual detector can be performed based on the known relative positions among the virtual source, the virtual detector, and the subject represented by one or more medical images generated by the second type of imaging method 120. Thereby, the projected image comes to correspond to one or more medical images generated by the first type of imaging method 120. The relative position between the virtual source and the virtual detector can be known from the relative position between the actual source and the actual detector of the first imaging method with respect to the subject at the time of generating the projection image by the first type of imaging method. These relative positions can be obtained by performing imaging using a standard clinical imaging protocol. The protocol can define parameters such as the distance between the actual X-ray source and the actual detector, the position of the subject with respect to the detector, and the image resolution. For example, when a so-called "PA" projection image is generated by the first imaging method with the subject standing upright and facing an upright X-ray detector at a specified distance between the subject and each of the X-ray source and the X-ray detector, the same relative position will be used to generate the projection image using the virtual source, the virtual detector, and the subject within the volume image generated by the second type of imaging method. As another example, the relative position can also be measured at the time of generating the projection image by the first imaging method. For example, the relative position can be measured using the actual X-ray source, the actual X-ray detector, and a (depth) camera that captures the object at the time of generating the projection image.

[0038] Alternatively or in addition, the process of projecting the volume image 1103 onto the virtual detector may include adjusting the relative positions among the virtual source, the virtual detector, and one or more medical images generated by the second type of imaging modalities 130, 130' such that the shape of one or more anatomical feature structures in the projected one or more images matches the shape of one or more corresponding anatomical feature structures in one or more medical images generated by the first type of imaging modality 120. In this case, due to the match of the shapes, the projected images will correspond to one or more medical images generated by the first type of imaging modality 120. In this case, an optimization process is executed, in which the relative positions among the virtual source, the virtual detector, and one or more medical images generated by the second type of imaging modalities 130, 130' are repeatedly adjusted until the difference between the shape of one or more anatomical feature structures in the projected image and the shape of one or more corresponding anatomical feature structures in one or more medical images generated by the first type of imaging modality 120 is less than a predetermined threshold. The starting point for such optimization can be an approximate relative position expected for the type of image and the region of interest within the image. An alternative starting point for such optimization can be a relative position expected from a standard imaging protocol or a measured relative position measured using a (depth) camera. In this example, the shape of the anatomical features within the images can be determined by applying a known image segmentation algorithm to each image.

[0039] In these examples where the volume image is projected onto the virtual detector, the process of projecting one or more medical images generated by the second type of imaging modality onto the virtual detector 240 using the virtual source 230 v can also be executed based on the known operating parameters of the projection imaging modality, i.e., the first type of imaging modality. For example, if the first imaging modality is a projection X-ray imaging modality, the process of projection may further include the kV of the X-ray source v p ​It can be executed based on one or more operating parameters such as energy, X-ray dose, exposure time, sensitivity of the X-ray detector, etc. By additionally using such parameters, an improved correspondence relationship between the projected image 1403 and one or more medical images generated by the first type of imaging method 120 can be achieved.

[0040] As described above, in addition to the problem that time-series images are acquired using different types of imaging methods during the clinical examination process, the time series 110 shown in FIG. 3 1…i There can also be differences in the manner in which the medical images within are acquired. For example, there can be differences in the posture of the subject, differences in the field of view angle of the medical imaging system, or differences in the dose of ionizing radiation used to acquire the images. Such factors can also exist even when the images are acquired from a single type of imaging method. These factors exacerbate the difficulty of identifying temporal changes in the region of interest and can even lead to an incorrect diagnosis of the subject's condition.

[0041] To compensate for the effects of these differences, thereby further improving the identification of temporal changes in the region of interest, and as a result, improving the assessment of the progression of the subject's medical condition, one or more additional processes can be performed on the normalized time series 140 of medical images 1…i Therefor.

[0042] Referring to the method shown in FIG. 1, in one example, the process S120a of generating the normalized time series 140 of medical images 1…i includes a process of warping (distorting) one or more of the images in the normalized time series 140 of medical images 1…i so that the shape of one or more anatomical feature structures 160 in the warped one or more images 140' 1,2 matches the shape of one or more anatomical feature structures 160 in the reference image 170.

[0043] In this example, the effect of warping is to provide images in which the anatomical feature structures have similar shapes. Thereby, the normalized time series 140 of medical images 1…iIt becomes possible to perform a more accurate comparison between images. In this example, the reference image 170 can be provided by an image from the time series 110 of the received medical images, an image from the normalized time series 140 of the medical images, or an atlas image. As some examples, FIG. 5 shows an example of an atlas image 170 of a lung field according to some aspects of the present disclosure, while FIG. 6 shows an example of an atlas image 170 of a thorax according to some aspects of the present disclosure. The warping process in this example includes, for example, warping one or more of the images in the normalized time series 140 of the medical images so that the shape of the lung or the shape of the thorax in the one or more warped images matches the shape of the lung or the shape of the thorax in the atlas images shown in FIGS. 5 and 6. 1…i from, an image from the normalized time series 140 1…i of the medical images, or can be provided by an atlas image. As some examples, FIG. 5 shows an example of an atlas image 170 L of a lung field according to some aspects of the present disclosure, while FIG. 6 shows an example of an atlas image 170 R of a thorax according to some aspects of the present disclosure. The warping process in this example includes, for example, warping one or more of the images in the normalized time series 140 1…i of the medical images so that the shape of the lung or the shape of the thorax in the one or more warped images matches the shape of the lung or the shape of the thorax in the atlas images shown in FIGS. 5 and 6.

[0044] The atlas images 170 shown in FIGS. 5 and 6 L,R show, by their intensity values, the probability of the structures belonging to the lung boundary and the ribs, respectively. The atlas image 170 L,R can be received, for example, by one or more processors 210 shown in FIG. 2. The atlas image can be received from a database of atlas images. The atlas image can be selected based on the similarity between the subject imaged in the time series 110 of the medical images from the database and the reference object represented in the atlas image 170. For example, the atlas image can be selected based on the similarity between the age, gender, and size of the subject and the reference subject. The atlas image 170 1…i in the time series of medical images, and the reference object represented in the atlas image 170 L,R . For example, the atlas image can be selected based on the similarity between the age, gender, and size of the subject and the reference subject. The atlas image 170 L,Rprovides an anatomical feature structure 160, i.e., a reference shape of the lung or rib. The atlas image may represent a preferred perspective of the anatomical feature structure in a reference subject. The atlas image may be acquired while the reference subject maintains a desired posture. As an example, when the anatomical feature structure is the lung, the atlas image of the lung may be a so-called anteroposterior "PA" chest view representing the lung, the bony thorax, the mediastinum, and the great vessels. Such an atlas image can be acquired with the subject standing upright, facing an upright X-ray detector, in a specified inspiration state, with the upper surface of the X-ray detector at a specified distance above the shoulder joint, the jaw lifted so as to be outside the image field, the shoulders rotated forward so that the scapulae are laterally displaced from the lung field, and under specified operating settings (e.g., X-ray kVp energy and exposure time) of the X-ray imaging system.

[0045] In the warping process, the normalized time series 140 of medical images 1…i is warped to the atlas image 170 L,R . The warping process can be performed using various known transformations. An example of a suitable transformation is an affine transformation. Other examples of suitable transformations include B-splines, thin plate splines, and radial basis functions. The warping process can be performed based on the mapping between a plurality of corresponding landmarks 180 i…j represented in both the warped image and the reference image 170. The warping process can be performed using a neural network. When the warping is performed by a neural network, the neural network may or may not explicitly use such landmarks. The landmarks 180 1…j can be provided by the anatomical feature structure or by reference markers. As some examples, anatomical feature structures such as bones such as ribs, scapulae, etc., or organ contours such as organs such as the lung contour, diaphragm, heart shadow, etc. can function as anatomical landmarks. Such feature structures are determined by X-ray attenuation in the normalized time series 140 of medical images 1…iIt is distinguishable. The reference markers formed from X-ray attenuation substances are also distinguishable in medical images and can also function as landmarks. The reference landmarks can be located at known reference positions on the surface or within the body. In the latter case, the reference markers can be, for example, those implanted for use as a surgical guide. Implantable devices such as pacemakers can also function as reference markers. The reference markers may also be provided by an intervention device inserted into the body.

[0046] The normalized time series 140 of medical images 1…i of the landmarks 180 i…j and the corresponding landmarks in the reference image 170 can be identified using various techniques. In one example, a feature structure detector is used to identify the anatomical landmarks in the normalized time series 140 1…i of medical images. The identified landmarks are then mapped to the corresponding labeled anatomical landmarks in the reference image 170. In this example, the reference image 170 includes a plurality of labeled anatomical landmarks, and the method described with reference to FIG. 1 is as follows: Applying a feature structure detector to each medical image in the normalized time series 140 1…i of medical images to identify a plurality of landmarks in the medical image corresponding to the labeled anatomical landmarks; also includes, and the mapping is determined using the identified corresponding landmarks.

[0047] In this example, the process of applying a feature structure detector to medical images within a time series can be performed, for example, using an edge detector, model-based segmentation, or a neural network. As an illustration, FIG. 7 shows an example of the result of applying a feature structure detector to a medical image 1401 according to some aspects of the present disclosure to identify a plurality of landmarks within the medical image. The medical image 1401 shown in FIG. 7 represents the chest and includes bone regions such as ribs and vertebrae, as well as the shadow of the heart and the outline of the lungs. In this example, the anatomical region used for the warping process is the lungs, and the feature structure detector identifies contours representing the right lung, the right side of the shadow of the heart, the right side of the diaphragm, the left lung, the left side of the shadow of the heart, and the left side of the diaphragm within the medical image 1401 as landmarks 180 1…6 respectively. These landmarks correspond to the landmarks in an atlas image 170 L such as the atlas image related to the lung fields shown in FIG. 5. The landmarks 180 1…6 identified in the medical image 1401 are then mapped to the corresponding landmarks in the atlas image 170 L in order to warp the medical image 1401 to the atlas image in the warping process.

[0048] The result of the warping process is a warped medical image 140' in which the shape of one or more anatomical feature structures 160 corresponds to the shape of one or more anatomical feature structures 160 in the reference image 170 1…i . This correspondence can be measured, for example, by comparing the shape of the anatomical feature structures in the image using the measured distances between the corresponding landmarks in the warped medical image 140' 1…i and the reference image 170. As another example, this correspondence is the warped medical image 140' 1…iIt can be measured by calculating the value of the Dice coefficient between the segmentation mask in the image and the segmentation mask in the reference image 170. When the calculated value of such a measurement value is within a predetermined range, these shapes can be regarded as corresponding to each other. The effect of the warping process is shown in FIG. 8, which shows an example of warping two 2D medical images 1401 and 1402 in a time series according to some aspects of the present disclosure to the reference image. The two 2D medical images 1401 and 1402 shown on the left side of FIG. 8 are projection X-ray images generated by the same type of imaging method, that is, a projection X-ray "PXR" imaging system. Images 1401 and 1402 represent the same anatomical region of the subject, that is, the lungs, but these images are acquired with the subject in different postures. The difference in the subject's posture confuses the identification of the temporal changes in the lung condition between images 1401 and 1402. The warped images 140'1 and 140'2 on the right side of FIG. 8 are warped so that the lungs of the subject correspond to the lung fields of the atlas image, as described above. As shown in the warped images 140'1 and 140'2, the warping process has the effect of compensating for the difference in the subject's posture when images 1401 and 1402 are acquired. Thus, the warping process enables a more reliable identification of the temporal changes in the subject's lungs and, as a result, a more reliable assessment of the progression of the subject's medical condition.

[0049] As described above, in order to compensate for the differences in the manner in which medical images are acquired, the normalized time series 140 of medical images 1…iOther processes can also be executed for this. These processes can be executed instead of the warping process or in addition to the warping process. The inventor noticed that the evaluation of the temporal changes in medical images is not only hindered by the differences in the shapes of anatomical regions between images, but also that the evaluation of such changes can be hindered by differences in the intensity scale (scale) of the images. More specifically, differences in intensity scale resulting from differences in the manner in which medical images are acquired over time can prevent their comparison. As an example, the intensity at any point in a 2D X-ray image depends on the values of the X-ray energy "kVp", exposure time, and the sensitivity of the X-ray detector used to acquire the image. Referring to FIG. 3, since the images in time series 110 1…i are acquired over a certain period, the images in the time series can be generated by a projection X-ray imaging system "PXR" with different settings, or these images can actually be generated by different X-ray imaging systems. In contrast to computed tomography images in which a Hounsfield unit value can be assigned to each image voxel, there is no corresponding absolute attenuation scale for pixel intensity values in 2D projection X-ray images.

[0050] In one example, the process S120a for generating a normalized time series 140 of medical images: 1…i has: a process of adjusting the intensity of the images in the normalized time series 140 of medical images based on the intensity at one or more positions in the reference image 170; and / or 1…i a process of adjusting the intensity of the images in the normalized time series 140 of medical images using an image style transfer algorithm; a process of adjusting the intensity of the images in the normalized time series 140 of medical images using an image style transfer algorithm; 1…i a process of adjusting the intensity of the images in the normalized time series 140 of medical images using an image style transfer algorithm; has.

[0051] These adjustment processes compensate for the variations between images resulting from differences in the manner in which the medical images are acquired.

[0052] This example will be described with reference to FIG. 9, which shows an example of warping two 2D medical images 1401, 1402 within a time series to a reference image, and adjusting the intensity of the medical images, according to some aspects of the present disclosure. In this example, the warping process performed on the 2D projection X-ray "PXR" medical images 1401 and 1402 to provide the warped images 140'1 and 140'2 is performed as described above.

[0053] In this example, the positions can be defined manually or automatically in the reference image 170. As an example, for the normalized time series 140 of the medical images in FIG. 9 1…i if representing the lungs, the positions within the reference image 170 shown in FIG. 5 can be defined within the lung fields or within the mediastinum. These positions can be predefined within the reference image 170, or can also be defined based on user input supplied via a user input device in combination with the displayed image. Then, techniques such as windowing processing and histogram normalization are used to adjust the intensity values of the normalized time series 140'1 and 140'2 of the medical images in the center of FIG. 9 to provide the intensity-adjusted images 140''1 and 140''2 on the right side of FIG. 9.

[0054] Alternatively or additionally, an image style transfer algorithm can be used to provide the intensity-adjusted images 140''1 and 140''2. For this purpose, various image style transfer conversions are known. These can be, for example, based on Laplacian pyramids, or (convolutional) neural networks. An example of an image style transfer algorithm is disclosed in the document WO2013 / 042018A1. This document discloses a technique for converting a slave image that includes steps of generating a color or grayscale conversion based on a master image and the slave image. This conversion is used to optically adapt the slave image to the master image. Using this technique, the intensity of the medical images 140 1…i in the time series is adjusted to the normalized time series 140' of the medical images 1…iThe adjusted intensity values in can be adjusted so as to have a similar appearance or style. The techniques disclosed in this document are applied to the time-series medical images 140 1…i by using as a slave image and using one of the reference images such as the atlas images shown in FIGS. 5 and 6, or the time-series medical images 110 1…i as a master image.

[0055] In a related example, the process of adjusting the intensity of the images in the normalized time series 140 of medical images 1…i based on the intensity at one or more positions within the reference image 170 is: calculating an average intensity value within a portion of the reference image 170; and normalizing the intensity values within each medical image in the normalized time series 140 of medical images 1…i using the above average intensity value; may include.

[0056] In this example, the portion of the reference image 170 used to normalize the intensity values of each medical image 140 1…i may be selected as the portion of the reference image where the image intensity is expected to be temporally invariant over the duration of the time series 110 of medical images 1…i For example, if the region of interest is the lung, the selected portion of the reference image may correspond to a portion of the spine. This is because the attenuation of the spine is not expected to change over time. As another example, if the region of interest is the lung, the selected portion may correspond to a part of the lung within the reference image where no disease is expected in the time series 110 of medical images 1…i For example, if the condition of the subject being examined is pulmonary edema, it is expected that temporal changes in image intensity will occur in the lower part of the lung as a result of the accumulation of fluid over time, while the upper part of the lung remains disease-free and thus has a stable image intensity. In this case, the upper part of the lung in the reference image 170 can be used as the portion of the reference image 170 used to normalize the intensity values in the medical image 140 1…i

[0057] ​The lung portion within the reference image 170 can be identified automatically or based on user input received via a user input device in combination with a display device. This portion of the reference image 170 can represent a combination of air within the lung and ribs surrounding the lung. The average intensity of pixel values in this portion of the reference image can then be calculated. Next, this average value is used to normalize the intensity values of the medical images 140 1…i within the time series. Referring to FIG. 5, this involves, for example, a process of identifying a corresponding disease-free portion of the lung towards the upper part of the lung in the lung field of the atlas image, a process of calculating the average value of intensity values in the corresponding disease-free portion, a process of calculating the scaling factor as the ratio of the average intensity value in the disease-free region of the medical image 1401 to the average intensity value in the lung portion within the atlas image, and a process of using the above scaling factor to scale the image intensity values within the medical image 140 1…i to provide adjusted intensity values within the intensity-adjusted medical images 140”1, 140”2. These processes are then repeated for other medical images 140 1…i within the time series. Since the lung portion within the atlas image is used to normalize each medical image 140 1…i within the time series, these medical images are scaled to a common reference intensity, which facilitates a more reliable comparison between image intensities in other parts of the images.

[0058] Additional portions of the reference image 170 can also be used to perform the above-described intensity normalization. For example, a second region of interest can be identified within the mediastinum or bone region of the reference image and can also be used to perform intensity normalization. By using two regions of interest with different attenuation values to normalize the intensity of the medical images 140 1…i within the time series, a more accurate mapping of intensity values and, as a result, a more reliable identification of changes in the regions of interest within the image are facilitated.

[0059] In this example, the normalized medical image 140” 1…iRegarding the adjusted intensity value in [context], the value of the quality metric can also be calculated and output. The value of the quality metric can be calculated, for example, based on the statistics of the image intensity values in a part of the reference image or the corresponding part of the medical image 110 1…i or based on the quality of the mapping between the landmarks used in the warping process. The value of the quality metric can be relatively low when the variance of the image intensity values in a part of the reference image is high, and vice versa. By outputting the value of the quality metric, the clinician performing the examination can reconfirm the accuracy of the method.

[0060] The inventors noticed that after the warping process described above, some regions in the normalized series 140' of the medical image outside the anatomical region used for warping 1…i can be warped into unnatural shapes. This can confuse the examining physician. In other examples, the method described with reference to FIG. 1 may: receiving an input defining a region of interest in the received time series 110 of medical images; and 1…i in the normalized time series 140 of medical images suppressing one or more image feature structures outside or within the region of interest; 1…i may also be included.

[0061] ​By suppressing one or more image feature structures according to this example, it is avoided that information that causes confusion to the examining clinician is presented. This example will be described with reference to FIG. 10, which shows an example of suppressing image feature structures outside the region of interest of the lung in the medical image 1401 according to some aspects of the present disclosure. FIG. 10a shows the image 1101 from the received time series of medical images. FIG. 10b shows the region of interest in the image 1101 by a dotted line. The region of interest can be automatically defined within the image 1101. For example, a segmentation algorithm or a feature structure detector can be used to automatically define the region of interest of the lung. Alternatively, the region of interest can also be manually defined based on user input received via a user input device combined with a display device for displaying the image 1101. FIG. 10c shows the contour of the region of interest, which is used to define a binary mask applied to the image 1101 to suppress feature structures outside the region of interest of the lung. FIG. 10d shows the normalized medical image 1401 in which the feature structures outside the region of interest of the lung are suppressed. As can be understood, the confusing feature structures outside the region of interest of the lung are removed in the image shown in FIG. 10d.

[0062] In the example shown in FIG. 10, the image feature structures outside the region of interest of the lung are suppressed by mapping their pixel values to pixel intensity values of the black level. However, these feature structures can also be suppressed by other methods instead. For example, their pixel values can also be mapped to pixel intensity values different from the pixel intensity values of the black level, or their pixel values can be gradually adjusted to a predetermined value to "feather out" the region of interest. The process of suppressing the image feature structures outside the region of interest can also be executed using a reference image such as an atlas image. For example, in this example, the method described with reference to FIG. 1 is: Receiving an input for identifying a region of interest in the reference image 170; Mapping the contour of the region of interest from the reference image 170 to each of the normalized medical images 140 1…i ; and The normalized medical image 140 1…iAdjusting the image intensity values outside the mapped contour within to suppress one or more image feature structures outside the region of interest; may also be included.

[0063] In this example, examples of various regions of interest are predefined in the reference image and can be used to suppress one or more image feature structures outside the region of interest. As another example, the region of interest can be defined by segmenting the reference image. In this example, since contours from the reference image will be used to suppress one or more image feature structures outside the region of interest, reliable contours are used.

[0064] A normalized time series 140 of medical images to compensate for differences in the manner in which the medical images are acquired 1…i Another process that can be performed on the is the removal of foreign objects within the image. Foreign objects such as implanted devices can be identified within the image based on X-ray attenuation values. For this purpose, a feature structure detector or a neural network can be used. The image intensity values of the foreign objects are set to a default level such as the black level or the white level to prevent their presence from distracting the clinician examining them.

[0065] As a further example, FIG. 11 shows an example of warping of two normalized medical images 1401, 1402 with respect to a reference image, adjustment of image intensity values within the image, and suppression of image feature structures outside the region of interest of the lungs, according to some aspects of the present disclosure. FIG. 11 shows the effect of a combined process of warping, adjustment of image intensity values, and suppression of feature structures outside the region of interest, and further shows a reduction in the amount of variation resulting from differences in the manner in which medical images are acquired during a long-term examination of a subject, which can be achieved by the methods described herein.

[0066] As described above, instead of providing one or more images of a normalized time series 140 of medical images by projecting volumetric medical images 1…i in a second set of examples 1…iOne or more of the images are provided by inputting medical images into a neural network NN1. A second set of examples will be described with reference to the methods shown in FIGS. 1 and 3.

[0067] In this second set of examples, the first type of imaging modality 120 is a projection imaging modality, and the second types of imaging modalities 130, 130' are volume imaging modalities. A common type of imaging modality is a projection imaging modality. The normalized time series 140 of medical images 1…i is generated by supplying the normalized time series of medical images as a combination of one or more medical images and one or more projection images from the time series 110 1…i generated by the first type of imaging modality 120. The one or more projection images are provided by inputting one or more medical images generated by the second types of imaging modalities 130, 130' into a neural network NN1. The neural network NN1 is trained to generate a projection image corresponding to the first type of imaging modality 120 for each of the input images.

[0068] The neural network NN1 can be provided by various architectures including, for example, a convolutional neural network CNN, Cycle GAN, or a transformer architecture. Generally, the training of a neural network includes the step of inputting a training dataset into the neural network and the step of repeatedly adjusting the parameters of the neural network until the trained neural network provides an accurate output. Training is often performed using a dedicated neural processor such as a graphics processing unit "GPU", a neural processing unit "NPU", or a tensor processing unit "TPU". Training often adopts a centralized approach of training the neural network using a cloud-based or mainframe-based neural processor. After training using the training dataset, the trained neural network can be deployed to a device for analyzing new input data during inference. Since the processing requirements during inference are significantly less than those required during training, the neural network can be deployed to various systems such as laptop computers, tablets, mobile phones, etc. Inference can be performed, for example, by a central processing unit "CPU", GPU, NPU, TPU, on a server, or within the cloud.

[0069] Therefore, the training process of the neural network NN1 shown in FIG. 3 includes the step of adjusting its parameters. The parameters, more specifically the weights and biases, control the operation of the activation function within the neural network. In supervised learning, the training process automatically adjusts the weights and biases so that when input data is presented, the neural network accurately provides the corresponding expected output data. To do this, the value of the loss function, that is, the error, is calculated based on the difference between the predicted output data and the expected output data. The value of the loss function can be calculated using functions such as negative log-likelihood loss, mean squared error, Huber loss, cross-entropy loss, etc. During training, the value of the loss function is usually minimized, and the training is terminated when the value of the loss function meets the stopping criterion. Sometimes, the training is terminated when the value of the loss function meets one or more of multiple criteria.

[0070] To solve the loss minimization problem, various methods such as gradient descent and quasi-Newton method are known. Various algorithms have been developed to implement these methods and their variants including, but not limited to, Stochastic Gradient Descent "SGD", Batch Gradient Descent, Mini-batch Gradient Descent, Gauss-Newton method, Levenberg-Marquardt method, Momentum, Adam, Nadam, Adagrad, Adadelta, RMSProp and Adamax "Optimizers". These algorithms use the chain rule to calculate the derivative of the loss function with respect to the model parameters. This process is called backpropagation (error backpropagation) because the derivative is calculated starting from the last layer or output layer and moving towards the first layer or input layer. These derivatives inform the algorithm how the model parameters should be adjusted to minimize the error function. That is, the adjustment to the model parameters is made to work in the reverse direction within the network starting from the output layer until reaching the input layer. In the first training iteration, the initial weights and biases are often randomized. Then, the neural network predicts the output data, which is also random. Then, backpropagation is used to adjust the weights and biases. The training process is repeatedly executed by making adjustments to the weights and biases in each iteration. Training ends when the error, i.e., the difference between the predicted output data and the expected output data, is within an acceptable range for the training data or some validation data. Thereafter, the neural network can be deployed, and the trained neural network makes predictions for new input data using the trained values of its parameters. If the training process is successful, the trained neural network accurately predicts the expected output data from the new input data.

[0071] In one example, neural network NN1 is configured to generate a projection image corresponding to the first type of imaging modality 120 for each of the input images: Receive training data including a plurality of volume training images representing regions of interest, wherein these volume training images are generated by a second type of imaging method 130, 130'; Receive ground truth data including corresponding ground truth projection images generated by a first type of imaging method 120 for each of the volume training images; Input the training data into a neural network NN1; and For each of the plurality of input volume training images: Use the neural network NN1 to predict a corresponding projection image; Adjust the parameters of the neural network NN1 based on the difference between the predicted projection image and the ground truth projection image; and Repeat the prediction and adjustment until a stopping criterion is met; Thereby being trained.

[0072] As an example, the first type of imaging method is a projection X-ray imaging method, and the second type of imaging method is a CT imaging method. In this example, the training data may include volume images of the chest generated by the CT imaging method. The training data may include hundreds, thousands, or more such images. The ground truth data includes projection images of the chest generated by the X-ray projection imaging method. The ground truth data is provided by a simulated X-ray projection image generated from the volume training images of the chest generated by the CT imaging method, in which case the volume images are projected to provide the simulated X-ray projection images. The projection can be performed according to the techniques described above. For example, to provide a ground truth simulated X-ray projection image, the volume images can be projected from a predetermined orientation using predetermined values related to the dose of the X-ray source, the exposure time, and the sensitivity of the X-ray detector. The ground truth projection images can alternatively be provided by a projection imaging system, in this example a projection X-ray imaging system. The adjustment of the parameters of the neural network NN1 can be performed according to the backpropagation technique described above. As a result, the trained neural network NN1 is trained to output the corresponding projection image when a new CT image is provided. The neural network NN1 can be similarly trained to predict an X-ray projection image from an image generated by an imaging method different from the CT imaging method. For example, the first imaging method can be an MRI imaging method as another example, and the neural network NN1 is trained to predict the corresponding X-ray projection image. In this case, the corresponding ground truth data can be provided by a projection X-ray imaging system. In this case, the training of the neural network NN1 is performed in a similar manner.

[0073] Normalized time series 140 of medical images 1…i as a combination of one or more medical images from the time series 110 generated by the first type of imaging method 120 1…i and one or more projection images generated by the neural network NN1, then the medical images 140 described above1…i warping, and similarly, the normalized time series 140 of medical images 1…i perform image intensity adjustment and suppression of one or more image feature structures outside the region of interest in the normalized time series 140 of the medical image 1…i to further reduce the influence due to differences in the manner in which the image is acquired.

[0074] In the example of the third set, it follows the right branch of the method shown in the flowchart of FIG. 1. In the example of the third set, the time series 110 of medical images 1…i one or more normalized measurements 150 representing the region of interest in 1…j are generated, and the one or more normalized measurements 150 1…j are output. Thus, unlike the examples of the first and second sets, there is no process of generating the normalized time series 140 of the position image. Instead, one or more normalized measurements 150 1…i are directly generated from the time series 110 of medical images 1…j 1…i

[0075] Referring to the method described above with reference to FIG. 1, the generating process has step S120b of generating one or more normalized measurements 150 representing the region of interest in the time series 110 of medical images 1…i from the time series 110 of medical images 1…i The outputting process has step S130b of outputting the one or more normalized measurements 150 1…i One or more normalized measurements 150 representing the region of interest 1…i are: 1…i the step of inputting the time series 110 of medical images 1…i into the neural network NN2; generated by, where the neural network NN2 is trained to generate a normalized measurement 150 representing the region of interest for each of the input images 110 1…i 1…i

[0076] ​​​​​An example of the third set will be described with reference to FIG. 12, which is a schematic diagram showing a second example of a method for compensating for differences in medical images according to some aspects of the present disclosure.

[0077] Referring to FIG. 12, the first imaging modality 120 can be, for example, a projection X-ray imaging modality labeled "PXR". The second imaging modality can be a CT imaging modality 130 labeled "CT" or a 3D ultrasound imaging modality 130' labeled "US". Generally, the normalized measurement values predicted by the second neural network NN2 can represent diagnostic metrics such as image intensity or a pulmonary perfusion index. These measurement values can represent, for example, the intensity, volume, or shape of a region of interest in a normalized time series 140' 1…i of a medical image. An example of a diagnostic metric is the pulmonary perfusion index. In the illustrated example, the normalized measurement value 150 1…i is the pulmonary perfusion index. In the example shown in FIG. 12, the normalized measurement value 150 1…i , i.e., the value of the pulmonary perfusion index, is generated by inputting a time series 110 1…i of medical images generated by the projection X-ray imaging modality "PXR", the CT imaging modality "CT", or the 3D ultrasound imaging modality "US" into the neural network NN2.

[0078] The second neural network NN2 can be provided by various architectures including, for example, a convolutional neural network CNN or a transformer architecture. In one example, the neural network NN2 generates a normalized measurement value 150 1…i representing the region of interest for each of the input images by: receiving training data including a plurality of training images representing the region of interest, the training images including a plurality of images generated by the first type of imaging modality 120 and a plurality of images generated by the second type of imaging modality 130, 130'; receiving ground truth data including the corresponding ground truth values of the normalized measurement values representing the region of interest for each of the training images; Input training data into neural network NN2; and For each of the plurality of input training images: Use neural network NN2 to predict the value of a normalized measurement representing the region of interest; Adjust the parameters of neural network NN2 based on the difference between the predicted value of the normalized measurement and the corresponding ground truth value; and Repeat the above prediction and adjustment until a stopping criterion is met; Thereby being trained.

[0079] The second neural network can be trained using the techniques described above to generate a normalized measurement 150 representing the region of interest for each of the input images 110 1…i 1…i

[0080] ​​As an example, the first type of imaging method may be a projection X-ray imaging method, and the second type of imaging method may be a CT imaging method. In this example, the training data may include chest X-ray projection images generated by the X-ray projection imaging method and chest volume images generated by the CT imaging method. The training data may include hundreds, thousands, or more such images. The normalized measurement value predicted by the second neural network NN2 may be, for example, a pulmonary perfusion index. The ground truth data is provided by the evaluation of the images in the training data by an expert, and thus may include the ground truth value of the pulmonary perfusion index. The adjustment of the parameters of the second neural network NN2 can be performed according to the backpropagation technique described above. The resulting trained second neural network NN2 is trained to output a predicted value of the pulmonary perfusion index when a new image from the first imaging method (e.g., CT imaging method) or the second imaging method (e.g., X-ray projection imaging method) is provided. The second neural network NN2 can also be trained to predict the pulmonary perfusion index from images from other types of imaging methods, such as an MRI imaging method. In this case, the training of the second neural network NN2 is performed in a similar manner. Usually, the second neural network NN2 is trained to generate a normalized measurement value 150 1…i for medical images from a single type of imaging method or for medical images from multiple types of imaging methods. Separate neural networks can also be trained to generate a normalized measurement value 150 1…i for different types of imaging methods.

[0081] In other examples, a computer program product is provided. The computer program product has instructions that, when executed by one or more processors, cause the one or more processors to execute a method for compensating for differences in medical images, the method comprising: a time series 110 of medical images 1…iA time series 110 of medical images including one or more medical images generated by a first type of imaging method 120 and one or more medical images generated by second types of imaging methods 130, 130', where the first type of imaging method is different from the second type of imaging method 1…i Receiving step S110 of receiving image data including the time series 110 of medical images; The time series 110 of medical images 1…i From the time series 110 of medical images, a normalized time series 140 of medical images 1…i Or one or more normalized measurement values 150 representing regions of interest in the time series 110 of medical images 1…i Generating steps S120a, S120b; and 1…j Outputting steps S130a, S130a', S130b of outputting each of the normalized time series 140 of medical images The normalized time series 140 of medical images 1…i And / or one or more measurement values derived from the normalized time series 140 of medical images, or one or more normalized measurement values 150 1…i 1…j Having.

[0082] In other examples, a system 200 for compensating for differences in medical images is provided. The system has one or more processors 210, which: Receiving (S110) image data including the time series 110 of medical images The time series 110 of medical images 1…i A time series 110 of medical images including one or more medical images generated by a first type of imaging method 120 and one or more medical images generated by second types of imaging methods 130, 130', where the first type of imaging method is different from the second type of imaging method 1…i Generating (S120a, S120b) a normalized time series 140 of medical images from the time series 110 of medical images The normalized time series 140 of medical images 1…i Or one or more normalized measurement values 150 representing regions of interest in the time series 110 of medical images 1…i 1…i 1…j Generating (S120a, S120b); and 1…i Outputting steps S130a, S130a', S130b of outputting each of the normalized time series 140 of medical images The normalized time series 140 of medical images 1…i And / or the normalized time series 140 of medical images 1…iOne or more measurement values derived from, or one or more normalized measurement values 150 1…j Output each of them (S130a, S130a’, S130b); It is configured as follows.

[0083] An example of the system 200 is shown in FIG. 2. The system 200 includes: a plurality of medical imaging systems that generate image data received in the process S110, such as, for example, the projection X-ray imaging system 220 shown in FIG. 2, and a CT imaging system or an MRI imaging system; a normalized time series 140 of medical images 1…i , a normalized time series 140 of medical images 1…i One or more measurement values derived from, and one or more normalized measurement values 150 1…j A display device such as a monitor 250 and a tablet for displaying output data such as; a patient bed 260; and a user input device (not shown in FIG. 2) for receiving user input related to the processes executed by the system, such as a keyboard, a mouse, a touch screen, etc. It should be noted that one or more of them may be included.

[0084] An enumerated list of examples of the present disclosure is shown below.

Example

[0085] A computer-implemented method for compensating for differences in medical images, the method comprising: A time series of medical images (110 1…i ) including one or more medical images generated by a first type of imaging method (120) and one or more medical images generated by a second type of imaging method (130, 130’), and the first type of imaging method is different from the second type of imaging method, receiving image data including a time series of medical images (S110); From the time series of medical images (110 1…i ), a normalized time series of medical images (140 1…i ) or one or more normalized measurement values (150 1…i ) representing a region of interest in the time series of medical images (110 1…jSteps (S120a, S120b) of generating Normalized time series of medical images (140 1…i ) and / or one or more measurement values derived from the normalized time series of the medical images (140 1…i ) or one or more normalized measurement values (150 1…j ) are output respectively in steps (S130a, S130a’, S130b); has.

Example

[0086] A computer - implemented method according to Example 1, wherein the generating step has a step (S120a) of generating a normalized time series of medical images (140 1…i ) from a time series of medical images (110 1…i ), and the outputting step has steps (S130a, S130a’) of outputting one or more measurement values derived from the normalized time series of medical images (140 1…i ) and / or the normalized time series of the medical images (140 1…i ), and the normalized time series of medical images (140 1…i ) represents an image generated by a normal - type imaging method.

Example

[0087] A computer - implemented method according to Example 2: The first type of imaging method (120) is a projection imaging method, the second type of imaging method (130, 130’) is a volume imaging method, and the normal - type imaging method is a projection imaging method; and The normalized time series of medical images (140 1…i ) is the normalized time series of medical images from the time series (110 1…i) is generated by providing a combination of one or more medical images and one or more projection images from , and the one or more projection images project one or more medical images generated by a second type of imaging method (130, 130') such that the one or more projected images correspond to one or more of the medical images generated by a first type of imaging method (120).

Example

[0088] A computer-implemented method according to Example 3, wherein the projecting step includes projecting one or more medical images generated by a second type of imaging method onto a virtual detector (240v) using a virtual radiation source (230v).

Example

[0089] A computer-implemented method according to Example 3 or Example 4, wherein: the projecting step is based on a known relative arrangement between the virtual radiation source, the virtual detector, and the subject represented in one or more medical images generated by a second type of imaging method; and / or the projecting step includes adjusting the relative arrangement between the virtual radiation source, the virtual detector, and one or more medical images generated by a second type of imaging method (130, 130') such that the shape of one or more anatomical feature structures in the one or more projected images matches the shape of one or more corresponding anatomical feature structures in one or more medical images generated by a first type of imaging method (120).

Example

[0090] A computer-implemented method according to any one of Examples 2 to 5, wherein the step (S120a) of generating a normalized time series (140 1…i ) of the medical images warps one or more of the images in the normalized time series (140 1…i ) of the medical images to obtain one or more warped images (140' 1,2) having a step of causing the shape of one or more anatomical feature structures (160) within ) to match the shape of one or more anatomical feature structures (160) within a reference image (170).

Example

[0091] A computer-implemented method according to Example 6, wherein the warping step is based on a mapping between a plurality of corresponding landmarks (180 i…j ) represented in both the warped image and the reference image (170).

Example

[0092] A computer-implemented method according to any one of Examples 2 to 7, wherein the step (S120a) of generating the normalized time series (140 1…i ) of medical images comprises: adjusting the intensity of an image within the normalized time series (140 1…i ) of medical images based on the intensity at one or more positions within a reference image (170); and / or adjusting the intensity of an image within the normalized time series (140 1…i ) of medical images using an image style transfer algorithm; and has.

Example

[0093] A computer-implemented method according to any one of Examples 2 to 8, the method comprising: receiving an input defining a region of interest in a received time series (110 1…i ) of medical images; and suppressing one or more image feature structures outside or within the region of interest in the normalized time series (140 1…i ) of medical images; and further has.

Example

[0094] A computer-implemented method according to Example 9, wherein the region of interest is defined within a reference image (170), the method comprising: Mapping the contour of the region of interest from the reference image to the images in the normalized time series (140 1…i ) of the medical image; and Adjusting the image intensity values of the images in the normalized time series (140 1…i ) of the medical image outside or inside the mapped contour to suppress one or more image feature structures outside the region of interest; Further comprising.

Example

[0095] A computer-implemented method according to any one of Examples 6 to 10, wherein the reference image (170) is an image from the received time series (110 1…i ) of the medical image, an image from the normalized time series (140 1…i ) of the medical image, or an atlas image.

Example

[0096] A computer-implemented method according to Example 2, wherein: The first type of imaging method (120) is a projection imaging method, the second type of imaging method (130, 130') is a volume imaging method, and the normal type of imaging method is a projection imaging method; The normalized time series (140 1…i ) of the medical image is generated by providing the normalized time series of the medical image as a combination of one or more medical images and one or more projection images from the time series (110 1…i ) generated by the first type of imaging method (120), and the one or more projection images are provided by inputting one or more medical images generated by the second type of imaging method (130, 130') into a neural network (NN1); and The neural network (NN1) is trained to generate a projection image corresponding to the first type of imaging method (120) for each of the input images.

Example

[0097] A computer-implemented method according to Example 12, wherein the neural network (NN1) is for generating a projection image corresponding to a first type of imaging method (120) for each of the input images: Receiving training data including a plurality of volumetric training images representing regions of interest, wherein the volumetric training images are generated by a second type of imaging method (130, 130'); Receiving ground truth data including corresponding ground truth projection images generated by the first type of imaging method (120) for each of the volumetric training images; Inputting the training data into a neural network (NN1); and For each of the plurality of input volumetric training images: Predicting a corresponding projection image using the neural network (NN1); Adjusting the parameters of the neural network (NN1) based on the difference between the predicted projection image and the ground truth projection image; and Repeating the predicting and adjusting until a stopping criterion is met, Thereby being trained.

Example

[0098] A computer-implemented method according to Example 1, wherein The generating step includes a step (S120b) of generating one or more normalized measurement values (150 1…i ) representing a region of interest from a time series (110 1…i ) of the medical images, and 1…i The outputting step includes a step (S130b) of outputting one or more normalized measurement values (150 ) representing a region of interest, 1…i One or more normalized measurement values (150 ) representing a region of interest are: 1…i From a time series (110 ) of medical images 1…i) is input into a neural network (NN2); and using the neural network (NN2) that responds to the input, a normalized measurement value (150 1…i ) representing a region of interest for each of the input images (110 1…i ) is generated; generated therefrom; The neural network (NN2) is trained using training data to generate a normalized measurement value (150 1…i ) representing a region of interest for each of the input images (110 1…i ), and the training data includes a plurality of training images representing regions of interest, the training images including a plurality of images generated by a first type of imaging method (120) and a plurality of images generated by a second type of imaging method (130, 130’), and corresponding ground truth values of normalized measurement values representing regions of interest for each training image.

Example

[0099] A computer-implemented method according to Example 14, wherein the neural network (NN2) is for generating a normalized measurement value (150 1…i ) representing a region of interest for each of the input images: receiving training data; and for each of the plurality of training images in the training data: inputting the training image into the neural network; predicting a value of a normalized measurement value representing a region of interest using the neural network (NN2); adjusting the parameters of the neural network (NN2) based on the difference between the predicted value of the normalized measurement value and the corresponding ground truth value; and repeating the prediction and adjustment until a stopping criterion is met; and thereby being trained.

[0100] The above examples are to be understood as illustrative of the present disclosure and not limiting. Further examples are conceivable. For example, the examples described in relation to the computer-implemented method can also be provided in corresponding manners by a computer program product, by a computer-readable storage medium, or by system 200. The features described in relation to any one example can be used alone or in combination with other described features, and can also be used in combination with one or more features of other examples of the above examples or combinations of other examples. Furthermore, equivalents and modifications not described above can also be employed without departing from the scope of the invention defined in the appended claims. In the claims, the term "comprising" does not exclude other elements or acts, and the singular form does not exclude the plural. The mere fact that certain features are described in mutually different dependent claims does not indicate that a combination of these features cannot be used advantageously. Any reference signs in the claims should not be construed as limiting its scope.

Claims

1. A computer-based method for compensating for differences in medical images, wherein the computer-based method is A step of receiving image data that includes a time series of medical images, which includes one or more medical images generated by a first type of imaging method and one or more medical images generated by a second type of imaging method, wherein the first type of imaging method is different from the second type of imaging method. The steps include generating a normalized time series of medical images representing images generated by a standard type of imaging method from the aforementioned time series of medical images, A step of outputting a normalized time series of the medical image and / or one or more measured values ​​derived from the normalized time series of the medical image. A computer implementation method having the following characteristics.

2. The first type of imaging method is a projection imaging method, the second type of imaging method is a volume imaging method, and the conventional type of imaging method is a projection imaging method. The normalized time series of the medical images is generated by providing the normalized time series of the medical images as a combination of one or more medical images and one or more projection images from the time series generated by the first type of imaging method, and is provided by the step of projecting the one or more projection images onto one or more medical images generated by the second type of imaging method such that the one or more projection images correspond to one or more medical images generated by the first type of imaging method. The computer implementation method according to claim 1.

3. The computer-aided method according to claim 2, wherein the projection step includes a step of projecting one or more medical images generated by the second type of imaging method onto a virtual detector using a virtual radiation source.

4. The projection step is based on a known relative arrangement between the virtual source, the virtual detector, and the subjects represented in one or more medical images generated by the second type of imaging method, and / or The projection step includes adjusting the relative arrangement between the virtual source, the virtual detector, and one or more medical images generated by the second type of imaging method such that the shape of one or more anatomical feature structures in the projected one or more images corresponds to the shape of one or more corresponding anatomical feature structures in one or more medical images generated by the first type of imaging method. The computer implementation method according to claim 2.

5. The computer implementation method according to claim 1, wherein the step of generating a normalized time series of medical images includes the step of warping one or more images in the normalized time series of medical images so that the shape of one or more anatomical feature structures in the warped one or more images corresponds to the shape of one or more anatomical feature structures in a reference image.

6. The computer implementation method according to claim 5, wherein the warping step is based on mapping between a plurality of corresponding landmarks represented in both the warped image and the reference image.

7. The step of generating a normalized time series of the medical images is: A step of adjusting the intensity of images within a normalized time series of the medical images based on the intensity at one or more locations in a reference image, and / or The step of adjusting the intensity of the images within a normalized time series of the medical images using an image style transfer algorithm. A computer implementation method according to claim 1, comprising:

8. The steps include receiving an input that defines a region of interest in the time series of the received medical images, The steps include: suppressing one or more image feature structures outside or inside the region of interest in the normalized time series of the medical images; The computer implementation method according to claim 1, further comprising the following:

9. The region of interest is defined within the reference image, and the computer implementation method is The steps include mapping the contour of the region of interest from the reference image to the image in the normalized time series of the medical image, The steps include: adjusting the image intensity values ​​of the images in the normalized time series of the medical images outside or inside the mapped contour to suppress one or more image feature structures outside the region of interest; The computer implementation method according to claim 8, further comprising the above.

10. The computer implementation method according to claim 1, wherein the reference image is provided by an image from a time series of the received medical images, an image from a normalized time series of the medical images, or an atlas image.

11. The first type of imaging method is a projection imaging method, the second type of imaging method is a volume imaging method, and the conventional type of imaging method is a projection imaging method. The normalized time series of the medical images is generated by providing the normalized time series of the medical images as a combination of one or more medical images and one or more projection images from the time series generated by the first type of imaging method, and the one or more projection images are provided by inputting one or more medical images generated by the second type of imaging method into a neural network and generating the one or more projection images using the neural network in response to the input. The neural network is trained to generate a projection image corresponding to the first type of imaging scheme for each of the input images using training data, wherein the training data includes a plurality of volume training images representing the region of interest generated by the second type of imaging scheme and a corresponding ground truth projection image generated by the first imaging scheme for each volume training image. The computer implementation method according to claim 1.

12. The neural network generates a projection image corresponding to the first type of imaging method for each of the input images. The aforementioned training data is received, For each of the multiple volume training images in the training data, The volume training images are input to the neural network. Using the aforementioned neural network, predict the corresponding projection image. The parameters of the neural network are adjusted based on the difference between the predicted projection image and the ground truth projection image. The above prediction and adjustments will be repeated until the stopping criteria are met. The computer implementation method according to claim 11, which is trained by the following.

13. A computer program that, when executed by one or more processors, includes instructions causing one or more processors to perform a method for compensating for differences in medical images, wherein the method is A step of receiving image data including a time series of medical images, wherein the time series includes one or more medical images generated by a first type of imaging method and one or more medical images generated by a second type of imaging method, and the first type of imaging method is different from the second type of imaging method. The steps include generating a normalized time series of medical images representing images generated by a standard type of imaging method from the aforementioned time series of medical images, A step of outputting a normalized time series of the medical image and / or one or more measured values ​​derived from the normalized time series of the medical image. A computer program that has [a certain characteristic].

14. A system for compensating for differences in medical images, comprising one or more processors, Image data including a time series of medical images is received, wherein the time series includes one or more medical images generated by a first type of imaging method and one or more medical images generated by a second type of imaging method, and the first type of imaging method differs from the second type of imaging method. From the aforementioned time series of medical images, a normalized time series of medical images representing images generated by a standard type of imaging method is generated. Outputs a normalized time series of the medical image and / or one or more measured values ​​derived from the normalized time series of the medical image. system.

Images