Systems and methods for training machine learning models and for providing an estimated internal image of a patient

Abstract

A deep learning model can be trained to provide an estimated image of a patient's interior based on a plurality of image sets, each image set comprising an internal image of a person's interior and an outline image of a person's outer contour at a particular point in time. The model is trained to establish an optimized parameterized conversion function G that specifies a correlation between a person's interior and a person's outer contour based on the image sets. The conversion function G can then be used to provide an estimated image of a patient's interior based on a patient's contour.

Description

Technical Field

[0001] This invention relates to a method, computer program product, and computer system for providing estimated images of a patient in medical applications. Background Technology

[0002] In radiotherapy, the patient receiving treatment is typically positioned on a treatment bed, but may also stand or sit in a chair. Movement within the patient can occur due to factors such as breathing, coughing, or discomfort. Such movement can cause radiation to miss its target, striking another organ, potentially one at risk. This can result in an underdose to the target and / or an overdose to other tissues or organs, both undesirable and potentially harmful. This problem has been addressed in several different ways.

[0003] Various attempts have been made to prevent such movement. For example, the possibility of movement during treatment may be restricted. In particular, patients may be instructed to hold their breath or be forced to move only within a certain range. This causes discomfort and is only possible to a certain extent and / or for a limited time.

[0004] Other attempts to address this issue include robust planning that takes into account the uncertainty surrounding the location of various critical organs. This can result in plans that are not optimal, as they must accommodate many different possible scenarios.

[0005] Different target tracking methods have also been proposed. European Patent Application No. 18180987 proposes a method involving imaging the patient during different phases of the respiratory cycle and planning the total dose as the sum of the phase doses, while taking into account the different locations of the target and other organs at different phases. This requires a large number of 3D images, typically CT images of the patient taken throughout the treatment, resulting in considerable additional radiation exposure to the patient, which is undesirable. An alternative method is to use MR imaging instead. This does not involve radiation, but is much slower than CT imaging and / or cannot provide the same image quality. Radiolabeling for target tracking has also been proposed. This is only able to track the target without detecting changes in the position of other organs. Summary of the Invention

[0006] The purpose of this invention is to take into account the periodic or unplanned movement of the patient during the treatment phase in radiotherapy treatment planning.

[0007] According to the present invention, this objective is achieved by a computer-based method for training a deep learning model, which is used to provide estimated images of the patient's interior based on multiple image sets, each image set including a first interior image of the person's interior at a specific time point, a contour image of the person's outer contour, and a second interior image of the person's interior. The method includes the following steps:

[0008] a. Submit the image set to a deep learning model, which is configured to output estimated images based on contour images and second interior images.

[0009] b. Train the model to establish an optimized parametric transformation function G, which specifies the correlation between the interior and exterior contours of a person based on an image set. For at least one image set, apply the model to the contour image and the second interior image of the image set, compare the output with the first interior image, and use the result of the comparison to train the model.

[0010] The present invention also relates to a method for providing an estimated image of the patient's interior at a first time point, comprising the following steps:

[0011] • Internal images of the patient's interior at a second time point prior to the first time point are fed into a deep learning model, which includes an optimized parameterized transformation function based on the correlation between the patient's contours and interior.

[0012] • Provide the deep learning model with the contour image of the patient's outline at the first time point.

[0013] • Output estimated images of the patient based on internal images, contour images, and an optimized parametric transformation function G from a deep learning model.

[0014] The method according to the invention is based on the fact that, in many cases, there is a correlation between a patient's external contours and internal structures, including the location of one or more internal structures such as organs or tissues. One or more images used to provide internal data are typically one or more images taken of the patient during treatment planning and procedures, such as one or more partial images and / or planning images. In methods for training deep learning models, these steps are preferably repeated for all image sets. Each contour image may be based on the same image as the corresponding internal image, such as a CT scan of the patient. Alternatively, the contour image may be based on separate image data, such as data obtained from a surface scanning device. Typically, the estimated images are used to plan medical procedures that require information about the patient's internal structure, such as radiotherapy treatment plans, or to modify existing plans. If surface scans are performed repeatedly, contour data from subsequent surface scans can be used to generate a 4D image consisting of a series of 3D images corresponding to different time points. In some embodiments, the first internal image and contour image of each image set are 4D images, and the model is trained to output a synthetic 4D image.

[0015] Essentially, a first internal image, taken simultaneously with the contour image, serves as the target for what the model's output should be when based on the contour image and the second internal image. Therefore, the training is based on a comparison between the model's output (i.e., the first estimated image) and the first internal image. The second internal image can be an image taken at an earlier time point, such as a planned or partial image of the patient, or it can be an estimated image from a previous step in training the model. Advantageously, the second internal image is taken temporally close to the contour image, for example, a partial image taken before the same portion as the contour image, or an estimated image generated by a previous step in training the model.

[0016] In methods of providing estimated images, the contour images are preferably based on data obtained from a surface scanning device. Such surface scanning devices are commonly available in hospitals, for example in radiotherapy delivery systems, for other purposes such as ensuring proper patient positioning for each treatment segment. Therefore, surface scanning data can often be obtained without requiring additional equipment. Furthermore, surface scanning data can be obtained without exposing the patient to additional doses.

[0017] Therefore, the present invention can provide one or more estimated internal images of a patient, which reduces the need to obtain actual images of the patient, for example, for updating treatment plans after a certain number of treatment sessions. The present invention makes it possible to estimate the dose delivered to the patient during each session without obtaining new images of the patient. This reduces the need to expose the patient to radiation in order to obtain images at different time points. This is also useful in situations where further imaging would cause discomfort to the patient.

[0018] Machine learning systems are available that, based on large datasets, can determine correlations between different types of data and use these correlations to process input data. According to the present invention, the correlation between a person's external contour and interior, particularly the location of organs or other targets within the patient's body, can be determined based on prior datasets where both the contour and interior are known. This correlation is represented in the form of a parameterized transformation function G, which is arranged to transform a contour image into an estimated image of the patient's interior.

[0019] In a preferred embodiment of the training method, the training step includes...

[0020] • Obtain an initial parameterized transformation function G, which is arranged to transform the contour image into an estimated image of the patient's interior based on a first image set from multiple image sets;

[0021] • The first estimated interior image is obtained by applying the parameterized transformation function G to the contour image of the first image set and the second interior image.

[0022] In the first comparison step, the first estimated internal image is compared with the internal image of the first image pair.

[0023] • And based on this comparison, adjust the initial parameterized transformation function G to form the first parameterized transformation function G1.

[0024] In this case, the training steps may also include:

[0025] • Apply the first parameterized transformation function G1 to the contour images of the second image set to obtain the second estimated image;

[0026] • Compare the second estimated image with the internal image of the second image pair, and adjust the initial parameterized transformation function G to form the second parameterized transformation function G2.

[0027] Typically, a series of such training steps are performed. The initial parameterized transformation function G can be adjusted for each training step, or it can be adjusted after multiple training steps.

[0028] In the latter case, the training steps may also include:

[0029] • A second estimated interior image is obtained by applying the parameterized transformation function G to the contour images of the second image set.

[0030] In the second comparison step, the second estimated internal image is compared with the internal image of the second image pair.

[0031] • And adjust the initial parameterized transformation function G based on the first and second comparison steps to form the first parameterized transformation function G1

[0032] The first and second internal images of each image set can be segmentation maps, in which case the model is trained to output the segmentation maps. The estimated image will then also be a segmentation map. Alternatively, the internal images of each image set can be images such as CT or MR images, and the model is trained to output a synthetic CT image or a synthetic MR image as the estimated image, respectively. In other words, the model is typically trained to output a synthetic image with the same modality, format, and level of detail as the internal images. Of course, additional transformation steps can be added to obtain images of another modality or format.

[0033] In addition to internal and outline images, each image set may also include at least one slice of MR images to provide additional information about the patient's interior.

[0034] The present invention also relates to a computer program product, which, when executed in a computer's processor, is arranged to cause the computer to perform the method according to any one of the preceding claims. The computer program product may be stored on a storage device, such as a non-transitory storage device.

[0035] The present invention also relates to a computer system including a processor and a program memory, the program memory including a computer program product according to the above description. Attached Figure Description

[0036] The invention will now be described in more detail by way of example and with reference to the accompanying drawings, wherein

[0037] Figure 1 An imaging system that can be used in this invention is shown.

[0038] Figure 2 It shows the movement of the patient's outline and the corresponding movement of the internal organs within the patient's body.

[0039] Figure 3 This is a flowchart of a method for training a deep learning model used in this invention.

[0040] Figure 4 It is used according to Figure 3 The flowchart describes a method for creating estimated images of patients using a deep learning model obtained through this approach.

[0041] Figure 5 The machine learning model that can be used according to the present invention is shown. Detailed Implementation

[0042] Figure 1This is an overview of a system 10 for radiotherapy treatment and / or treatment planning. As will be understood, such a system can be designed in any suitable manner, and the design shown in Figure 7 is merely an example. A patient 1 is located on a treatment bed 3. The system includes an imaging / treatment unit having a radiation source 5 mounted in a gantry 7 for emitting radiation to the patient located on the treatment bed 3. Typically, the treatment bed 3 and the gantry 7 can move relative to each other in several dimensions to provide radiation to the patient as flexibly and accurately as possible. These components and their functions are well known to those skilled in the art. Many devices are typically provided for beam shaping in the lateral and depth directions, and these will not be discussed in detail herein. The system also includes a computer 21 that can be used for radiotherapy treatment planning and / or for controlling radiotherapy treatment. As will be understood, the computer 21 can be a separate unit not connected to the imaging / treatment unit.

[0043] Computer 21 includes processor 23, data memory 24, and program memory 25. Preferably, one or more user input devices 28, 29 are also provided, in the form of a keyboard, mouse, joystick, voice recognition device, or any other available user input device. The user input devices may also be arranged to receive data from external memory units.

[0044] When the system is used for planning, data storage 24 includes clinical data and / or other information for obtaining a treatment plan. Typically, data storage 24 includes one or more patient images to be used in the treatment plan. For training purposes, the data storage holds a training set of input data, as will be discussed in more detail below. Each input dataset includes a contour image of at least a portion of the patient taken approximately simultaneously and an interior image of the contour, as well as any other data that may aid training. To generate estimated interior images, the data storage includes at least an initial interior image of the patient and a contour image of the patient taken at a time different from the interior images. Program storage 25 stores data arranged to cause the processor to execute according to… Figure 3 or Figure 4 At least one computer program for the method. The program memory 25 also stores a computer program arranged to cause the computer to execute the combined... Figure 3 or Figure 4 The methods and steps described enable computer-controlled radiotherapy treatment for patients.

[0045] Depending on the detail of the internal images used in the training set, the estimated image can include different levels of detail. It can be a segmented image that simply shows the location and shape of one or more organs or structures inside the patient's body, or it can have a level of detail comparable to a CT image.

[0046] As will be understood, data memory 24 and program memory 25 are shown and discussed only schematically. There may be several data memory units, each storing one or more different types of data, or a single data memory storing all data in a suitably structured manner, and the same applies to program memory. One or more memories may also be stored on other computers. For example, a computer may be configured to execute only one method, while another computer is used to perform optimizations.

[0047] Figure 2 A portion of the patient's torso 40 is shown to illustrate the potential correlation between the patient's movement during the respiratory cycle and the location of structures 42 (such as tumors or organs) at risk within the patient's body. The initial location of structure 42 is indicated by a solid line. As the patient inhales and exhales, the outer contour 44 of the anterior part of the patient will move outward and inward, indicated by a first arrow 46. Simultaneously, structure 42 will move in directions different from the contour, generally downward and outward, as indicated by a second arrow 48. Examples of changes in the position of the contour and structure are shown by dashed lines.

[0048] According to the present invention, machine learning is used to train a deep learning model using 4D images and corresponding surface contours. In a preferred embodiment, the 4D images are CT images, but they can be acquired using any suitable imaging technique, including MRI. The input data is preferably an image pair, a first surface image and a first CT image taken at a first time point T0, and a second surface image and a second CT image taken at a second time point T1, and so on. Figure 3 Examples of this type of training method are shown in the image.

[0049] In the first step S31, multiple pairs of images are provided, each pair consisting of a surface contour of the person and a 3D interior image of the person, captured substantially simultaneously. As discussed above, from segmented images to a complete CT image, the interior image may include a level of detail depending on the desired level of detail of the resulting estimated image. The surface contour can be provided from a surface scanner available near the imaging device, or contour data for training can be obtained from the 3D image. In the second step S32, the image pairs are used for machine learning to establish a relationship between the surface contour and the location of at least one region of interest inside the human body. As is common in the prior art, this involves generating an optimized parameterized function to transform the image of the surface contour into an estimated image of the contour's interior. Typically, this is achieved by submitting a first surface image to the function, usually along with early interior images of the patient, possibly other image data, and an activation dataset. The output of this function is compared with a first CT image captured simultaneously with the first surface image. The result of the comparison is used to refine the function. The function is not refined after each training step, but rather after a certain number of steps, or only at the end of the process. Next, a second surface image is submitted, and a potentially refined function, along with the previously used earlier interior image or another earlier interior image and the activation dataset, is used to provide the second output data. The second output data is compared to a second CT image, and the result of this comparison is used to further refine the function and the activation dataset. This process is repeated for multiple sets of surface and CT images. The result of this process is function O31, which can be used to convert the patient's contour data into an estimated image of the patient's interior within the contour.

[0050] Preferably, a recurrent convolutional neural network (RCNN) is used. RCNN considers information from previous steps and thus provides information about the surface or image at a previous time point. A function is established that correlates a pair of images obtained simultaneously, and this function can be used in subsequent processes to create estimated or synthetic images of other patients.

[0051] Once a deep learning model has been trained—that is, an optimized parameterized function has been generated—knowledge about changes in the patient's contours can be used to determine the location of internal organs within the patient. The input data for this process consists of information related to the patient's surface contours and internal images (typically partial images). Specifically, surface data from different time points can be used as input data to return estimated or synthetic images of the patient at different time points. Figure 3 A function is established to create an estimated image, relating the contour to the internal CT image defined in the deep learning model. Therefore, Figure 4 It has a first step S41 and a second step S42, in which contour data from the patient is provided, for example, in... Figure 3In the second step S42, the model created in step S32 is used to create a contour-based estimated or synthetic image of the patient's interior.

[0052] Any type of recurrent neural network architecture can be used. What all recurrent neural networks have in common is that information from earlier time points is incorporated into the model. Figure 5 A basic example of a recurrent neural network is shown. As is common in the prior art, this model is arranged to take an input data sequence x... <t>< / t> The model uses parameterized functions, each represented by a rectangle. The execution of each function is called a step. Each circle within a rectangle represents a portion of the parameterized function being optimized, also called a layer, which includes weights and computations. This model can be used based on input data and activation data vectors a. <t>To generate the output dataset In each case, the label <t>This represents a specific point in time related to the data. Therefore, for example, x... <1> This includes the initial patient image taken at t=0, and the contour image taken at t=1. Similarly, Based on input data x <1> and activation data a <0> The estimated image of the patient at t=1, as is known in the art, can be a zero vector, but can also be a suitable input activation dataset. As is known in the art, the equation associated with a basic recurrent convolutional neural network is (where * denotes convolution):

[0053] a0 <t>< / t> =g0(W a0 *a0 <t-1>< / t-1> +W x0 *x <t>< / t> +b a0 )

[0054] a1 <t>< / t> =g1(W a0 *a1 <t-1>< / t-1> +W x0 *a0 <t>< / t> +b a1 )

[0055] …

[0056] a n <t>< / t> =g n (W an *a n <t-i>< / t-i> +W xn *a n-1 <t>< / t> +b an )

[0057] …

[0058]

[0059] in

[0060] g n This indicates functions that can be different or the same.

[0061] a n <t>< / t> This represents the activation data used for level n at time t+1.

[0062] W an This represents the weights applied to activations from earlier time points.

[0063] W xn This represents the weights applied to activations from previous layers, and

[0064] W yn This represents the weights applied to activations from the last layer. It is the output at time t, and b an and b y It is the bias value.

[0065] The values ​​in W and b are the values ​​being optimized.

[0066] During model training, the weights W and biases b are updated, typically after each step. Figure 5 This occurs after each rectangle in the algorithm, but alternatively after multiple steps, or after the last step. The comparison between the estimated image and the input internal image is typically represented as a penalty term in the cost function used when optimizing the parameters. This is based on the sum of differences identified in each comparison, as discussed above in conjunction with step S32.

[0067]

[0068] The operator "-" represents a comparison, not necessarily subtraction. The parameters are optimized to minimize this difference.

[0069] Another possible penalty term is based on the use of a classification function D, which attempts to distinguish between real and generated images. The network can be optimized in conjunction with G. D is optimized to minimize the classification error between real and generated images. G is optimized to maximize this classification error.

[0070] Note Figure 5 The exemplary models shown are merely examples. As those skilled in the art will know, there are several types of neural networks, and any suitable neural network can be used according to the invention. Recurrent Convolutional Neural Networks (RCNNs) are preferred, and a type of RCNN known as Long Short-Term Memory (LSTM) has been found to be particularly suitable for the method according to the invention.< / t> < / t>

Claims

1. A computer-based method for training a deep learning model to provide estimated images of a patient's interior based on multiple image sets, each image set including a first interior image of the person's interior at a specific time point, a contour image of the person's outer contour, and a second interior image of the person's interior, comprising the following steps: a. Submit the image set to the deep learning model; b. By applying the model to the contour image and the second interior image of the image set for each image set, comparing the output with the first interior image of the image set, and using the comparison result to train the model to establish an optimized parameterized transformation function G that specifies the correlation between the interior and the outer contour of the person, wherein the second interior image is earlier in time than the first interior image.

2. The computer-based method according to claim 1, wherein, The training steps include: • Obtain an initial parameterized transformation function G, which is arranged to transform the contour image into an estimated image of the patient's interior based on a first image set in the plurality of image sets; • A first estimated interior image is obtained by applying the parameterized transformation function G to the contour image and the second interior image of the first image set. In the first comparison step, the first estimated internal image is compared with the internal images of the first image set. • And adjust the initial parameterized transformation function G based on the comparison to form a first parameterized transformation function G1, which can be applied to the second image set.

3. The computer-based method according to claim 2, wherein, The training steps also include: • Apply the first parameterized transformation function G1 to the contour image of the second image set to obtain the second estimated image; • Compare the second estimated image with the internal image of the second image pair, and adjust the initial parameterized transformation function G to form the second parameterized transformation function G2.

4. The computer-based method according to claim 2, wherein, The training steps also include: • A second estimated interior image is obtained by applying the parameterized transformation function G to the contour image of the second image set. In the second comparison step, the second estimated internal image is compared with the internal image of the second image pair. • And adjust the initial parameterized transformation function G based on the first comparison step and the second comparison step to form a first parameterized transformation function G1.

5. The method according to any one of claims 1 to 4, wherein, The first and second inner images of each image set are segmentation maps, and the model is trained to output segmentation maps.

6. The method according to any one of claims 1 to 4, wherein, The first and second internal images of each image set are CT images, and the model is trained to output synthetic CT images.

7. The method according to claim 1, wherein, Each image set also includes at least one slice of the MR image to provide additional information about the patient's interior.

8. The method according to any one of claims 1 to 4, wherein, The first internal image and the outline image of each image set are 4D images, and the model is trained to output a synthetic 4D image.

9. A computer-based method for providing estimated images of the patient's interior at a first time point, comprising: a. An internal image of the patient at a second time point prior to the first time point is provided to a deep learning model, the deep learning model comprising an optimized parameterized transformation function based on the correlation between the patient's contours and interior, wherein the deep learning model is trained by the method according to any one of claims 1 to 8. b. Provide the deep learning model with a contour image of the patient's outline at the first time point. c. Output an estimated image of the patient based on the internal image, the contour image, and the optimized parametric transformation function G from the deep learning model.

10. The method according to claim 9, wherein, The contour image is based on data obtained from a surface scanning device.

11. The method according to claim 9 or 10, wherein, The estimated image is a segmented image of the patient.

12. The method according to claim 9 or 10, wherein, The estimated image is the patient's CT image.

13. The method according to claim 9 or 10, wherein, Steps b and c are repeated for several subsequent contour images to produce a set of estimated images that constitute the 4D image.

14. A computer program product, when executed in a processor of a computer, is arranged to cause the computer to perform the method according to any one of the preceding claims.

15. A computer system comprising a processor and a program memory, the program memory comprising a computer program product according to claim 14.

Images