Apparatus for predicting the progression of a state, apparatus for training a model to predict the progression of a state, and method for operating these apparatuses.

Abstract

A method for predicting a condition progression comprises obtaining measurement data relating to at least one measurement of an object up to a particular time point, the data being generated based on an output of a sensor configured to make the at least one measurement of the object. At least one parameter of a parameterized time-dependent function is generated using a trained model and based on the measurement data, the parameterized time-dependent function being dependent on successive time values, and the parameterized time-dependent function being indicative of a predicted condition progression of the object over time after the particular time point. The parameterized time-dependent function is evaluated using the at least one parameter for at least one time point after the particular time point.

Description

[Technical Field]

[0001] This invention relates to the prediction of progress profiles. Furthermore, this invention also relates to training a model to predict progress profiles. [Background technology]

[0002] For example, in medical imaging, the condition of a subject can be evaluated based on medical images and / or other measurements such as body temperature and heart rate. Attempts are also being made to predict a patient's future condition based on medical images.

[0003] Non-patent document 1 discloses training a model to segment a GA collaboratively and using a deep neural network to predict segmentation at a future point in time. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Zhang et al., "A multi-scale deep convolutional neural network for joint segmentation and prediction of geographic atrophy in SD-OCT images," 2019, 16th IEEE International Symposium on Biomedical Imaging (ISBI 2019), pp. 565-568. [Non-Patent Document 2] Apostolopoulos, S., De Zanet, S., Ciller, C., Wolf, S., and Sznitman, R., "Pathological oct retinal layer segmentation using branch residual u-shape networks," Medical Image Computing and Computer Assisted Intervention, pp. 294-301, (2017). [Non-Patent Document 3] Arcadu, F., Benmansour, F., Maunz, A., Willis, J., Haskova, Z., and Prunotto, M., "Deep learning algorithm predicts diabetic retinopathy progression in individual patients," npj Digital Medicine, 92(2) (2019). [Non-Patent Document 4] Boyer, DS; Schmidt-Erfurth, U.; van Lookeren Campagne, M.; Henry, EC; Brittain, C., "The pathopysiology of geographic atrophy secondary to age-related macular degeneration and the complement pathway as a therapeutic target," Retina 37(5) (2017). [Non-Patent Document 5] Engwer, C., Hillen, T., Knappitsch, M., and Surulescu, C., "Glioma follow white matter tracts: a multiscale dti-based model," Journal of Mathematical Biology, Vol. 71 (September 2014). [Non-Patent Document 6] Fleckenstein, M., Mitchell, P., Freund, K. B., Sadda, S., Holz, F. G., Brittain, C., Henry, E. C., Ferrara, D., "The progression of geographic atrophy secondary to age-related macular degeneration", Ophthalmology 125(3), 369 - 390 (2018)

Non-Patent Document 7

Non-Patent Document 8

Non-Patent Document 9

Non-Patent Document 10

Non-Patent Document 11

[0005] The objective of this invention is to enable the provision of improved progress profile prediction. [Means for solving the problem]

[0006] According to one aspect of the present invention, a method for predicting the progression of a state is provided. The method is Acquiring measurement data relating to at least one measurement of an object up to a specific point in time, wherein the data is generated based on the output of a sensor configured to perform the at least one measurement of the object. Using a trained model, generate at least one parameter of a parameterized time-dependent function based on the measured data, wherein the parameterized time-dependent function represents the progression of the predicted state of the subject over time after a particular point in time; It includes.

[0007] The method may further include evaluating the parameterized time-dependent function using the at least one parameter for at least one time point after the specific time point.

[0008] According to another aspect of the present invention, a method is provided for training a model to predict the progression of a state. The method is as follows: Acquiring training data comprising measurement data relating to at least one measurement of at least one object, wherein the measurement data is based on the output of a sensor configured to perform the at least one measurement of the object, and the training data further comprises information indicating at least one time point associated with the object for each object, and the state of the object at at least one time point. Generating at least one parameter of a parameterized time-dependent function using a model and based on the measured data of a specific object among the at least one object, wherein the parameterized time-dependent function generates a parameter that depends on consecutive time values ​​and indicates the predicted progression of the state of the specific object over time. The parameterized time-dependent function is evaluated using the at least one parameter for at least one time point associated with the specific object, and the predicted state of the object at at least one time point is obtained. The process involves comparing the predicted state of the specific object at at least one point in time associated with the specific object with the information in the training data indicating the state of the specific object at at least one point in time associated with the specific object, and obtaining a comparison result. The model is updated based on the comparison results, It includes.

[0009] In a particular embodiment, at least one first object among the at least one objects has at least one time point in a first set associated with it, at least one second object among the at least one objects has at least one time point in a second set associated with it, and at least one time point in the first set is different from each time point in the second set.

[0010] The at least one parameter can indicate the point in time when the state changes, or the rate at which the state changes.

[0011] The step of evaluating the parameterized time-dependent function may include applying a threshold to the parameters generated using the model, the threshold depending on the time point at which the parameterized time-dependent function is evaluated.

[0012] The at least one parameter can encompass at least one coefficient of the term of the parameterized time-dependent function.

[0013] The parameterized time-dependent function may encompass a Fourier series or a Taylor series.

[0014] The measurement data may include at least one N-dimensional input dataset associated with the object, generated by the data acquisition device, where N is an integer value of 1 or more. Preferably, N is an integer value of 2 or more.

[0015] In a particular example, the N-dimensional input dataset is at least a two-dimensional image dataset, and the data acquisition device is an imaging device.

[0016] The step of generating the at least one parameter may include generating the at least one parameter for each of a plurality of locations corresponding to locations in the N-dimensional input dataset, and defining the parameterized time-dependent function separately for each of the plurality of locations.

[0017] The measurement data may include at least one at least two-dimensional image dataset associated with the object, generated by the imaging device.

[0018] The step of generating the at least one parameter may include generating the at least one parameter for each of a plurality of locations corresponding to locations in the at least two-dimensional image dataset, and defining the parameterized time-dependent function separately for each of the plurality of locations.

[0019] The aforementioned model can encompass convolutional neural networks.

[0020] According to another aspect of the present invention, an apparatus for predicting the progression of a state is provided. The apparatus is An input configured to receive measurement data relating to at least one measurement of an object up to a specific point in time, wherein the measurement data is generated based on the output of a sensor configured to perform the at least one measurement of the object, Non-transient storage media encompassing the model, A processor system configured to generate at least one parameter of a parameterized time-dependent function using the model and based on the measurement data, wherein the parameterized time-dependent function represents the progression of the predicted state of the object over time after a specific point in time, It includes.

[0021] According to another aspect of the present invention, an apparatus is provided for training a model to predict the progression of a state. The apparatus is An input configured to receive training data comprising measurement data relating to at least one measurement of at least one object, wherein the measurement data is based on the output of a sensor configured to perform the at least one measurement of the object, and the training data further comprises information indicating at least one time point associated with the object for each object, and the state of the object at at least one time point; A storage medium for storing the model, A processor system, wherein the device includes, Generating at least one parameter of a parameterized time-dependent function using the model and based on the measurement data of a specific object among the at least one object, wherein the parameterized time-dependent function represents the predicted progression of the state of the specific object over time. The parameterized time-dependent function is evaluated using the at least one parameter for at least one time point associated with the specific object, and the predicted state of the object at at least one time point is obtained. The process involves comparing the predicted state of the specific object at at least one point in time associated with the specific object with the information in the training data that indicates the state of the specific object at at least one point in time associated with the specific object, and obtaining a comparison result. The model is updated based on the comparison results, A processor system configured to perform the following: It includes.

[0022] By using a parameterized time-dependent function as the basis for predicting the progression profile of a condition, the prediction is not constrained to a specific point in time, but can instead be evaluated at any point in time, particularly in the future or at any point after the last available measurement. This allows for comparison of predictions with reality as new diagnostic data becomes available. This provides flexibility in planning. For example, it can provide flexibility in treatment planning, including, but is not limited to, monitoring adherence and response to treatment.

[0023] When training a model, a parameterized time-dependent function allows for the handling of ground truth data that may be available at aperiodic or irregular time intervals for different subjects.

[0024] A simple example of a time-dependent function can be a step function. Thresholding techniques can be particularly advantageous for monotonically progressive diseases, in which case the parameters generated using the model can indicate the rate of change in the state. This is particularly beneficial for a small number of parameters (e.g., just one parameter) that need to be estimated by the model.

[0025] More complex parameterized functions, such as finite Fourier or Taylor series, allow for the efficient encoding of relatively complex time-dependent behavior and can be useful for more detailed predictions of the progress profile. This can potentially allow for more efficient training of the model.

[0026] Those skilled in the art will understand that the above-mentioned features can be combined in any way deemed useful. Furthermore, modifications and variations described in relation to systems can also be applied to methods and computer program products, and similarly, modifications and variations described in relation to methods can also be applied to systems and computer program products.

[0027] The embodiments of the present invention will be described below with reference to the drawings, using examples. These drawings are illustrative and may not be drawn to scale. Similar items may be marked with the same reference numeral throughout the drawings. [Brief explanation of the drawing]

[0028] [Figure 1] This is a flowchart that explains how to predict the progression of a state. [Figure 2] This flowchart explains how to train a model to predict the progression of a state. [Figure 3] This is a block diagram illustrating a device for predicting states or training models. [Figure 4A] An example of a B-scan with GA annotations is shown. [Figure 4B] Figure 4A shows the B scan with segmented layers. [Figure 5] Figures 5(a) through (i) show examples of input thickness maps for a projected volume segment network. Figure 5(j) shows manual segmentation of atrophy. [Figure 6] An example training workflow following this disclosure is outlined below. [Figure 7A] The dice scores of the evaluated methods are shown. [Figure 7B] The area difference scores for the evaluated methods are shown. [Figure 8] A summary of results from a single patient is provided. [Modes for carrying out the invention]

[0029] Next, specific exemplary embodiments will be described in more detail here with reference to the attached drawings.

[0030] The details of the configuration and elements disclosed in this description are provided to aid in a comprehensive understanding of the exemplary embodiments. Therefore, it is clear that these exemplary embodiments are executable without their specific definitions. Furthermore, well-known operations or structures are omitted here, as unnecessary details would obscure the explanation.

[0031] The following discloses specific embodiments of geographic atrophy of the visual system that can be diagnosed based on a three-dimensional depiction of the retinal structure using optical coherence tomography (OCT). However, it should be understood that this technique is generally applicable to any disease or condition, and that any other imaging techniques, such as computed tomography, magnetic resonance imaging, and ultrasound imaging, and any other imaging equipment, can be used, rather primarily depending on the disease or condition to be evaluated. Furthermore, this technique can also be applied to two-dimensional images such as photographs and X-ray images, and one-dimensional data such as blood pressure or body temperature. In addition, multiple measurements of the same or different types can be combined and evaluated together. Moreover, measurements can be preprocessed so that a model is trained using preprocessed data rather than raw data. For example, in the case of geographic atrophy, the OCT dataset can be transformed into a number of layer thicknesses.

[0032] It will be understood that the techniques disclosed herein can be applied to N-dimensional input datasets, such as N-dimensional images, where N can be 1, 2, 3, or any positive integer value. An example of a suitable 3D image dataset is an MRI volume scan.

[0033] In many diseases, the disease affects the condition of specific areas of the body. This can be detected using one of the many available imaging techniques. The area of ​​disease growth may not be uniform. As the disease progresses, the area of ​​disease growth may increase. However, the shape of the affected area may also change over time, making it difficult to predict which anatomical areas will be affected by the disease and when. The techniques disclosed herein can help address these concerns.

[0034] Well-known training techniques are based on curated data, where acquisitions occur at specific points in time, for example, at perfectly equal and regular intervals for each patient. However, this is impractical for most clinical applications, given that each patient may have different acquisition frequencies and, potentially, interruptions in visits. In addition, the results of the inference cannot be trusted when interested in predicting different time intervals beyond the time when the model was trained. The technique disclosed herein does not assume that acquisitions occur at uniform points in time, thus enabling far more flexible training and prediction. The method predicts a general growth pattern using a parametric function that encodes the course of the disease over time. This allows for extrapolation to estimate what the final state of a patient's visual system may be, enabling, for example, the design of a treatment plan.

[0035] In our method, the constraints on predicting future points in time at specific intervals are relaxed, and instead, a single continuous model of the progression of the predicted state is constructed. A detailed explanation discloses, but is not limited to, the techniques disclosed herein, as an example of map atrophy. This single continuous model can be provided, for example, in the form of a set of levels. More generally, the single continuous model can be provided in the form of parameters of a parametric time-dependent function. Only a few parameters are needed to represent a potentially relatively complex progression profile in this method.

[0036] For example, available input data can be incorporated into a single input tuple I, which can be a matrix or tensor, for example. A set of at least one set of parameters to be estimated can be represented by an output prediction G, which can be a vector or matrix, for example. The model to be trained is a mapping f that maps the input tuple I to the output prediction G. θ Represents mapping f θ This can be embodied, for example, as a neural network. One example of a neural network is a convolutional neural network. Instead of a neural network, it is possible to use a statistical model such as a linear regression model or a nonlinear regression model. The symbol θ represents a model parameter, such as the configuration of neurons in the neural network, which can be updated during the training procedure.

[0037] In certain embodiments, for example, in the case of a monotonically progressive disease or condition, the output prediction G can be represented by a single numerical value. The result is that G can predict all future growth of the affected area, for example, the GA area, at the corresponding time point t. i Threshold T for i By finding the value of G, the threshold T i It is possible to determine the extent of the affected area through testing whether it is larger or smaller. This makes our approach far more flexible compared to existing techniques that are trained to predict atrophy only at specified time intervals (e.g., 6 or 12 months after the last measurement). To predict the progression profile, it is possible to define the progression of the affected area at any given future point in time.

[0038] In the case of a domain such as GA, the output prediction G may include a single numerical value for each pixel (or voxel) that you want to predict. i and each corresponding threshold T i Regarding the value of G > T i The pixels / voxels such that G≦Ti Separate images can be generated by setting different values or colors for pixels / voxels that result in different values or colors.

[0039] In certain embodiments, the output prediction G can include at least one parameter of a parametric function. An example of a suitable parametric function is a Taylor sequence with N terms. [[ID=*]]

Number

[0040] In this case, the parameters are a n (where n is all integers from 0 to N - 1 including N - 1) and b. The result is the sum of N + 1 parameters. Thus, the model can be trained to output the parameters a n (where n is all integers from 0 to N - 1 including N - 1) and b when the input tuple I is applied to the model. In certain embodiments, the value b can be predefined in advance. For example, b can be fixed to a specific value such as zero so that, particularly when the data is normalized, time point 0 coincides with data acquisition. In such a case, the model only needs to predict N parameters a n (where n is all integers from 0 to N - 1 including N - 1). Thus, for example, the model will output a tuple (a0, a1, a2, ···, a N-1 ). For example, N can be an integer within the range from 1 to 20, preferably within the range from 2 to 20, more preferably within the range from 2 to 10, and even more preferably within the range from 2 to 5.

[0041] Another example of a parametric function is a Fourier series with 2N + 1 terms determined over a period P:

Number

[0042] In this, the parameter is a n (n is all integers from 0 to n, including N) and b n (where n is all integers from 1 to n, including N). Therefore, in this case, for example, the model is a tuple (a0, a1, a2, ..., a N ;b1,b2,···,b N This will output ). Here again, for example, N can be an integer in the range of 1 to 20, preferably in the range of 2 to 20, more preferably in the range of 2 to 10, and even more preferably in the range of 2 to 5.

[0043] Another example of a parametric function is a Fourier series expressed in amplitude-phase form with 2N+1 terms:

number

[0044] In this case, the parameter is A n (where n is all integers from 0 to n, including N) and φ n (where n is all integers from 1 to n, including N). Therefore, in this case, for example, the model is a tuple (A0, A1, A2, ..., A N ;φ1,φ2,···,φ N This will output ). Here again, for example, N can be an integer in the range of 1 to 20, preferably in the range of 2 to 20, more preferably in the range of 2 to 10, and even more preferably in the range of 2 to 5.

[0045] The choice of parametric time-dependent function can be selected in terms of the typical progression of the state to be predicted in a larger population.

[0046] In certain embodiments, a model (neural network, statistical model) can be trained on processed data. In other words, the model's input can be preprocessed data, such as segmented or normalized data, rather than raw sensor data. In addition, the output may not be the actual parameters of a parameterized function, but post-processing operations such as normalization can be performed to take into account different time intervals or different amplitudes, for example.

[0047] While the examples in this disclosure highlight the medical application area, it will be understood that this technology can also be applied to other time-dependent applications such as mechanical component failure, non-destructive testing, non-medical imaging, or weather forecasting. Measurement data can be acquired, for example, using X-rays, photographs, radar, or other sensor technologies. The conditions to be predicted may include, for example, the predicted time of failure of a component, or the predicted efficiency of a machine or component expressed as a function of time. The term “subject” may, as used herein, refer to, for example, an object or a patient.

[0048] Figure 1 illustrates a method for predicting the progression of a condition. In certain embodiments, this method may be implemented by a computer. In step 101, measurement data related to at least one measurement of the object is acquired. The measurement data may be acquired, for example, from a medical imaging method. The measurement data may be associated with a measurement at one time, or with measurements taken at several different times. The most recent measurement is taken at a specific point in time. The data is generated based on the output of a sensor configured to perform at least one measurement of the object. For example, the output signal of a sensor is sampled and processed to acquire subjective measurement data. For example, data from an optical sensor is combined, and optical coherence tomography is used to form a three-dimensional volume. The data can then be processed in a way that assists in predicting the condition. In the case of geographic atrophy, this may include the identification of several layer thicknesses. In other applications, but not limited to these, appropriate preprocessing may include segmentation, noise reduction, edge enhancement, normalization, and similar processes.

[0049] In step 102, the measured data is used to generate at least one parameter of a parameterized time-dependent function. For this purpose, the measured data can be further preprocessed to make it suitable input to the model, which in certain embodiments may be a neural network. Since this optional preprocessing can be application-specific, further details of it are omitted in this disclosure. The input to the model is determined based on the measured data and input to the model. The model is configured to produce an output in response to its input. The output of the model may contain at least one parameter of a parametric time-dependent function. Optionally, the output of the model is processed to obtain that at least one parameter. For example, the output of the model may be normalized. Other optional post-processing operations of this kind may include, but are not limited to, scaling operations and combination operations that combine multiple outputs of the model into a single parameter.

[0050] For example, a parameterized time-dependent function depends on continuous time values.

[0051] In step 103, the parameterized time-dependent function can be evaluated at any desired point in time when the parameterized time-dependent function is effective. This method allows for prediction of future states. In certain embodiments, the evaluation may involve comparing the parameter with a time-dependent threshold. Using each time-dependent threshold, it is possible to create a segment of the parameter. Thus, it is possible to generate a segment for any desired point in time.

[0052] In certain embodiments, the evaluation of a parameterized time-dependent function at a given time t can be computed by inserting at least one parameter and time t into an equation (which may, for example, be based on a Taylor or Fourier series).

[0053] In a particular embodiment, parameters are generated using a model for each of the multiple pixels (e.g., pixels or voxels) in an image dataset. The value of each pixel can be evaluated using a time-dependent function parameterized for any time value t. Thus, for each time t, it is possible to generate an image using the value of each pixel.

[0054] In certain embodiments, a parameterized time-dependent function can have multiple arguments. In such cases, the function g can be written, for example, as g(t,x), where g represents a parameterized time-dependent function that depends on time t and at least one other argument x. Thus, a parameterized time-dependent function can depend on time and at least one further argument. For example, a parameterized time-dependent function can further depend on the amount of time until the next treatment, expressed in months. Changing the treatment time will then change the output of the parameterized time-dependent function. If the disease is treated early, the disease may progress less or more slowly, but if the disease is treated late, the treatment may become less effective, resulting in a larger or faster disease progression. This means that the time-dependent function may depend on at least one further argument in addition to the time at which the prediction is made. One example of such a further argument is the time at which a specific event, such as treatment, will occur.

[0055] Figure 2 illustrates a method for training a model to predict the progression of a state. For example, this method can begin with an initial model. The initial model can be set, for example, using random model parameters or based on prior knowledge. The model can include, for example, a neural network or a statistical model.

[0056] In step 201, training data is acquired. The training data may include measurements relating to at least one measurement for at least one object. For example, one measurement may be provided for each of multiple objects. Alternatively, a set of measurements taken over time may be provided for each of multiple objects. In certain embodiments, the number of measurements may differ for each object. The measurements may be similar to the measurement data used in the method of Figure 1.

[0057] The measurement data is based on the output of a sensor configured to perform at least one measurement on the subject. For example, the data can be generated by an imaging device used to image a part of a patient.

[0058] The training data can include, for each object, information representing at least one time point associated with that object and the state of the object at that time point. This information can be used as ground truth data. The model is trained in this way and outputs similar information in response to the measured data of the objects. Based on this, training data can be generated in the form of inputs and outputs that demonstrate the desired behavior of the model.

[0059] In certain embodiments, information indicating the state of an object at at least one point in time is determined based on further measurements taken with respect to the object at that point in time. For example, if measurement data is available for a first point in time and a second point in time that follows, the measurement data for the second point in time can be used to determine information indicating the state of the object at the second point in time (which should be predicted by the model).

[0060] In certain embodiments, information indicating the state of the object at a second time point is automatically extracted from measurement data at the second time point, for example, using a computerized algorithm. In other embodiments, information indicating the state of the object at a second time point is manually created by a human expert based on measurement data at the second time point. In yet another embodiment, the measurement data itself can be used to predict the state to be predicted.

[0061] In a particular embodiment, training pairs are generated, each training pair containing measurement data relating to an object at a first time point and information about the object's state at a second time point, where the second time point is later in time than the first. The measurement data at the first time point can be used to generate the model's input, and the information about the state at the second time point can be used to evaluate the model's performance and improve it.

[0062] In step 202, the model is used to generate at least one parameter of a time-dependent function, which is parameterized based on measurement data of a specific object among at least one of the objects. As described above, the measurement data can be preprocessed to obtain the corresponding input to the model. The parameterized time-dependent function represents the predicted state of the progression of the specific object over time.

[0063] In step 203, for at least one time point associated with a particular object, a parameterized time-dependent function is evaluated using at least one parameter to obtain the predicted state of that object at that time point. This step provides data suitable for comparison with the ground truth state of the object stored in the training data.

[0064] In step 204, the predicted state of a particular object at at least one point in time associated with that object is compared with information in the training data that indicates the state of that particular object at at least one point in time associated with that object. This comparison step can yield comparison results such as an error measure or goodness of fit.

[0065] One of the advantages of the techniques disclosed herein is the possibility of using uncurated data, in the sense that the time intervals that make ground truth information about the state available do not need to be the same for all objects. For example, a first object can have a first set of time points associated with it, and information indicating the state of the object at each of those first time points. A second object can have a second set of time points associated with it, and information indicating the state of the object at each of those second time points. In uncurated data, it is possible that at least one time point in the first set is not present in the second set. In addition, it is even possible that a time point associated with a first object is not associated with any other object. Ground truth information for any given time point can be compared to the model output obtained by evaluating a parametric time-dependent function at that time point.

[0066] In step 205, the model is updated based on the comparison results. In a particular embodiment, the comparison results of multiple objects are combined to form a composite comparison result, and the model is updated based on that composite comparison result.

[0067] After the model is updated, step 206 determines whether training is complete. If training is not yet complete, the process restarts from step 201 using the updated model as the current model. If step 206 determines that the model is complete, the process ends in step 207. Subsequently, the process in Figure 1 can be performed using the final model of the method in Figure 2.

[0068] Figure 3 shows a device 300 for predicting the progression of a state. In certain embodiments, the device 300 can also be configured to train a model to predict the progression of a state. The device 300 includes an input 301. The input 301 may include a communication port for receiving data, such as a network interface or a data bus. The device 300 may further include a storage medium 302. The storage medium 302 may include computer memory. The storage medium 302 may also include a non-transient computer-readable medium on which computer instructions are stored. When executed by a processor 303, the device can be configured to perform any step of the method shown herein. The storage medium 302 may also store models, such as neural network models. The storage medium 302 may also store measurement data and / or training data. The processor system 303 can control the device 300 by executing instructions in the storage 302. While the details of the training procedure can be embodied in different ways, it will be understood that at least one parameter is generated based on the model's output, and the model is updated based on the results of evaluating the parameterized time-dependent function at a specific point in time when ground truth data is available.

[0069] Specific embodiments are disclosed in more detail below. It will be understood that these detailed descriptions of embodiments are intended to serve as an explanation rather than limit the scope of protection.

[0070] Geographic atrophy (GA) is a disease affecting the visual system, characterized by the loss of photoreceptor cells in the retina. GA can have different functional outcomes depending on the location: when the central (fovea) part of the retina is affected, the degree of visual impairment is greater than when the peripheral areas are damaged. For this reason, it is crucial to be able to predict the direction in which the disease is likely to propagate and the rate of its diffusion. Current approaches to predicting GA progression only address the next (arbitrarily selected or imposed) point in time, requiring highly curated data at a fixed point in time. Due to the nature of patient visits, obtaining this type of data in a real-world clinical setting is often difficult, even when patients are following treatment protocols. These scenarios hinder the development of new disease progression strategies. Here, we present a novel approach to predict continuous GA growth based on single-scan layer thickness. The method is framed as a problem of a set of levels that does not require uniform acquisition intervals, modeling the complete progression pattern of a patient's GA.

[0071] 1. Introduction GA is a degenerative disease of the retina, an advanced form of age-related macular degeneration (AMD) that causes irreversible damage to photoreceptors (PRs). This subsequently leads to vision loss, manifesting as dark spots in the patient's visual field. Depending on the location within the retina, GA can have different effects on vision. It is an incurable chronic disease, and therefore physicians focus on slowing disease progression. The mechanisms of progression are not fully understood, and ongoing research is also directed in that direction (Non-Patent Literature 6). Automated detection and progression prediction algorithms can reduce the workload associated with image analysis and provide insights into contributing factors. Existing approaches to GA progression are primarily based on optical coherence tomography (OCT), which enables high-resolution three-dimensional visualization of retinal structures. Non-Patent Literature 10 describes a method for predicting the probability of GA growth based on diverse retinal thicknesses and projection images using random forest (RF). This uses time-weighted thresholds to predict GA areas at future points in time. Non-patent document 14 describes training a model to segment a GA collaboratively and using a deep neural network to predict segmentation at future points in time. Non-patent document 3 describes predicting the progression of diabetic retinopathy by applying deep learning (DL) to color photographs of the fundus.

[0072] Furthermore, disease progression prediction is performed at different levels of granularity in various fields beyond ophthalmology. Brain tumor growth has been modeled probabilistically using biological models (Non-Patent Literature 5, 8), or more recently, using probabilistic deep modeling (Non-Patent Literature 11). In this, Petersen et al. have proposed trained distributions of growth trajectories that seem reasonable, based on the premise that tumor growth behavior differs from patient to patient. All of the above methods are trained using curated data acquired continuously at specific points in time. However, this is not practical for most clinical applications, given that each patient may have different acquisition frequencies and, as is possible, interruptions in hospital visits. In addition, if we are interested in predicting different time intervals beyond the original time when the model was trained, the results of the inference cannot be trusted. In contrast, the method presented here does not require that acquisitions occur at uniform time points, thus enabling far more flexible training and prediction. The method predicts a general growth pattern that shows the retinal regions most prone to progression. By setting a threshold for the predicted disease profile, a GA prediction at future time points is obtained. This allows clinicians to extrapolate and estimate what the final state of a patient's visual system might be, and then design a treatment plan accordingly.

[0073] 2. Method A 3D OCT volume of size D×H×W acquired at time t0.

number

[0074] Previous literature has shown that information from the retinal layers is particularly important for atrophy detection (Non-Patent Documents 9, 14). Therefore, we first perform layer and fluid segmentation using the algorithm from Non-Patent Document 2. An example of the output is shown in Figure 4B. GA is the loss of PR cells, but adjacent retinal layers are also affected by this cell death. Therefore, we calculate the two-dimensional ANFAS thickness measure corresponding to each segmented layer (as illustrated in Figure 5): - Retinal nerve fiber layer (RNFL) - Ganglion cell layer (GCL) and internal plexiform layer (IPL) - Inner granular layer (INL) and outer plexiform layer (OPL) - Outer nuclear layer (ONL) - Photoreceptors (PR) and retinal pigment epithelium (RPE) - Choroidal capillaries (CC) and choroidal stromas (CS) The thickness of the three fluid types is also similar: - Intraretinal fluid (IRF) - Subretinal fluid (SRF) - Retinal pigment epithelial detachment (PED).

[0075] Figure 5 shows an example of an input thickness map for a network. In this regard, Figure 5(a) shows RNFL, Figure 5(b) shows GCL+IPL, Figure 5(c) shows INL+OPL, Figure 5(d) shows ONL, Figure 5(e) shows PR+RPE, Figure 5(f) shows CC+CS, Figure 5(g) shows IRF, Figure 5(h) shows SRF, Figure 5(i) shows PED, and Figure 5(j) shows manual segmentation of atrophy. These figures correspond to the unfathomable projection of the OCT volume.

[0076] All these thickness maps are a single input tensor

number

number

number

[0077] The higher the value of a given pixel in the map, the higher the probability that the corresponding area is a genetic algorithm (GA) or will develop into one.

number

[0078] The result was,

number

number

[0079] 2.1. Training The data is annotated by experienced clinicians, with bounding boxes drawn around the atrophic areas within each OCT volume slice (B-scan); an example is shown in Figure 4A. Although only training is performed on projection images, even this makes the annotations within the B-scans more accurate. Ground truth is then calculated as a two-dimensional unfathomable projection of the atrophy. Our dataset consists of 10 patients with multiple OCT acquisitions over time; however, these acquisitions do not need to be evenly spaced in time.

[0080] Figure 4A shows an example of a B-scan with a GA annotation. Figure 4B shows a B-scan with segmented layers and PED fluid 401.

[0081] Figure 6 shows an overview of an example training workflow according to an embodiment of this disclosure. The figure illustrates the following steps from left to right: input image for time t1

number

number

number

number

number

number

number

[0082] The convolutional neural network (CNN) f defined above θ To enable the discovery of the parameter θ, we introduce a new training procedure illustrated in Figure 6. For each patient, at time t a ≠t b and t a <t b Regarding all possible pairs of acquisitions P a,b Extract each pair P. a,b Regarding the first thickness map I from the previous visit a Based on this, predict f θ (I a ) calculates I a Ground Truth Y corresponding to a , and I b The corresponding future atrophy state Y b For this, prediction matching is performed twice. The matching optimizes the dice similarity coefficient Y. a and Y b For each

number

number

[0083] Threshold T b and T a The threshold was left unbounded in the sense that the threshold that best approximates the current GA is calculated each time. Each unique threshold represents a set of levels, and by selecting the best one, it defines the region that best fits ground truth GA. The threshold was chosen to be unbounded because the absolute rate of GA progression is highly patient-dependent in terms of genetic predisposition and lifestyle (Patent Documents 4, 6). In addition,

number

[0084] 2.2. Reasoning During inference, only the data obtained at time t0 is used for prediction.

number

[0085] 2.3. Details of the implementation The following provides details of an embodiment as a non-limiting example. Here, a standard encoder / decoder-style network is embodied with four downsampling steps and 16, 32, 64, 128, and 256 filters at each level (Non-Patent Literature 12). Optimization is performed using the aforementioned loss and an Adam optimizer with a learning rate of 0.0001 (Non-Patent Literature 7). The original thickness map is scaled to a size of 150x150. Due to the limited training data, dynamic data augmentation is performed to be consistent with natural image changes. This includes image rotation (-14.3, +14.3 degrees) and translation (-1%, +1% of image size on the x and y axes). Left / right swapping is also used, given that the left and right eyes are horizontal mirror images.

[0086] 3. Results 3.1. Data and Baselines We used a dataset of 10 patients, each with 3 to 5 acquired data points at various time intervals, resulting in a total of 51 OCT volumes, imaged using a Heidelberg Spectralis OCT machine. Intervals varied from 5 to 22 months. However, as will be shown later, our flexible training procedure does not require fixed time intervals. The dataset was manually annotated by experts for each time-point slice. Atrophy was selected using boundary boxes enclosing horizontal regions (incomplete RPE and lateral retinal atrophy (iRORA) and complete RPE and lateral retinal atrophy (cRORA) regions) (Non-Patent Literature 13) (see Figure 4A). Ground truth was obtained by projecting these boundary boxes into two dimensions (Figure 5(j)). To fully utilize the limited number of patients, we performed one-out cross-validation at the patient level, excluding one patient for the trial at each fold. We compare our method with two baselines:

[0087] 1. Our method: Proposed method (see Section 2)

[0088] 2. Present only: During training, the network is trained using only current thickness images and ground truth, without access to future ground truth. This shows how well the current atrophy shape indicates future progression.

[0089] 3. Non-patent document 14 (Zhang et al.): This method predicts current and future atrophy in conjunction. In contrast to our method, this method is trained only for a specific time interval (next acquisition). Progressive atrophy is calculated using the prediction of current atrophy and features of low-level images. Current and future atrophy are returned on two separate channels.

[0090] 3.2. Discussion Figure 7A shows the dice scores (on the vertical axis 'D') of the evaluated methods for predicting progress over time intervals expressed in months (on the horizontal axis 't') ("We" 701, "Currently Only" 702, "Zhang et al." 703). Figure 7B shows the area difference scores (on the vertical axis 'AD') in pixels for the evaluated methods for predicting progress over time intervals expressed in months. A higher dice score and a lower area difference indicate better performance.

[0091] As can be seen from Figure 7A, our method outperformed the baseline across all time intervals, with the most significant difference observed in long-term predictions (>15 months). 'Current Only' and 'Zhang et al.' performed similarly in short time intervals, but their performance declined over time. This suggests that the current atrophy prediction (current only) is a good indicator of progress in the short term, even when not trained with future data, because small changes leading to GA are already detectable at time t0. 'Zhang et al.' outperformed 'Current Only' over longer intervals, but did not achieve the same performance as 'ours'.

[0092] Figure 8 shows an overview of the results for a single patient as an example. Column 1 I i This shows (unaligned) fundus images corresponding to all captured points in time. Column Y i The correspondence with the one inside is the ground truth segment derived from OCT. Third column f(I i ) is input I i This shows the predicted level set based on [the given data]. The right-hand portion of this figure shows the optimal level set for a specific point in time.

[0093] Figure 8 illustrates the complete progression profile in a single patient case. The first row shows that our method is predictable for future GA, even over long intervals. Unlike the method of Zhang et al., which optimizes the network to fit discrete atrophy progression, our level-set formula ensures that the progression pattern is smooth and continuous. The accuracy achieved by our method over shorter time periods is lower than the two baselines, which is reflected in the larger area difference error for the shortest time intervals (Figure 7B). This is due to being trained to match progression over both short and long periods. In conclusion, it can be summarized that our method performs better than other methods over longer visit intervals.

[0094] 4. Conclusion In this disclosure, in particular, a novel method for learning to predict the progression profile of continuous GA is proposed. In contrast to previous approaches, our method is point-based, which does not need to be equally separated in time. It is possible to provide accurate estimates for future atrophy areas beyond the next subsequent point in time. Our method has been evaluated on a dataset including patients with uncurated visit intervals and has shown promising results toward real-world sparse acquisition schemes, predicting progression profiles beyond an accurate 15-month threshold.

[0095] Some or all aspects of the present invention may be suitable for implementation in the form of software, particularly computer program products. A computer program product may encompass a computer program stored on a non-transient computer-readable medium. Alternatively, a computer program may be represented by signals, such as optical or electromagnetic signals, carried by a transmission medium such as fiber optic cables or air. A computer program may have, in part or in whole, form source code, object code, or pseudocode suitable for execution by a computer system. For example, the code may be executable by one or more processors.

[0096] The examples and embodiments described herein are not intended to limit the invention but rather to illustrate it. Those skilled in the art will be able to design alternative embodiments without departing from the spirit and scope of this disclosure as defined by the accompanying claims and their equivalent concepts. Reference numerals in parentheses within the claims should not be construed as limiting the scope of the claims. Items described as separate entities in the claims or descriptions may be embodied as a single hardware or software item combining the features of the described items. [Explanation of Symbols]

[0097] 300 equipment 301 Input 302 Storage media, storage 303 Processor, Processor System 401 PED fluid 601 Result Image 602 Result Image

Claims

1. A method for operating a device for predicting the progression of a state, The process of acquiring measurement data by a processor system relating to at least one measurement of an object up to a specific point in time, wherein the data is generated based on the output of a sensor configured to perform the at least one measurement of the object, and the measurement data comprises at least one N-dimensional input dataset of pixels or voxels associated with the object, generated by an imaging device, where N is a positive integer, and the N-dimensional input dataset is at least a two-dimensional image dataset associated with the object. The processor system generates, for each of the multiple pixels or voxels in the N-dimensional input dataset, a trained model, and generates at least one parameter of a parameterized time-dependent function based on the measurement data, wherein the parameterized time-dependent function depends on consecutive time values, and the parameterized time-dependent function separately indicates the progression of the predicted state of the object over time after a particular point in time for each of the multiple pixels or voxels. A method of including.

2. The method according to claim 1, further comprising the processor system evaluating the parameterized time-dependent function using the at least one parameter for at least one time point after the specific time point.

3. A method for operating a device for training a model to predict the progression of a state, The process involves obtaining training data by a processor system, which includes measurement data relating to at least one measurement of at least one object, wherein the measurement data is based on the output of a sensor configured to perform the at least one measurement of the object, and the training data further includes information indicating at least one time point associated with the object for each object, and the state of the object at at least one time point, wherein the measurement data includes at least one N-dimensional input dataset of pixels or voxels associated with the object, generated by an imaging device, where N is a positive integer, and the N-dimensional input dataset is at least a two-dimensional image dataset associated with the object. The processor system generates, for each of the multiple pixels or voxels in the N-dimensional input dataset, at least one parameter of a parameterized time-dependent function, using a model and based on the measurement data of a specific object among the at least one object, wherein the parameterized time-dependent function generates a parameter that depends on consecutive time values ​​and separately indicates the predicted progression of the state of the specific object over time for each of the multiple pixels or voxels. The processor system evaluates the parameterized time-dependent function using the at least one parameter for the at least one time point associated with the specific object, and obtains the predicted state of the object at the at least one time point. The processor system compares the predicted state of the specific object at at least one time point associated with the specific object with the information in the training data indicating the state of the specific object at at least one time point associated with the specific object, and obtains a comparison result. The processor system updates the model based on the comparison results, A method of including.

4. At least one first object among the at least one object has at least one time point in a first set associated with it, At least one second object among the aforementioned at least one object has at least one time point in the second set associated with it, At least one time point in the first set is different from each time point in the second set. The method according to claim 3.

5. The method according to any one of claims 1 to 4, wherein the at least one parameter indicates the time at which the state changes, or the rate at which the state changes.

6. The method of claim 5, wherein evaluating the parameterized time-dependent function involves the processor system applying a threshold to the parameters generated using the model, the threshold depending on the time point in which the parameterized time-dependent function is evaluated.

7. The method according to any one of claims 1 to 6, wherein the at least one parameter comprises at least one coefficient of the term of the parameterized time-dependent function.

8. The method according to claim 7, wherein the parameterized time-dependent function includes a Fourier series or a Taylor series.

9. The method according to any one of claims 1 to 8, wherein the model includes a convolutional neural network.

10. A device for predicting the progression of a state, An input configured to receive measurement data relating to at least one measurement of an object up to a specific point in time, wherein the measurement data is generated based on the output of a sensor configured to perform the at least one measurement of the object, and the measurement data comprises at least one N-dimensional input dataset of pixels or voxels associated with the object, generated by an imaging device, where N is a positive integer value, and the N-dimensional input dataset is at least a two-dimensional image dataset associated with the object. Non-transient storage media encompassing the model, A processor system configured to cause the device to generate, using the model and based on the measurement data, at least one parameter of a parameterized time-dependent function for each of the plurality of pixels or voxels in the N-dimensional input dataset, wherein the parameterized time-dependent function depends on consecutive time values ​​and separately indicates the progression of the predicted state of the object over time after a particular point in time for each of the plurality of pixels or voxels; A device that includes a certain feature.

11. A device for training a model to predict the progression of a state, An input configured to receive training data comprising measurement data relating to at least one measurement of at least one object, wherein the measurement data is based on the output of a sensor configured to perform the at least one measurement of the object, and the training data further comprises information indicating at least one time point associated with the object for each object, and the state of the object at at least one time point, wherein the measurement data comprises at least one N-dimensional input dataset of pixels or voxels associated with the object, generated by an imaging device, where N is a positive integer, and the N-dimensional input dataset is at least a two-dimensional image dataset associated with the object, A storage medium for storing the model, A processor system, wherein the device includes, For each of the plurality of pixels or voxels in the N-dimensional input dataset, generate at least one parameter of a parameterized time-dependent function using the model and based on the measurement data of a specific object among the at least one object, wherein the parameterized time-dependent function depends on consecutive time values, and the parameterized time-dependent function separately indicates the predicted progression of the state of the specific object over time for each of the plurality of pixels or voxels. The parameterized time-dependent function is evaluated using the at least one parameter for the at least one time point associated with the specific object, and the predicted state of the object at the at least one time point is obtained. The process involves comparing the predicted state of the specific object at at least one point in time associated with the specific object with the information in the training data that indicates the state of the specific object at at least one point in time associated with the specific object, and obtaining a comparison result. The model is updated based on the comparison results, A processor system configured to perform the following: A device that includes a certain feature.

Images