Correcting machine learning model training data

Abstract

Thus, the proposed concepts are directed to providing schemes, solutions, concepts, designs, methods and systems relating to improving (i.e., improving accuracy and / or reliability) a machine learning model by correcting data used to train such a model. In particular, timestamps of training data describing events are modified according to a time shifting function and a predetermined range of temporal uncertainty. In this manner, the uncertainty / inaccuracy of the recording of timestamps can be compensated, so that the quality of the training data can be improved.

Description

Technical Field

[0001] The present invention relates to the field of training machine learning models, and in particular to the field of correcting data used to train machine learning models. Background Art

[0002] Event data, which are data related to the occurrence of an event, are often recorded manually. As a result, the timestamp of the event described by the event data is very easy to be inaccurate due to recording / human error and / or because of the inaccuracy of the clock used to reference the time of the event.

[0003] In many fields, event data is used to train machine learning (i.e., artificial intelligence) models. Since the quality of machine learning models is highly dependent on the quality of the data used to train them, the inaccuracies associated with recorded event data are problematic.

[0004] By way of specific example, in the medical field, machine learning models can be used to predict the future condition of a subject. This can be particularly beneficial during surgical procedures, where such predictions can be important for subject safety. However, since it is not uncommon for timestamps for event data (i.e., drug administration events, subject repositioning events, etc.) to be recorded incorrectly, the accuracy and reliability of such machine learning models can be reduced. Ultimately, this can threaten the safety of the subject.

[0005] Therefore, there is a need for a means to mitigate the impact of such inaccuracies in event data used to train machine learning models. Summary of the invention

[0006] The invention is defined by the claims.

[0007] According to an example according to one aspect of the present invention, a method for correcting machine learning model training data is provided, the method comprising:

[0008] obtaining training data including timestamp values ​​describing timings of occurrences of events; and

[0009] The timestamp values ​​of the obtained training data are modified according to a time shift function configured to adjust the timestamp values ​​based on a predetermined temporal uncertainty range.

[0010] Therefore, the proposed concepts aim to provide schemes, solutions, concepts, designs, methods and systems related to improving (i.e., increasing the accuracy and / or reliability) machine learning models by correcting the data used to train such models. In particular, the timestamps of the training data describing events are modified according to a time shift function and a predetermined time uncertainty range. In this way, the uncertainty / inaccuracy of the recording of the timestamps can be compensated so that the quality of the training data can be improved.

[0011] It is known that data describing events often have unreliable (i.e., inaccurate, imprecise, etc.) timestamps. This is because in such cases timestamps are usually recorded by individuals who are susceptible to errors from various sources. Therefore, when such unreliable data is used to train a machine learning model, the quality of the model may be compromised.

[0012] Therefore, embodiments of the present invention are intended to overcome these problems by correcting / modifying the data used to train the machine learning model. This is achieved by adjusting the original timestamp value by a time shift function that takes into account the uncertainty of the timestamp value. In some embodiments, the timestamp value can be shifted by a random amount within a predetermined time uncertainty range. When this modification of the timestamp value is applied to training data that describes the timing of many events occurring, the effects of any errors in the individual timestamp values ​​used to train the machine learning model can be suppressed.

[0013] In other words, by modifying the timestamp values ​​in this way, the resolution of the timestamp values ​​for all occurrences of the event may be reduced. This reduction in resolution may correspond to uncertainty, thereby offsetting the effect of errors in recording the timestamps.

[0014] In many cases, events are often triggers for subsequent changes in a system. By way of example, a drug administration event will typically trigger a change in the physiological state of a subject. Accurately understanding the timing of such events is critical to understanding the causal relationship between the occurrence of the event and the impact of the event. However, in many environments, accurate recording of the timing of such events may not be a priority, or may not be possible. Therefore, when data collected around such events is used to train a machine learning model, the machine learning model may have an incorrect understanding of the causal relationship between the event and subsequent system changes. Therefore, by accounting for these inconsistencies in the recorded timestamps via a time shift function and a predetermined time uncertainty range, the machine learning model training data can be improved.

[0015] Thus, in the present invention, the timestamp value describing the timing of the occurrence of an event is modified / shifted / changed - reducing the effective precision of the timestamp, but increasing the accuracy of the timestamp.

[0016] In some embodiments, the predetermined time uncertainty range may indicate a predicted difference between the timestamp value and the actual timing of the event occurrence.

[0017] Preferably, the modification to the timestamp value should not be greater than the difference between the timing of the recorded event and the true timing of the event. This ensures that the effective precision of the timestamp is not reduced more than is necessary to improve the accuracy of the timestamp.

[0018] In some embodiments, the predetermined time uncertainty range may be based on the event type corresponding to the event occurrence.

[0019] In practice, the uncertainty associated with a timestamp is often highly dependent on the type of event that the timestamp describes. For example, timestamps for events where it is difficult for the recorder to discern the precise timing of the event may have higher levels of uncertainty. Furthermore, uncertainty may be higher for events that inherently require the timestamp to be recorded retrospectively (i.e., when the recorder is involved in the actuation of the event, or in situations where the recorder cannot have any recording device handheld).

[0020] In other words, information about the event type can be utilized in order to determine possible errors in the recording of the timestamp. Thus, utilizing this information can achieve a more appropriate predetermined time uncertainty range for modifying the timestamp value based thereon.

[0021] In some embodiments, the time shift function may be configured to adjust the timestamp value based on a predetermined time uncertainty range and a probability distribution algorithm.

[0022] Therefore, the modification of the timestamp value can be appropriately performed. In this case, when the modification is applied to many timestamp values, the average accuracy can be further improved.

[0023] In some embodiments, the probability distribution algorithm may follow a uniform distribution. In other embodiments, the probability distribution algorithm may follow a normal distribution. In further embodiments, the probability distribution algorithm may follow an asymmetric probability distribution, and preferably follows a lognormal distribution.

[0024] Different types of probability distribution algorithms may be more suitable for different types of use cases. This may depend on the type of event being described or the preferences of the user who wants to correct the training data.

[0025] According to another aspect of the present invention, there is provided a method for generating a state prediction model, the state prediction model being suitable for outputting a state prediction indicating a future physiological state of an object, the method comprising: obtaining time series data, the time series data comprising state data describing at least one physiological characteristic; and event data, the event data comprising a timestamp value describing the timing of an event occurrence; correcting the event data according to a method for correcting machine learning model training data; and training the state prediction model using a training algorithm, the training algorithm being configured to receive an array of training inputs and known outputs, wherein the training inputs comprise corrected event data and state data, and the known outputs comprise the state data.

[0026] According to other aspects of the present invention, a method for generating a state prediction model is provided, wherein the state prediction model is suitable for outputting a state prediction indicating a future physiological state of an object, the method comprising: obtaining time series data, which includes state data describing at least one physiological characteristic, and event data including timestamp values ​​describing the timing of event occurrence; correcting the event data by modifying at least one of the timestamp values ​​of the event data according to a time-shift function, the time-shift function being configured to adjust the timestamp value based on a predetermined time uncertainty range; and training the state prediction model using a training algorithm, the training algorithm being configured to receive an array of training inputs and known outputs, wherein the training inputs include corrected event data and state data, and the known outputs include the state data.

[0027] Therefore, the above-mentioned method of correcting / modifying training data can be used to train a state prediction machine learning model so that the state prediction model can output more accurate and / or reliable predictions related to the future physiological state of the object.

[0028] In fact, the future state of the object depends heavily on the events that occur for the object (i.e., drug administration events, treatment events, repositioning events) and the current state of the object. The state of the object is usually recorded by sensors (i.e., vital signs monitors), which inherently means that the timestamp is accurate. However, event data is usually recorded by a care provider who can record the time of the event retrospectively and is in a high-pressure environment where errors may be common. Therefore, the timestamp value of the event related to the object may be very inaccurate.

[0029] Therefore, it is often the case that a state prediction model is trained on a data set containing many errors, resulting in inaccurate and unreliable output predictions. Embodiments of the present invention are directed to alleviating this problem by modifying timestamps according to a time-shift function configured to adjust timestamp values ​​based on a predetermined time uncertainty range. Thus, an improved (i.e., more accurate and reliable) state prediction model can be trained.

[0030] In some embodiments, the state data may include vital sign data. The state data may also include at least one of heart rate, blood pressure, and oxygen saturation level. Such data may be automatically acquired by a sensor attached to the object. Therefore, the timestamp of the state data may be considered to be close to the real situation timing.

[0031] In some embodiments, the event data may include intervention information describing the treatment of the subject.The event data may include at least one of a medication administration event, a movement event, and a treatment event.

[0032] Such data may correspond to inaccurate timestamp values ​​due to human error. However, such described events may also have a significant impact on the state of the object. Therefore, accurate timestamp values ​​are needed to properly evaluate the link between the event and the state of the object, as well as to predict future states.

[0033] In some embodiments, the training algorithm is a stochastic gradient descent algorithm.

[0034] According to another aspect of the present invention, there is provided a method for generating a state prediction indicating a future physiological state of an object, the method comprising: generating a state prediction model according to a method for generating a state prediction model suitable for outputting a state prediction indicating a future physiological state of the object; obtaining time series data associated with the object, the time series data comprising state data describing at least one physiological characteristic of the object; and event data comprising a timestamp value describing the timing of an event occurring corresponding to the object; and obtaining a state prediction of the object based on inputting the time series data into the generated state prediction model.

[0035] Thus, by generating a state prediction model using corrected training data as described above, improved (i.e., more accurate, precise, and reliable) predictions of subject state can be obtained. This can have a significant positive impact on subject outcomes, since predicting future state is key to determining appropriate steps in subject care.

[0036] According to another aspect of the present invention, there is provided a computer program comprising a computer program code module, which, when the computer program is run on a computer, is suitable for implementing a method for correcting machine learning model training data, generating a state prediction model suitable for outputting a state prediction indicating a future physiological state of an object, and generating a state prediction indicating a future physiological state of an object.

[0037] According to an additional aspect of the invention, a system for correcting machine learning model training data is provided, the system comprising: an interface configured to obtain training data including timestamp values ​​describing the timing of occurrence of events; and a data manipulation unit configured to modify the timestamp values ​​of the obtained training data according to a time-shift function configured to adjust the timestamp values ​​based on a predetermined time uncertainty range.

[0038] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] For a better understanding of the invention and in order to more clearly show how it may be implemented, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0040] Figure 1 A flow chart of a method for correcting machine learning model training data according to an embodiment of the present invention is presented;

[0041] Figure 2 A flow chart of a method for generating a state prediction model adapted to output a state prediction indicative of a future physiological state of a subject according to another embodiment of the present invention is presented;

[0042] Figure 3 A flow chart of a method for generating a state prediction indicative of a future physiological state of a subject according to another embodiment is presented;

[0043] Figures 4A-4C depicts a diagram of correction of timestamps of event data in time series data according to aspects of an embodiment of the present invention;

[0044] Figure 5 A simplified block diagram of a system for correcting machine learning model training data according to an embodiment is presented; and

[0045] Figure 6 is a simplified block diagram of a computer within which one or more portions of the embodiments may be employed. DETAILED DESCRIPTION

[0046] The present invention will be described with reference to the accompanying drawings.

[0047] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, system and method, are intended only for illustrative purposes and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the apparatus, system and method of the present invention will be better understood from the following description, the appended claims and the accompanying drawings. Although specific measures are recited in mutually different dependent claims, this does not indicate that a combination of these measures cannot be used advantageously.

[0048] By studying the drawings, the disclosure and the claims, those skilled in the art can understand and implement variations to the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality.

[0049] It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the various drawings to indicate the same or similar parts.

[0050] The present invention proposes a concept for achieving improvement (i.e., increasing accuracy and / or reliability) of a machine learning model by correcting the data used to train such a model. In particular, the timestamps of the training data describing an event are modified according to a time shift function and a predetermined time uncertainty range. In this way, the uncertainty / inaccuracy of the recording of the timestamps can be compensated so that the quality of the training data can be improved.

[0051] In practice, it has been recognized that the accuracy of the timestamp values ​​corresponding to (critical) event data can be increased by individually adjusting the timestamp values ​​randomly within a range of values. For example, while it may not be accurate that an event occurred at a particular time, it may be accurate that the event occurred within 10 minutes. Therefore, the present invention may result in an effective loss of precision of the timestamp value, but with improved accuracy. In other words, the resolution of the timestamp value is reduced in order to ensure accuracy.

[0052] Therefore, the present invention for correcting timestamps can be used in a variety of fields to improve machine learning models. The present invention can be applied to any training data set that utilizes timestamps, where the timestamps may be considered inaccurate, or the accuracy of the timestamps is uncertain. This may be particularly beneficial in the medical field, and more specifically for predicting the future physiological state of a subject based on current / past state data and event data. In fact, the timestamps of event data can be very inaccurate in such fields, so correction of the training data set can achieve improved (i.e., accurate and reliable) machine learning models.

[0053] Now turn to Figure 1 , depicts a flow chart of a method 100 for correcting / modifying / changing (i.e., improving accuracy) machine learning model training data according to an embodiment of the present invention. In short, the method 100 can receive data to be used to train a machine learning model and change the timestamps of the data so that the timestamps are accurate. This is achieved by reducing the overall accuracy of the timestamps of the data (i.e., reducing the resolution of the timestamps).

[0054] Specifically, training data including a timestamp value describing the timing of event occurrence is obtained in step 110. In other words, a timestamp value (ie, a date-time value) indicating when the event described by each training data points to is obtained / received.

[0055] In practice, the (recorded) timestamp values ​​may have been recorded by a human (i.e., not automatically), and therefore may have inaccuracies. In other embodiments, the timestamp values ​​may have inaccuracies due to clock differences in the sensors used to detect the occurrence of the event. Some timestamp values ​​of the training data may be accurate (i.e., identical to the real situation). However, in general, the timestamp values ​​of the training data have an associated uncertainty as to the accuracy of the values.

[0056] The training data(s) may be obtained from a database, or may be received when recording it. Alternatively, the training data may be retrieved from a corpus of information collected from various sources.

[0057] At step 120, the timestamp values ​​of the obtained training data are modified according to a time shift function configured to adjust the timestamp values ​​based on a predetermined time uncertainty range.

[0058] In other words, since timestamp values ​​have an overall level of uncertainty / inaccuracy associated with them (i.e., due to sources of inaccurate recording), they may be altered / changed / modified based on the overall level of uncertainty in an attempt to compensate for the overall level of uncertainty / inaccuracy. For example, it may be known that an event may have occurred within 10 minutes of the timestamp value (i.e., because it was a guess by the recorder). In this case, the timestamp value is modified taking into account this inaccuracy.

[0059] The time-shift function is a function that takes as input a (recorded) timestamp value and a predetermined time uncertainty range and outputs a modified / changed timestamp value. The modification may be deterministic (i.e., based on the timestamp value being fixed), or may be probabilistic (i.e., determined based on randomness within a variance range). In some embodiments, the training data with the corresponding modified timestamp values ​​may then be used to train a machine learning model.

[0060] In some embodiments, the time shift function can be configured to adjust / modify the timestamp value based on a predetermined time uncertainty range and a probability distribution algorithm. Thus, the timestamp value(s) can be changed according to an algorithm configured to distribute the timestamp value along a predetermined time uncertainty range. By way of example, one timestamp value can be moved forward by a few minutes, another timestamp value can be moved backward by a few minutes, and another timestamp value can not be changed at all.

[0061] More specifically, the probability distribution algorithm may follow one of a uniform distribution, a normal distribution, an asymmetric probability distribution, and a log-normal distribution. In fact, the probability distribution algorithm may follow any distribution suitable for randomly varying the timestamp within a range known to the skilled person. In each case, the distribution may take as input a value based on a predetermined time uncertainty range, and a timestamp value, to output a modified / corrected timestamp value.

[0062] Continuing, the predetermined time uncertainty range may be a fixed value set according to user preference. Alternatively, the predetermined time uncertainty range may be automatically set according to information about the timestamp and / or event.

[0063] Furthermore, the predetermined time uncertainty / inaccuracy range may indicate a predicted difference between the timestamp value and the actual timing of the event occurrence. In practice, the predetermined time uncertainty range may be considered as a range of values ​​for the possible actual timings at which the event described by the timestamp value has occurred.

[0064] In some embodiments, the predetermined time uncertainty range may be based on the event type corresponding to the event occurrence. In fact, the event type may be a factor in the uncertainty level associated with the timestamp. For example, when the event type necessarily means that the person recording the event cannot record a timestamp when the event occurs, this may indicate a higher level of uncertainty than the opposite case.

[0065] Although the following description will now be given about Figure 1 The invention described (correcting the timestamp value of the training data) is used to generate a state prediction model, but it should be understood that it can be applied to various fields. In fact, the present invention can be applied to any data set used to train a machine learning model, which has a timestamp that may be considered inaccurate.

[0066] Figure 2 2 is a flow chart of a method 200 for generating a state prediction model suitable for outputting a state prediction indicating a future physiological state of a subject according to an embodiment of the present invention. In other words, method 200 provides a machine learning model for generating a prediction related to a state of a subject in response to an input of a time series data of the subject.

[0067] Time series (training) data is obtained at step 210. The time series data includes state data describing at least one physiological characteristic (i.e., a characteristic of the subject indicating a biological attribute of the subject), and event data (i.e., data describing changes in the subject's environment) including timestamp values ​​describing the timing of event occurrences.

[0068] In other words, time series data describing an object over time is obtained / received. The data may be real-time, obtained directly from a sensor / at the time it is recorded, or may be received from a centralized storage device. The time series data may describe the state and events of one object over time, or may correspond to multiple different objects.

[0069] The state data may include vital sign data, including at least one of heart rate, blood pressure and oxygen saturation level. It should be noted that the skilled person will understand that the state data may include any physiological (ie biological) data that may be useful in determining the state of the subject.

[0070] Event data may include intervention information describing the subject's treatment, including at least one of a drug administration event, a movement event, and a treatment event. It should be noted that the skilled person will understand that event data may include any data related to a change in the subject's environment, which may have an impact on the subject's state.

[0071] The method 200 then moves to step 100, where the event data is corrected. This can be done according to the above reference Figure 1 The method 100 described herein is implemented. In any case, at least one of the timestamp values ​​of the event data is modified according to a time shift function configured to adjust the timestamp value based on a predetermined time uncertainty range. In this way, inaccuracies of the timestamps of the event data can be taken into account, thereby improving the quality of the event data.

[0072] At step 220, a state prediction model (i.e., a neural network-based machine learning model or an artificial intelligence model) is trained using a training algorithm configured to receive an array of training inputs and known outputs, wherein the training inputs include corrected event data and state data, and the known outputs include the state data.

[0073] In other words, the state prediction machine learning model is trained based on the state data and the corrected event data. The training can be by any method known in the art. For example, the state prediction model can be trained using a stochastic gradient descent algorithm.

[0074] By way of explanation, the structure of machine learning algorithms based on artificial neural networks (NNs) is inspired by the human brain. Neural networks consist of layers, each layer comprising a number of neurons. Each neuron comprises a mathematical operation. In particular, each neuron may comprise a different weighted combination of a single type of transformation (e.g., the same type of transformation, sigmoid function, etc., but with different weights). During the processing of input data, the mathematical operation of each neuron is performed on the input data to produce a numerical output, and the output of each layer in the neural network is sequentially fed to the next layer. The last layer provides the output.

[0075] There are several types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and graph neural networks (GNNs).

[0076] Methods for training machine learning algorithms are well known. Typically, such methods include obtaining a training data set, including training input data entries and corresponding training output data entries. An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and the corresponding training output data entry is used to modify the machine learning algorithm. The process can be repeated until the error converges and the predicted output data entry is sufficiently similar to the training output data entry (e.g., ±1%). This is generally referred to as a supervised learning technique.

[0077] For example, the weights of the mathematical operations of each neuron can be modified until the error converges. Known methods for modifying neural networks include gradient descent, back propagation algorithms, and the like.

[0078] The training input data entries of the state prediction machine learning model used in method 200 correspond to example state data and event data. The training output data entries correspond to state data describing the state of the object in response to previous states and previous events. That is, the machine learning algorithm is trained using a training algorithm configured to receive a training input array and a corresponding known output, wherein the training input includes state data and event data, and the corresponding known output includes subsequent state data. In this way, the machine learning algorithm is trained to output a state prediction indicating the future physiological state of the object.

[0079] Figure 3 A flow chart of a method 300 for generating a state prediction indicative of a future physiological state of a subject is presented. Indeed, this may be particularly useful in a medical setting, where predictions about a subject's future state may enable better decisions about actions to be taken.

[0080] First, a state prediction model is generated. Figure 2 The embodiment of the method 200 described is implemented. Therefore, a state prediction machine learning model is obtained, which can generate a state prediction indicating the future physiological state of the subject based on the time series data of the subject.

[0081] The method 300 then moves to step 310, where time series data associated with the subject is obtained. The time series data includes state data describing at least one physiological characteristic of the subject, and event data including timestamp values ​​describing timing of event occurrences corresponding to the subject. The time series data can be similar to the data used to train the state prediction model, but it is related to the subject.

[0082] Event data may describe an event experienced by an object, or may be a hypothetical event used to determine an expected change in the state of an object when the event occurs.

[0083] At step 320, a subject state prediction is obtained, which is based on inputting the time series data into the generated state prediction model. A care provider can use this subject state prediction to select appropriate actions / treatments to improve subject outcomes.

[0084] By way of illustrative example, embodiments of the present invention for predicting future subject states will now be described. More specifically, biometric (i.e., subject state) monitoring during clinical anesthesia is critical to subject safety. Existing systems and methods can detect the occurrence of abnormalities such as hypotension and bradycardia, and alert the caregiver / anesthetist with an alarm. However, when the alarm is triggered, the abnormal condition has usually already begun.

[0085] Therefore, enabling advanced / early alerts for care providers and anesthesiologists regarding the subject's status / condition would be very useful. Even if the alert is only a few minutes early, appropriate actions and measures can be taken to improve subject outcomes. Therefore, predicting future subject status is very useful in practice.

[0086] To predict the future state of a subject, the predictive capabilities of a machine learning model can be employed. The input data to the machine learning model includes current state data (i.e., vital signs, such as blood pressure and heart rate) and event data (i.e., records of interventions, such as medication administration and patient repositioning). Using the input data, the machine learning model outputs a state object prediction. For example, the subject state prediction may be related to the subject's future blood pressure.

[0087] However, a problem with training machine learning models is the inaccuracy of timestamp values ​​associated with event data. Typically, object state data is automatically collected from sensors and recorded in real time, so the timestamp values ​​associated with such data can be substantially accurate. In contrast, event data is typically entered manually into the system after a procedure / event. Care providers often perform a medical procedure first and enter event data (including corresponding timestamps) after the event trigger has been resolved. Therefore, the timestamp values ​​of event data naturally tend to deviate from the actual true situation timestamp values.

[0088] Furthermore, the amount of event data available is typically very small compared to vital signs data. However, event data (particularly data related to drug administration) has a very large impact on the future state of the subject and its prediction. This is because most of these events are precisely performed to control the state of the subject (i.e., the subject's vital signs, including blood pressure and heart rate). Therefore, inaccuracies in timestamp values ​​in rare / sparse but highly influential event data used to train machine learning models degrade the predictive performance of the machine learning models. Embodiments of the present invention are intended to overcome such inaccuracies while utilizing informative event data.

[0089] In order to provide a solution to the above problem, embodiments of the present invention include the following:

[0090] (i) Input of time series data. Each unit of the time series data has a corresponding timestamp value. The time series data can consist of several columns of object status data (i.e., vital signs) and event data (i.e., intervention records).

[0091] (ii) Machine learning model. The machine learning model outputs predictions about the future state of an object based on the input of time series data. The machine learning model is trained by a training algorithm.

[0092] (iii) Training algorithm. For example, the training algorithm may be stochastic gradient descent (SGD), which is used to train a machine learning model. The training algorithm may read a sample of the time series data at a time (which is called a mini-batch, using a small amount of data in a single training iteration), and update (i.e., train) the parameters of the machine learning model based on the time series data. In order to effectively train the machine learning algorithm, this operation is repeated many times, such as tens of thousands to millions of times.

[0093] (iv) Data Manipulation Unit. Data manipulation corrects / modifies / changes the timestamps corresponding to event data within a given range. The Data Manipulation Unit can be used whenever a training algorithm is used.

[0094] About Figure 5 A-5C discusses examples of data manipulation unit functionality in more detail. Figure 5 A-5C depicts an illustration of modification of timestamps of event data in time series data in accordance with aspects of an embodiment of the present invention.

[0095] For example, an event with a timestamp value equal to 10:03 may have occurred slightly earlier than it did, such as perhaps at 10:01. In this sense, the timestamp value is incorrect. However, if the timestamp value instead reflects that the event occurred between 10:00 and 10:05, it is no longer incorrect.

[0096] In fact, this Figure 5The second and third columns of A-5C record the occurrence of events Med_A and Med_B. - These events have corresponding timestamp values ​​27:00:00 and 34:00:00, respectively. Then in Figure 5 A time-shift function is applied in B, which shifts Med_A to have a timestamp value of 24:00:00 and Med_B to have a timestamp value of 36:00:00. In effect, Figure 5 C represents the range of values ​​that a timestamp value may have.

[0097] In other words, embodiments of the present invention intentionally reduce the resolution of the timestamp value of the event occurrence. This makes it possible to overcome the inaccuracy problem and allow the use of incorrect event data while still effectively training the machine learning model.

[0098] This effective resolution reduction can be achieved by randomly shifting the timestamp values ​​of the event data within some specified width (ie, a predetermined temporal uncertainty range).

[0099] By way of example, suppose the timestamp for the administration of drug A is recorded as 10:03. By modifying the timestamp value, the timestamp value can be between 10:07 and 9.59. In practice, this can be written as:

[0100] t_new=t+ep

[0101] Where t_new is the modified timestamp, ep is the randomly sampled value, and t is the recorded / original timestamp value.

[0102] If you want to shift the timestamp values ​​evenly by 3 minutes before and after the original timestamp value, you can use a uniform distribution:

[0103] ep~U(-180,180)

[0104] Alternatively, a normal distribution can be used:

[0105] ep~180*N(0,1)

[0106] In many cases, we may know that the timestamps of event data for some events often lag behind the real time of the timestamp, and rarely come earlier. In such cases, an asymmetric probability distribution can be used, such as the lognormal distribution:

[0107] ep~-180*logN(0,1)

[0108] To reiterate, the training of a machine learning model can be performed by repeating the process of updating the parameter values ​​of the machine learning model multiple times. The parameter updates can be performed based on portions of the time series data called mini-batches. A time-shift function is applied to each mini-batch. Furthermore, the time-shift function is applied to each timestamp of different events separately.

[0109] Therefore, reducing the resolution of timestamps by random shifting can solve the problem of event data with inaccurate timestamps. The present invention can be applied to sporadic event data in many different fields to effectively train machine learning models.

[0110] 4, a simplified block diagram of a system 400 for correcting machine learning model training data is provided in accordance with an embodiment. The system 400 includes an interface 410 and a data manipulation unit 420.

[0111] The interface 410 is configured to obtain training data comprising timestamp values ​​describing the timing of occurrence of events. The training data may be obtained from a database or from another corpus of information.

[0112] The data manipulation unit 420 is configured to modify the timestamp values ​​of the obtained training data according to a time shift function configured to adjust the timestamp values ​​based on a predetermined time uncertainty range. Figure 1 The functions of the data manipulation unit 420 and the interface 410 are described in more detail. In practice, the system 400 is capable of performing reference Figure 1 Method 100 is described.

[0113] Thus, corrected training data may be obtained from system 400, which may be used for more efficient training of machine learning models.

[0114] In another embodiment, a machine learning model generation unit may be provided, which is configured to generate a machine learning model according to a reference Figure 2 The method 200 described generates a state prediction model suitable for outputting a state prediction indicating a future physiological state of the subject. In other embodiments, a subject state prediction unit may also be provided, which is configured to generate a state prediction model suitable for outputting a state prediction indicating a future physiological state of the subject. Figure 3 The described method 300 generates a state prediction indicative of a future physiological state of a subject.

[0115] Figure 6An example of a computer 1000 in which one or more portions of an embodiment may be employed is illustrated. The various operations discussed above may use the capabilities of the computer 1000. For example, one or more portions of the system for correcting machine learning model training data may be incorporated into any element, module, application, and / or component discussed herein. In this regard, it should be understood that the system functional blocks may run on a single computer, or may be distributed across several computers and locations (e.g., connected via the Internet).

[0116] Computer 1000 may include, but is not limited to, PCs, workstations, laptop computers, PDAs, handheld devices, servers, storage, and the like. In general, in terms of hardware architecture, computer 1000 may include one or more processors 1010, memory 1020, and one or more input and / or output (I / O) devices 1030 communicatively coupled via a local interface (not shown). The local interface may be, for example, one or more buses or other wired or wireless connections, as known in the art, without limitation. The local interface may have additional elements for implementing communication, such as a controller, a buffer (cache), a driver, a repeater, and a receiver. In addition, the local interface may include an address connection, a control connection, and / or a data connection to implement appropriate communication between the above components.

[0117] Processor 1010 is a hardware device for executing software that may be stored in memory 1020. Processor 1010 may be substantially any custom or commercially available processor, central processing unit (CPU), digital signal processor (DSP), or auxiliary processor among several processors associated with computer 1000, and processor 1010 may be a semiconductor-based microprocessor (in the form of a microchip) or a macroprocessor.

[0118] The memory 1020 may include any one or a combination of volatile memory elements (e.g., random access memory (RAM) such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic tape, compact disk read-only memory (CD-ROM), magnetic disk, diskette, cassette, cartridge, etc.). In addition, the memory 1020 may incorporate electrical, magnetic, optical, and / or other types of storage media. Note that the memory 1020 may have a distributed architecture in which various components are remote from each other but accessible by the processor 1010.

[0119] The software in the memory 1020 may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. The software in the memory 1020 includes an applicable operating system (O / S) 1050, a compiler 1060, a source code 1070, and one or more applications 1080 according to the example implementation. As shown, the application 1080 includes many functional components for implementing the features and operations of the example embodiments. The application 1080 of the computer 1000 may represent various applications, computing units, logic, functional units, processes, operations, virtual entities and / or modules according to the example embodiments, but the application 1080 is not meant to be limiting.

[0120] The operating system 1050 controls the execution of other computer programs and provides scheduling, input-output control, file and data management, memory management, and communication control and related services. The inventors contemplate that the application 1080 used to implement the exemplary embodiments may be applicable to all commercial operating systems.

[0121] Application 1080 may be a source program, an executable program (object code), a script, or any other entity that includes a set of instructions to be executed. When it is a source program, the program is typically translated via a compiler (e.g., compiler 1060), an assembler, an interpreter, etc., which may or may not be included in memory 1020 to operate properly in conjunction with operating system 1050. In addition, application 1080 may be written in an object-oriented programming language having data classes and method classes, or a procedural programming language having routines, subroutines, and / or functions, such as, but not limited to, C, C++, C#, Pascal, BASIC, API calls, HTML, XHTML, XML, ASP scripts, JavaScript, FORTRAN, COBOL, Perl, Java, ADA, .NET, etc.

[0122] I / O devices 1030 may include input devices such as, but not limited to, a mouse, keyboard, scanner, microphone, camera, etc. In addition, I / O devices 1030 may also include output devices such as, but not limited to, a printer, a display, etc. Finally, I / O devices 1030 may further include devices for communication input and output such as, but not limited to, a NIC or a modulator / demodulator (for accessing remote devices, other files, devices, systems, or networks), a radio frequency (RF) or other transceiver, a telephone interface, a bridge, a router, etc. I / O devices 1030 also include components for communicating over various networks such as the Internet or an intranet.

[0123] If the computer 1000 is a PC, workstation, intelligent device, etc., the software in the memory 1020 may also include a basic input and output system (BIOS) (omitted for simplicity). The BIOS is a set of necessary software routines that initialize and test the hardware at startup, start the O / S 1050, and support the transfer of data between hardware devices. The BIOS is stored in some type of read-only memory (e.g., ROM, PROM, EPROM, EEPROM, etc.) so that the BIOS can be executed when the computer 800 is started.

[0124] When the computer 1000 is operating, the processor 1010 is configured to execute software stored in the memory 1020, to communicate data to and from the memory 1020, and to generally control the operation of the computer 1000 according to the software. The applications 1080 and the O / S 1050 are read in whole or in part by the processor 1010, possibly buffered in the processor 1010, and then executed.

[0125] When the application 1080 is implemented in software, it should be noted that the application 1080 can be stored on virtually any computer-readable medium for use by or in conjunction with any computer-related system or method. In the context of this document, a computer-readable medium can be an electronic, magnetic, optical, or other physical device or means that can contain or store a computer program for use by or in conjunction with a computer-related system or method.

[0126] Application 1080 can be included in any computer-readable medium for use by or in conjunction with an instruction execution system, device, or apparatus (e.g., a computer-based system, a system including a processor), or other system that can fetch instructions from and execute instructions to an instruction execution system, device, or apparatus. In the context of this document, a "computer-readable medium" can be any component that can store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, device, or apparatus. For example, a computer-readable medium can be, without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, apparatus, or propagation medium.

[0127] about Figure 1-3 The method described and about Figure 5The system can be implemented in hardware or software or a mixture of the two (e.g., as firmware running on a hardware device). Insofar as the embodiments are partially or entirely implemented in software, the functional steps illustrated in the process flow chart can be performed by appropriately programmed physical computing devices (such as one or more central processing units (CPUs) or graphics processing units (GPUs)). Each process - and its individual component steps as illustrated in the flow chart - can be performed by the same or different computing devices. According to an embodiment, a computer-readable storage medium stores a computer program including computer program code, which is configured to cause one or more physical computing devices to perform the encoding or decoding method as described above when the program is run on one or more physical computing devices.

[0128] Storage media may include volatile and non-volatile computer memory, such as RAM, PROM, EPROM and EEPROM, optical disks (such as CD, DVD, BD), magnetic storage media (such as hard disks and magnetic tapes). The various storage media may be fixed within the computing device or may be transportable so that one or more programs stored thereon may be loaded into a processor.

[0129] In terms of the embodiment being implemented in part or in whole in hardware, the frame shown in block diagram Figure 4 can be a separate physical component, or a logical subdivision of a single physical component, or can all be implemented in an integrated manner in one physical component. In an embodiment, the function of a frame shown in the drawings can be divided between multiple components, or in an embodiment, the functions of multiple frames shown in the drawings can be combined in a single component. Hardware components suitable for use in embodiments of the present invention include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs). One or more frames can be implemented as a combination of dedicated hardware that performs some functions and one or more programmed microprocessors and associated circuits that perform other functions.

[0130] The flow chart and block diagram in the accompanying drawings illustrate the possible formalized architecture, function and operation of the system, method and computer program product according to different embodiments of the present invention.In this regard, each square block in the flow chart or block diagram can represent a module, a segment or a part of an instruction, which includes one or more executable instructions for implementing the logical function of (one or more) specifications.In some alternative embodiments, the function marked in the square block can occur in a sequence different from that marked in the accompanying drawings.For example, the two square blocks shown in succession can actually be performed substantially simultaneously, or the square blocks can sometimes be performed in reverse order, depending on the function involved.It will also be noted that the combination of each square block in the block diagram and / or the flow chart illustration and the square block in the block diagram and / or the flow chart illustration can be implemented by a dedicated hardware-based system that performs a specified function or action or performs a combination of special-purpose hardware and computer instructions.

Claims

1. A method (100) for correcting machine learning model training data, the method comprising: obtaining (110) training data, the training data comprising timestamp values ​​describing timings of occurrences of events; as well as The timestamp values ​​of the obtained training data are modified (120) according to a time shift function, the time shift function being configured to adjust the timestamp values ​​based on a predetermined time uncertainty range.

2. The method according to claim 1, wherein: The predetermined time uncertainty range indicates a predicted difference between the timestamp value and an actual timing of the occurrence of the event.

3. The method according to claim 2, wherein: The predetermined time uncertainty range is based on an event type corresponding to the event occurrence.

4. The method according to any one of claims 1 to 3, wherein: The time shift function is configured to adjust the timestamp value based on the predetermined time uncertainty range and a probability distribution algorithm.

5. The method according to claim 4, wherein: The probability distribution algorithm follows a uniform distribution.

6. The method according to claim 4, wherein: The probability distribution algorithm follows a normal distribution.

7. The method according to claim 4, wherein: The probability distribution algorithm follows an asymmetric probability distribution, and preferably follows a log-normal distribution.

8. A method (200) of generating a state prediction model, the state prediction model being adapted to output a state prediction indicative of a future physiological state of a subject, the method comprising: Obtaining (210): time series data, the time series data including state data describing at least one physiological characteristic; and event data, the event data comprising a timestamp value describing the timing of the occurrence of the event; Correcting (100) event data according to any one of claims 1-7; as well as A state prediction model is trained (220) using a training algorithm configured to receive an array of training inputs and known outputs, wherein the training inputs include the corrected event data and the state data, and the known outputs include the state data.

9. A method (200) of generating a state prediction model, the state prediction model being adapted to output a state prediction indicative of a future physiological state of a subject, the method comprising: Obtaining (210): time series data, the time series data including state data describing at least one physiological characteristic; and event data, the event data comprising a timestamp value describing the timing of the occurrence of the event; correcting (100) the event data by modifying at least one of the time stamp values ​​of the event data according to a time shift function, the time shift function being configured to adjust the time stamp value based on a predetermined time uncertainty range; as well as A state prediction model is trained (220) using a training algorithm configured to receive an array of training inputs and known outputs, wherein the training inputs include the corrected event data and the state data, and the known outputs include the state data.

10. The method according to claim 9, wherein: The status data comprises vital sign data, and preferably comprises at least one of: heart rate, blood pressure and oxygen saturation level.

11. The method according to claim 9 or 10, wherein: The event data comprises intervention information describing the treatment of the subject, and preferably comprises at least one of: a medication administration event, a movement event and a treatment event.

12. The method according to any one of claims 9 to 11, wherein: The training algorithm is the stochastic gradient descent algorithm.

13. A method (300) of generating a state prediction indicative of a future physiological state of a subject, the method comprising: Generate (200) a state prediction model according to any one of claims 8-11; Obtaining (310): time series data associated with the subject, the time series data comprising state data describing at least one physiological characteristic of the subject; and event data, the event data comprising a timestamp value describing a timing of an event corresponding to the subject occurring; The object state prediction is obtained (320) based on inputting the time series data into the generated state prediction model.

14. A computer program comprising computer program code means adapted to implement the method according to any one of claims 1 to 13 when the computer program is run on a computer.

15. A system (400) for correcting machine learning model training data, the system comprising: An interface (410) configured to obtain training data, the training data comprising timestamp values ​​describing timing of occurrence of events; as well as A data manipulation unit (420) is configured to modify the timestamp values ​​of the obtained training data according to a time shift function, wherein the time shift function is configured to adjust the timestamp values ​​based on a predetermined time uncertainty range.

Images