Predicting Animal Health Outcomes from Laboratory Test Monitoring

Abstract

The present disclosure relates to techniques for predicting animal health outcomes from laboratory test monitoring using machine learning models. In particular, aspects are directed to obtaining sets of data for multiple animal subjects over a period of time. The sets of data include clinical observation data, weight measurement data, outcome status data, or any combination thereof. The sets of data are processed into a training set of numerical values. A machine learning model is trained on the training set to predict whether the animal subjects' health is normal, whether the animal subjects are likely to require veterinary attention in the future, or whether the animal subjects are likely to have an unplanned death outcome in the future. The machine learning model is output.

Description

[Technical Field]

[0001] The present disclosure relates to clinical animal testing, and in particular to techniques for using machine learning models to predict an animal's health outcome from that animal's past clinical laboratory test data. [Background technology]

[0002] Animals are often used to test drugs before they are available for human use. In animal testing, clinical laboratories are medical facilities that offer a wide range of laboratory procedures to assist veterinary technicians in diagnosing, treating, and managing animal subjects. Clinical laboratories report most laboratory test or other test results as individual numerical or categorical values. However, individual results, viewed in isolation, typically have limited diagnostic value. To properly use the results for the diagnosis and management of animal subjects, clinicians must usually integrate many individual results from animal subjects and interpret them in the context of clinical data and medical knowledge, judgment, and experience. While this manual approach to data interpretation is most often the current standard, computational approaches to laboratory data integration and analysis offer enormous potential for increasing diagnostic value. In particular, many animal subjects will have hundreds or even thousands of these individual test results, often spanning several years. As a result, many clinicians can easily overlook key results or important patterns and trends within sets of laboratory data. Furthermore, important diagnostic information can sometimes be contained within patterns across multiple data elements that may be too subtle or complex to identify without the aid of a computational approach. Additionally, the human brain faces significant challenges when simultaneously considering large numbers of data points, so even the most experienced clinicians may not be able to extract all useful information from existing clinical and laboratory data. To address these limitations and problems, a machine learning-based approach is disclosed herein that combines knowledge discovery with knowledge application, offering the opportunity to provide decision support based on previously unknown patterns. The use of machine learning models to predict health outcomes for animals from historical clinical laboratory test data improves data analysis and provides better animal care. Summary of the Invention

[0003] In various embodiments, a computer-implemented method is provided, the computer-implemented method including: obtaining sets of data for a plurality of animal subjects over a period of time, the sets of data including (i) clinical observation data, (ii) weight measurement data, (iii) outcome status data, (iv) veterinary medical record data, or (v) any combination thereof; training a machine learning model with the training set to predict whether the animal subjects are in good health, whether the animal subjects are likely to require veterinary attention in a future period of time, whether the animal subjects are likely to have an unplanned death outcome in a future period of time, or whether a therapeutic treatment is likely to be administered to the animal subjects in a future period of time; and outputting the machine learning model.

[0004] In some embodiments, processing the set of data into a numerical training set includes determining free text entries in the set of data, applying an embedding model to the free text entries to generate vectors of the free text entries, reducing the size of the vectors using a principal component analysis reduction method, and including the vectors in the training set.

[0005] In some embodiments, processing the set of data into a training set of numeric values ​​includes determining categorical variable entries in the set of data, converting the categorical variable entries to numeric values ​​using a mapping between the numeric values ​​and the categorical variable entries, and including the numeric values ​​in the training set.

[0006] In some embodiments, the method further comprises labeling the values ​​in the training set with an unscheduled mortality index, a veterinary request index, or a veterinary treatment index before training the machine learning model.

[0007] In some embodiments, labeling the values ​​of the training set includes determining that clinical observation data for an animal subject of the plurality of animal subjects includes a veterinary request indicator, and labeling the training set with the veterinary request indicator for the animal subject.

[0008] In some embodiments, labeling the numerical values ​​of the training set includes determining that the outcome status data for an animal subject of the plurality of animal subjects includes an unscheduled mortality indicator, and labeling the training set with the unscheduled mortality indicator for the animal subject.

[0009] In some embodiments, labeling the numerical values ​​of the training set includes determining that veterinary treatment record data for an animal subject of the plurality of animal subjects includes a veterinary treatment indication, and labeling the training set with the veterinary treatment indication for the animal subject.

[0010] In some embodiments, training the machine learning model includes generating a first decision tree based on values ​​for the clinical observation data, the weight measurement data, the veterinary treatment record data, and the outcome status data; determining an error associated with the first decision tree; and generating a second decision tree based on the error and the values ​​for the clinical observation data, the weight measurement data, the veterinary treatment record data, and the outcome status data, wherein the machine learning model includes the first decision tree and the second decision tree.

[0011] In some embodiments, training the machine learning model includes generating a chronologically ordered time graph for each animal subject based on values ​​for clinical observation data, weight measurement data, veterinary treatment record data, and outcome status data, converting the chronologically ordered time graph into a pre-processed table for classification, and automatically adjusting the weights based on predetermined conditions.

[0012] In some embodiments, the machine learning model comprises an additive gradient boosting algorithm or a multi-layer graph neural network algorithm.

[0013] In various embodiments, a computer-implemented method is provided, the computer-implemented method including: obtaining a set of data about an animal subject over a period of time, the set of data including (i) clinical observation data, (ii) weight measurement data, (iii) outcome status data, (iv) veterinary treatment record data, or (v) any combination thereof; inputting the set of data into a machine learning model trained to predict an outcome for the animal subject, the outcome including a veterinary call in a future time period, an unplanned mortality outcome for the animal subject, or a likelihood that a therapeutic treatment will be administered to the animal subject in a future time period; predicting an outcome for the animal subject using the machine learning model; and outputting a classification based on the outcome for the animal subject.

[0014] In some embodiments, the machine learning model is an additive gradient boosting algorithm or a multi-layer graph neural network algorithm.

[0015] In some embodiments, the classification includes comparing the outcome for the animal subject to a determined threshold value and, based on the comparison, classifying the animal subject as having a veterinary need in the future, as having a high likelihood of an unplanned death outcome for the animal subject in the future, or as having a therapeutic treatment that should be administered to the animal subject in the future.

[0016] In some embodiments, the computer-implemented method further comprises providing a recommendation based on the classification of the animal subject.

[0017] In some embodiments, the computer-implemented method further includes providing the classification and / or recommendation to a user through a graphical user interface (GUI).

[0018] In some embodiments, the computer-implemented method further includes, prior to receiving the set of data for the animal subject, obtaining sets of data for a plurality of animal subjects over a period of time, the sets of data including clinical observation data, weight measurement data, outcome status data, veterinary treatment record data, or any combination thereof; processing the sets of data into a training set of numerical values; training a machine learning model with the training set to predict whether the animal subject's health is normal, whether the animal subject is likely to require veterinary attention in a future period of time, whether the animal subject is likely to have an unplanned death outcome in a future period of time, or whether a therapeutic treatment is likely to be administered to the animal subject in a future period of time; and outputting the machine learning model.

[0019] In some embodiments, processing the set of data into a numerical training set includes determining free text entries in the set of data, applying an embedding model to the free text entries to generate vectors of the free text entries, reducing the size of the vectors using a principal component analysis reduction method, and including the vectors in the training set.

[0020] In some embodiments, processing the set of data into a training set of numeric values ​​includes determining categorical variable entries in the set of data, converting the categorical variable entries to numeric values ​​using a mapping between the numeric values ​​and the categorical variable entries, and including the numeric values ​​in the training set.

[0021] In some embodiments, the computer-implemented method further includes labeling the values ​​in the training set with an unscheduled mortality index, a veterinary request index, or a veterinary treatment index before training the machine learning model.

[0022] In some embodiments, labeling the values ​​of the training set includes determining that clinical observation data for an animal subject of the plurality of animal subjects includes a veterinary request indicator, and labeling the training set with the veterinary request indicator for the animal subject.

[0023] In some embodiments, labeling the numerical values ​​of the training set includes determining that the outcome status data for an animal subject of the plurality of animal subjects includes an unscheduled mortality indicator, and labeling the training set with the unscheduled mortality indicator for the animal subject.

[0024] In some embodiments, labeling the numerical values ​​of the training set includes determining that veterinary treatment record data for an animal subject of the plurality of animal subjects includes a veterinary treatment indication, and labeling the training set with the veterinary treatment indication for the animal subject.

[0025] In some embodiments, training the machine learning model includes generating a first decision tree based on values ​​for the clinical observation data, the weight measurement data, the veterinary treatment record data, and the outcome status data; determining an error associated with the first decision tree; and generating a second decision tree based on the error and the values ​​for the clinical observation data, the weight measurement data, the veterinary treatment record data, and the outcome status data, wherein the machine learning model includes the first decision tree and the second decision tree.

[0026] In some embodiments, training the machine learning model includes generating a chronologically ordered time graph for each animal subject based on values ​​for clinical observation data, weight measurement data, veterinary treatment record data, and outcome status data, converting the chronologically ordered time graph into a pre-processed table for classification, and automatically adjusting the weights based on predetermined conditions.

[0027] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods or processes disclosed herein.

[0028] In some embodiments, a computer program product tangibly embodied in a non-transitory machine-readable storage medium is provided, the computer program product comprising instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0029] The terms and expressions which have been employed are used as terms of description rather than of limitation, and the use of such terms and expressions is not intended to exclude any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as set forth in the claims. Thus, while the invention has been specifically disclosed by embodiments and optional features, it is to be understood that modifications and variations of the concepts disclosed herein may be adopted by those skilled in the art, and that such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. [Brief explanation of the drawings]

[0030] The invention will be better understood in view of the following non-limiting drawings.

[0031] [Figure 1A] 1 depicts a block diagram illustrating a machine learning system for training and deploying machine learning models, according to various embodiments. [Figure 1B] 1 illustrates an example of a time forward-in-time graph, according to various embodiments. [Figure 2] 1 shows a flowchart illustrating a process for training a machine learning model according to various embodiments. [Figure 3]1 shows a flowchart illustrating a process for predicting animal health outcomes using machine learning models, according to various embodiments. [Figure 4A] 1 illustrates an exemplary graphical user interface (GUI) displaying animal species, according to various embodiments. [Figure 4B] 10 illustrates an exemplary secondary user interface (UI) displaying an animal's health score, according to various embodiments. [Figure 5] 1 illustrates an example computing environment for implementing the disclosed techniques, according to various embodiments.

[0032] In the accompanying figures, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes between the similar components. When only a first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION OF THE INVENTION

[0033] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments will provide those skilled in the art with an effective description for implementing various embodiments. It will be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0034] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0035] Also, it is noted that particular embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. While a flowchart or diagram may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0036] I. Introduction Machine learning (ML) has had a profound impact on many areas of modern society. For example, it is used to filter spam messages from text documents such as emails, analyze various images to distinguish between them, and extract meaningful data from large datasets through data mining. ML allows for the discovery of patterns, building models, and making predictions by learning from training data. ML algorithms are used in a wide range of domains, including biology and genomics. Deep learning (DL) is a subset of ML that differs from other ML processes in many ways. Most ML models perform well due to custom-designed representations and input features. Using input data generated through this process, ML trains algorithms, optimizing the weights of each feature and optimizing the final prediction. DL attempts to learn multiple levels of representations using a hierarchy of multiple layers. In recent years, DL has surpassed ML in many areas, including speech, vision, and natural language processing. DL and ML are also increasingly being used in the medical field, primarily in image analysis, pharmaceutical research and development, data mining from medical documents, and speech processing. In addition to image and text data from hospital-generated medical charts, various types of laboratory data may also be analyzed, which primarily consist of numbers assigned to various units of measurement. However, very few DL and / or ML models have been developed to analyze laboratory data from animal subjects involved in drug trials.

[0037] In practice, physical examinations and laboratory test results performed by veterinarians are commonly required to assess the condition and predict outcomes of animal subjects. Electronic clinical decision support represents an important tool for improving the efficiency of evaluating these various clinical datasets and converting the data into useful information. The primary purpose of clinical decision support is to provide veterinarians, clinicians, and others with timely information to inform healthcare decisions. Examples of clinical decision support tools include order sets created for specific conditions or types of animal subjects, recommendations and databases that can provide information relevant to specific animal subjects, reminders for preventive care, and alerts regarding potentially dangerous situations. Rule-based algorithms provide the basis for most traditional clinical decision support tools. Rule-based algorithms tend to be easy to develop, validate, implement, and explain, and can often be adapted directly from guidelines or literature. However, rule-based algorithms applied to clinical practice provide decision support based on previously established knowledge.

[0038] To address these limitations and problems, a machine learning-based approach is disclosed herein that combines knowledge discovery with knowledge application, providing the opportunity to provide decision support based on previously unknown patterns. These previously unknown patterns are discovered and implemented using a machine learning model to make inferences regarding clinical decision support (e.g., recommending a veterinarian to notify a technician of a possible disease state for a patient). In one exemplary embodiment, a method is provided that includes obtaining a set of data for a plurality of animal subjects over a period of time, the set of data including (i) clinical observation data, (ii) weight measurement data, (iii) outcome status data, (iv) veterinary treatment record data, or (v) any combination thereof; training a machine learning model with a training set of values ​​to predict whether the animal subjects' health is normal, whether the animal subjects are likely to require veterinary attention in a future period (i.e., a high probability), whether the animal subjects are likely to have an unplanned death outcome in a future period, or whether a therapeutic treatment is likely to be administered to the animal subjects in a future period; and outputting the machine learning model.

[0039] In another exemplary embodiment, a method is provided that includes obtaining a set of data about an animal subject over a period of time, the set of data including (i) clinical observation data, (ii) weight measurement data, (iii) outcome status data, (iv) veterinary medical record data, or (v) any combination thereof; inputting the set of data into a machine learning model trained to predict an outcome for the animal subject, the outcome including a veterinary call in a future time period, an unplanned mortality outcome for the animal subject, or a therapeutic treatment to be administered to the animal subject in a future time period; predicting an outcome for the animal subject using the machine learning model; and outputting a classification based on the outcome for the animal subject.

[0040] As used herein, the terms "substantially," "approximately," and "about" are defined as largely, but not necessarily, the fully specified (and including the fully specified), as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "substantially," "approximately," or "about" may be substituted with "within [a percentage]" of what is specified, where percentage includes 0.1, 1, 5, and 10 percent. As used herein, when an action is "based on" something, this means that the action is based at least in part on at least a portion of something. As used herein, the term "outcome" or "animal health outcome" refers to a value determined as a result of analyzing a single type of clinical data (e.g., clinical observation data, weight measurement data, veterinary medical record data, or outcome status data) or a score determined as a result of analyzing one or more types of clinical data. In some cases, the value is a numeric value, such as 1.0, 2.5, 19.7, 25.0, etc. In other cases, the value is an alphabetical value, such as negative, abnormal, positive, normal, etc. In other cases, the value is alphanumeric, such as an anomaly between 2.0 and 5.0.

[0041] II. Animal Health Outcome Prediction Models and Techniques In various embodiments, machine learning models and techniques are provided to support veterinarians and healthcare organizations by predicting animal health based on the animal's health history, accurately identifying animal subjects prone to veterinary assistance or unplanned death, and potentially providing early notification of a diagnosis. Traditionally, animal health outcomes are predicted manually by veterinarians or other operational staff who analyze health data about animal subjects. When many animal subjects are being cared for and assessed at once, veterinarians and operational staff may be analyzing large amounts of data. Therefore, subtle changes in animal health can be difficult to detect, and if changes are captured, they may be captured when it is too late to intervene to improve the animal subject's health.

[0042] To overcome this and other challenges, the technology described herein uses machine learning models to predict outcomes for animal subjects. The machine learning models can predict negative outcomes (e.g., veterinary consultation, unplanned death, or administration of therapeutic treatment) and promote the health and well-being of the animal by notifying appropriate entities before a negative outcome occurs for the animal subject. The machine learning models can be classification models using ensemble or weak learner approaches, or they can be classification models using graph neural networks. Data can be received in various formats that can be converted into numeric values ​​usable by the machine learning models. For example, a word embedding model can be used to convert free text entries into vectors of numeric values ​​that represent the semantic meaning of the free text. Additionally, categorical variables, which can be predefined, can be mapped to numeric values. Thus, a mapping can be used to convert a particular categorical variable to its corresponding numeric value.

[0043] In machine learning, ensembles combine multiple hypotheses to form a more optimal hypothesis that makes accurate predictions. Ensemble learning combines several base algorithms to form a single optimized predictive algorithm. For example, a typical decision tree for classification takes several factors and converts them into a rule question, and given each factor, it makes a decision or considers another factor. The results of a decision tree can be ambiguous when there are multiple decision rules, for example, when the threshold for making a decision is unclear or when new subfactors are introduced for consideration. Here, ensemble methods can help form a more optimal hypothesis. Instead of relying on a single decision tree to make the right decision or be accurate, ensemble methods take several different trees and aggregate them into a single final, more optimal hypothesis that acts as a powerful predictor.

[0044] In some embodiments, the machine learning model implements the boosting technique as an ensemble method. The boosting algorithm attempts to build a powerful learner (predictive model) from the errors of several weak models. Boosting creates "n" models during model training. Initially, boosting begins by creating a first model (e.g., a decision tree) from the training data. Once the first model is created and the errors from the first model are recorded by the boosting algorithm, the incorrectly classified samples or records are used as input for the subsequent model. Subsequent models are generated from the previous model (e.g., the first model) by attempting to reduce the error from the previous model. Models are added sequentially, each correcting its predecessor, until the training data is predicted according to an acceptance criterion (e.g., greater than 90% accuracy) or until the maximum number of models has been added to the ensemble. Essentially, boosting attempts to reduce bias errors that occur when a model fails to identify relevant trends in the data. This occurs by evaluating the difference between the model's predicted values ​​and the actual or ground truth values ​​assigned to the training data. Various types of boosting may be implemented, such as adaptive boosting (AdaBoost), gradient tree boosting, or XGBoost.

[0045] In certain cases, the machine learning model implements additive gradient boosting techniques. Additive gradient boosting combines multiple weak classifiers to build a single strong classifier. Weak classifiers perform better than random guessing but still perform poorly at assigning classes to objects. A single weak classifier may not be able to accurately predict the class of an object, but when multiple weak classifiers are grouped together, each learning incrementally from other misclassified objects, a single strong model can be generated. The classifiers can be any classifier, such as decision trees, logistic regression, etc. Generating a single strong model can be implemented through a training process that includes generating weak classifiers (e.g., decision trees) using training data based on weighted samples (e.g., animal health outcomes). The weight of each sample indicates how important it is to be correctly classified. Initially, for a first model, all samples may have equal weights. Weak classifiers for each variable may be generated, and a determination may be made as to how well each weak classifier classifies samples into their target classes. For example, a first subset of animal subjects may be evaluated, and for each weak classifier, a determination is made as to how many samples are correctly or incorrectly classified as involving a veterinarian's request or an unplanned death. Based on the determination, an error of the determination may be calculated. A next weak classifier may then be generated based on the error (e.g., the gradient of the error) to improve the classification in the next weak classifier. The training process is then repeated until all samples are correctly classified or a maximum iteration level is reached.

[0046] Artificial neural networks, such as multilayer neural networks, can also be used in machine learning to make accurate predictions. These networks train using many examples of input data (called features or variables) along with output data. The goal is to find an optimal function that efficiently and accurately transforms the input into the output. The basic unit in a neural network is the neuron, which may be expressed as a linear equation (w*x+b), where the gradient (w) and intercept (b) are known as learnable parameters. An essential step in neural networks is the introduction of an activation function, a differentiable, nonlinear function. This function modifies the output of the linear equation, allowing each neuron to adopt a more complex structure. These neurons collectively form layers within the neural network. The network's ability to understand complex, nonlinear relationships between input and output data increases with the addition of more layers. The term "deep" is derived from deep neural networks, which can have tens to hundreds of layers.

[0047] Training a neural network involves multiple iterations on the same data set. During these iterations, learnable parameters are incrementally updated, with the goal of minimizing the discrepancy between the network's current predictions and the actual expected output. This iterative parameter update procedure is known as gradient descent. Due to multiple passes over the same data, the neural network may begin to memorize the details of the output rather than grasping the generalized function. To address this, a portion of the data is reserved during training as a representative subset. This reserved portion is useful for calculating the validation error, which is evaluated against the training error. A clear indication of overfitting occurs when the holdout error does not decrease with the update, but the learned error decreases after each update.

[0048] A well-performing model is usually identified during the first iteration, where both the holdout error and the learned error are relatively small and comparable to previous iterations, suggesting that the model is learning effectively without falling into the trap of overfitting.

[0049] In some embodiments, the machine learning model implements multilayer graph neural network technology as a neural network method. Multilayer graph neural networks can capture known relationships between examples of input data by constructing a graph. Input data from connected examples can share (aggregate) information before undergoing the typical gradient descent learning process of traditional neural networks, a process known as message passing. The number of times message passing occurs can be specified, but if done excessively, it can lead to over-smoothing of information. Sharing information across examples in this way allows domain knowledge of relationships to guide the learning process, often leading to better models. For example, if data is collected sequentially over time, a directed forward-in-time graph can be created using simple sequential connections between observations. This approach will provide a more accurate description of the data than treating the data as independent, as is often the case with typical neural networks. Various types of graph neural networks exist, differing in the methods used to aggregate information across connected observations. Examples include graph convolutional neural networks (GCNs) and graph attention networks (GATs).

[0050] FIG. 1A is a block diagram illustrating a machine learning system 100 according to various embodiments. As shown in FIG. 1A, the machine learning system 100 includes various subsystems or services: a predictive model training subsystem or service 110 for building and training models; and an implementation subsystem or service 115 for implementing one or more models using a computing system (e.g., computing environment 510 of FIG. 5 ). The predictive model training subsystem or service 110 builds and trains one or more predictive models 120 a through 120 n (where “n” represents any natural number) (which may be individually referred to herein as predictive models 120 or collectively as predictive models 120) for use by other subsystems or services. For example, the predictive models 120 may include a model for predicting a class (also referred to herein as a classification) for an animal subject, where the classes identified from multiple target classes include: 0 = normal status for the animal subject; 1 = high probability (i.e., high probability) that veterinary attention for the animal subject will be required in the future. A 1 may also represent other classifications of interest, such as a high likelihood of an unplanned death outcome for the animal subject in the future time period, or a high likelihood that a therapeutic treatment will be administered to the animal subject in the future time period. In some cases, the predictive model 120 may provide multi-class classification, and different numbers may be used to represent different classes. For example, 2 = a high likelihood of an unplanned death outcome for the animal subject in the future time period, 3 = a high likelihood that a therapeutic treatment will be administered to the animal subject in the future time period. Still other types of predictive models may be implemented in other examples in accordance with the present disclosure.

[0051] The predictive model 120 may be a machine learning (“ML”) model such as a convolutional neural network (“CNN”), e.g., an Inception neural network, a residual neural network (“Resnet”), a recurrent neural network, e.g., a long short-term memory (“LSTM”) model or a gated recurrent unit (“GRU”) model, or a multi-layer graph neural network, e.g., a convolutional graph network (GCN), or other variants of deep neural networks (“DNN”) (e.g., multi-label n-binary DNN classifiers or multi-class DNN classifiers). The predictive model 120 may also be other suitable ML models trained to provide predictions, such as generalized linear models (GLMs), generalized additive models (GAMs), support vector machines, bagging models such as random forest models, boosting models, shallow neural networks, or combinations of one or more of such techniques, such as CNN-HMMs or MCNNs (multi-scale convolutional neural networks). A model may also be an ensemble of base models (e.g., decision trees or neural networks) combined via bagging, boosting, or stacking to create an optimal predictive model, e.g., a boosted model such as an AdaBoost or gradient boosting model. The machine learning system 100 may employ the same or different types of predictive models to provide predictions to a user. In particular cases, the predictive model 120 performs predictions using an additive gradient boosting algorithm. Still other types of predictive models may be implemented in other examples in accordance with this disclosure.

[0052] To train the various predictive models 120a-120n, the training subsystem or service 110 is composed of two main components: a data preparation module 130 and a model trainer 140. The data preparation module 130 loads a dataset 145, splits the dataset 145 into training and validation sets 145a-n, and performs a process to preprocess the dataset 145 so that the system can train and test the predictive models 120. The splitting of the dataset 145 into training and validation sets 145a-n may be performed randomly (e.g., 90 / 10% or 70 / 30%), or the splitting may be performed according to more complex validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, nested cross-validation, etc. to minimize sampling bias and overfitting.

[0053] The dataset 145 is obtained from a clinical laboratory or a healthcare system (e.g., an animal subject record system, a clinical trial testing system, etc.). In some cases, the dataset 145 is obtained from a data storage structure, such as a database, a laboratory, or a hospital information system, associated with one or more modalities for acquiring health data about the subject. Additionally or alternatively, the data preparation module 130 may standardize the format of the data. In certain cases, the dataset 145 includes (i) clinical observation data, (ii) weight measurement data, and (iii) outcome status data. The clinical observation data may include health data of the animal subject generated by a physical examination of the animal subject. The clinical observation data may include fields with predefined and selectable or definable categorical variables. The weight measurement data may include a measurement of the weight of the animal subject along with an indication of the time the measurement was taken. The outcome status data may include an indication of the survival rate or morbidity rate of the animal subject. The veterinary medical record data may include historical health data of the animal subject generated during a veterinary visit.

[0054] In some cases, the dataset 145 is stored or standardized by the data preparation module 130 so that it is stored in a data structure appropriate for training (e.g., a list, graph, table, matrix, etc.). Table 1 provides an example of various clinical observation data. Table 2 provides an example of weight measurement data. Table 3 provides an example of outcome status data. Table 4 provides an example of a treatment table specifying the time at which a given drug or therapeutic treatment was administered. The data structure used to store the dataset 145 may be prepared using a design based on any one of Tables 1-4 or a combination thereof. For example, the data structure may be a matrix of size m×n×p, with m rows storing data for m animal subjects, each row corresponding to one animal subject. The n columns of the matrix may correspond to an ordered list of entries, each entry corresponding to one item in Tables 1-4. The data stored in each cell of the matrix is ​​the value of the corresponding item taken at a particular time. For each item, an animal subject may have p different values ​​stored in the p dimensions of the matrix.

[0055] In some cases, the dataset 145 is stored in a data structure including an adjacency graph, an adjacency table, and / or an adjacency list, similar to the matrices described above. FIG. 1B shows an example of a time forward-in-time graph, according to various embodiments. Each node in FIG. 1B represents data associated with an animal subject at a time. Different node patterns in FIG. 1B represent different types of data. For example, hollow circles may represent clinical observations, and cross patterns may represent weight measurements. In FIG. 1B, there are a total of n observations, with each node corresponding to an individual clinical observation, weight measurement, outcome status, or veterinary treatment record. The nodes are connected forward-in-time, allowing each node to propagate information about an observation only to future observations. In some cases, the graph may be reverse, bidirectional, or multidirectional.

[0056] In some cases, an adjacency graph and / or adjacency list is used when the predictive model 120 is a multi-layer graph neural network (GNN) model. Using an adjacency graph (or adjacency table, adjacency list) when training a GNN offers advantages that contribute to the efficiency and effectiveness of the learning process. First, an adjacency matrix / table / list provides a compact and efficient way to represent the connectivity structure of a graph. It captures the relationships between nodes in a concise format, which is essential for scaling up to large graphs. Second, in many cases, graphs for animal subjects are sparse, meaning that most nodes are not directly connected. An adjacency matrix / table / list efficiently encodes this sparsity, allowing GNN algorithms to focus computations only on relevant neighbors, reducing computational overhead. In addition, GNNs rely on aggregating information from neighboring nodes to update a node's representation, and an adjacency matrix / table / list simplifies the process of identifying and accessing neighbors, enabling efficient message passing. Furthermore, an adjacency matrix / table / list can be utilized for parallel processing during training. Operations such as message aggregation and updating can be parallelized across nodes, improving overall training speed. In some cases, graphs representing training animal subjects are substantially larger, with millions of nodes and edges, making efficient representation of adjacency matrices increasingly important. GNNs can handle graphs with millions of nodes and edges by leveraging the sparsity and compactness of adjacency matrices. In addition, adjacency matrices / tables / lists are not limited to specific graph types. They can be used for various graph structures, including directed and undirected graphs and graphs with self-loops, and allow for easy transformation of graphs by adding or removing edges, which is useful in scenarios where the graph evolves over time. Furthermore, adjacency matrices / tables / lists can be used to visualize graph connectivity and relationships, helping to understand the structure of the graph in a GUI.

[0057] While traditional machine learning algorithms make the assumption that each observation is independent, GNNs provide an explicit way to represent dependencies between observations. This can be particularly advantageous when dealing with temporal data that is clearly correlated over time, such as when tracking the health of animal subjects. GNNs also allow for the infusion of domain knowledge, which can then be formatted in the graph structure itself. For example, an experienced veterinary technician might suggest that because food consumption observations are made daily, it might make sense to directly connect observations related to an animal subject's food consumption over time in the graph, even if there are other observations in between. This would enable GNNs to learn from both temporal dependencies and knowledge about the type of observation. Traditional machine learning algorithms do not have this capability and would rely on individual examples to learn such dependencies. [Table 1] [Table 2] [Table 3] [Table 4]

[0058] In some cases, the data in dataset 145 is collected and stored using the same process. Data may be collected by periodic physical and visual inspection of animal subjects and their habitats, digital or analog measurement tools such as scales, clinical laboratory tests, clinical measurements such as neurological examinations, clinical interventions such as treatments, or cataloging overall outcomes. These periodic inspections may be performed by trained staff (e.g., veterinarians, veterinary technicians, or other veterinary operation and animal specialists) who are capable of identifying irregularities in behavior, food consumption, appearance, and other clinical health indicators in animal subjects. Data is collected by trained staff daily, several times a day, or on-demand and is typically stored in a relational database. In some cases, observations made by trained staff are entered into the database via a GUI, where an entry consists of a single observation made on a single animal subject. The database contains information about each animal subject and is separated by a unique identifier (e.g., PRETEST_NUMBER in Table 1) so that trained staff can enter clinical observations for that particular animal subject.

[0059] In some cases, the unique identifier also serves as a primary key to link various data tables. In some cases, each type of observation is placed in its own tabular data table, where a row consists of an observation instance for a single animal subject at a particular time. In some cases, there are four data tables: Table 1 for clinical observations, Table 2 for weight measurements, Table 3 for outcome data, and Table 4 for therapeutic treatments administered. Other metadata regarding the species (SPECIES_NAME), sex of the animal (SEX), and location where the animal subject is located (SITE_NAME) may also be recorded and included with each observation data entry. Each entry may also have a date and timestamp for when the observation was made, for example, as listed under DATE_TIME_TAKEN.

[0060] Data entered into the GUI can be done in several ways: for example, data can be entered via a drop-down menu where only predefined fields can be selected (e.g., animal sex), a character-limited free text field where trained staff can enter any comments within the character limit, a date-time field that stores time, a numeric field, or a Boolean field.

[0061] In some cases, the data used as training data may also be collected from historical or publicly available databases. The data may be entered or stored in data tables or other suitable data structures using techniques similar to those described above.

[0062] The training process for the predictive model 120 may include preprocessing the dataset 145 to normalize the dataset 145 into numeric values ​​that can be interpreted by one or more algorithms trained as the predictive model 120. For example, the data preparation module 130 may determine that the dataset 145 includes free text entries, or entries that are definable and not selectable from a predefined list. For example, MODIFIER_1 in Table 1 is an example of a free text entry. Upon detecting a free text entry, the data preparation module 130 may apply an embedding model to the free text entry to generate a vector for the free text entry. For example, the embedding model may be Word2Vec, GloVe, or any other suitable word embedding model. The embedding model may be pre-trained on open-source biomedical and scientific corpora to generate vectors of size 200 (or any other suitable size). The vectors may numerically represent the semantic meaning of the free text. The data preparation module 130 may then reduce the size of the vectors so that they can be more easily used in downstream calculations. For example, a principal component analysis reduction method may be used to reduce the vectors from size 200 to size 20 (or any other suitable size). Reducing the size of the vectors can allow the predictive model 120 to learn by drawing attention to only those vector components that contain the most information. Once the reduced-size vectors are generated, they can be included in a training data set.

[0063] The data preparation module 130 may further convert categorical variables to numeric values. For example, the data preparation module 130 may determine that the dataset 145 includes categorical variable entries, or entries that are predefined and selectable from a predefined list. As an example, the entry for "large animal" for MEASUREMENT_NAME in Table 1 is a categorical variable entry that can be converted to a numeric value. Each possible categorical variable entry can be mapped to a predefined numeric value that can be selected arbitrarily. For example, categorical variable entries for MEASUREMENT_NAME may include "large animal," "veterinarian request," "neurological exam," etc. The mapping can associate "large animal" with a numeric value of 1, "veterinarian request" with a numeric value of 2, "neurological exam" with a numeric value of 3, etc. The mapping can be stored in a database accessible by the data preparation module 130. Thus, upon detecting a categorical variable entry, the data preparation module 130 can use the mapping to convert the categorical variable entry to a numeric value and then include the numeric value in the training set.

[0064] In some instances, prior to training a machine learning model, data is extracted and preprocessed into a data structure appropriate for training. The extraction process may involve querying data tables (Tables 1-4) from a database for all animal subjects on demand using a standard query language. In some instances, each table will remain separate and be preprocessed individually after extraction. In some instances, tables are combined or concatenated to be stored in a new table.

[0065] Preprocessing steps may involve standardizing data or data structures. In some cases, text strings containing more than a predetermined number of unique options are transformed into embedding vectors using embedding methodologies such as Bag of Words, Word2Vec, or large-scale language model transformer embedding. For example, if a text string contains an open text field where a user can write anything within a character limit, a phrase such as "low food consumption" would be mapped to an embedding vector that can represent its component words and / or semantic meaning. If there are fewer unique options than a predetermined limit, the text field may be integer-encoded, where each unique option is assigned a unique integer value. For example, if a text string contains the sex of an animal, the two unique options may be "male" and "female," encoded as 0 and 1, respectively.

[0066] In some cases, tables containing clinical observations are further preprocessed by removing any measurements that would be indicative of an outcome. For example, if the clinical observation table records an examination that is only performed upon detailed veterinary request (a potential target outcome), it would be excluded during training to avoid information leakage.

[0067] In some cases, weight measurements (WEIGHT in Table 2) are normalized using one of the commonly used normalization techniques before training, such as standard scalar normalization or min-max normalization. Once the clinical observation and weight tables are preprocessed, they may be merged into a single table using the animal subject's unique identifier as the merge key. This combined table is then sorted chronologically by the animal subject's unique number and date-time, from oldest to most recent measurement. Weight measurements may be presumed to be the same over time unless a new measurement is taken.

[0068] Depending on the particular embodiment of the predictive model, additional preprocessing steps may be required prior to training. For example, in the case of a graph neural network, a directed acyclic graph may be constructed for each animal subject from the merged clinical observations and weight tables, where each node represents a measurement over time.

[0069] Once the preprocessed and merged clinical observation and weight tables (input feature tables) and label tables are created, they are stored in some type of storage device with a tag that identifies when the data was created. From here, certain embodiments of machine learning algorithms can read these data from the storage device, and a model can be trained to predict a target label. The training process is triggered on demand, requiring the user to identify which tagged version of the training data to use.

[0070] The training data 145a for the predictive model 120 may include historical data and labels 150 corresponding to the ground truth of normal health, veterinary calls, unscheduled death outcomes occurring for the animal subject, or therapeutic treatments administered to the animal subject. The historical data includes clinical observations and weight measurements. An open veterinary call represents a set of animal observations identified by a veterinary technician as being of concern and therefore likely to result in greater attention from a veterinarian and a prescribed treatment plan for the animal subject. The veterinary call indicator may be included in the clinical observations table as a categorical variable entry (e.g., in the MEASUREMENT_NAME column). The unscheduled death indicator may be included in the outcome status data as a categorical variable entry (e.g., in the DEATH_CODE_NAME column). The veterinary treatment indicator may be included in the outcome status data as a categorical variable entry (e.g., a Boolean variable in the GIVEN column). In some examples, for each dataset, an indication of the correct outcome inferred by the predictive model 120 may be provided as ground truth information for the labels 150 (e.g., unscheduled death indicator, veterinary call indicator, or veterinary treatment indicator). Normal health may be inferred by the absence of unscheduled death indicator, veterinary call indicator, or veterinary treatment indicator. In some cases, the labels 150 may be obtained from a data structure used to maintain data consistency across training samples. The behavior of the predictive model 120 may then be adapted (e.g., via backpropagation) to minimize the difference between the inferences generated for various entities and the ground truth information.

[0071] In some cases, training labels include data extracted from multiple sources, including a clinical observations table (Table 1), an outcome status table (Table 3), and a treatments table (Table 4). In some cases, veterinary requests recorded in the clinical observations table may represent meaningful outcome categories for prediction purposes. In such cases, veterinary requests may be identified in the clinical observations table with associated date-time stamps and essential animal metadata. Similarly, cases involving unscheduled animal deaths from the outcome status table and administered treatments from the treatments table may be separated, preserving their associated metadata and date-time details. These separate datasets are subsequently merged into a unified table organized by unique animal subject identifiers and chronological date-time sequences. To facilitate supervised learning, in some cases, specific target outcomes are encoded as integers, with each class mapped to a distinct non-zero integer value. The integer value of zero is specifically reserved to indicate time intervals during which the animal subject has not yet experienced any associated outcome. For example, if an animal subject presents with an unremarkable observation during a given week without a concurrent veterinarian request, its label for routine observations during that period is designated as zero. This integrated and encoded table serves as the core training label for the supervised learning effort.

[0072] In some cases, clinical observations may be conducted by veterinary technicians or other trained clinical professionals as part of routine evaluation and monitoring. These observations may be conducted through visual inspection of individual animal subjects or through some form of group observation. For example, an individual observation may involve noticing that an animal subject appears to be losing weight, while a group observation may involve finding feces in a location where multiple animal subjects tend to congregate, with uncertainty as to which animal subject is responsible. These clinical observations may be conducted daily or several times a day and may be recorded using a computer-based system. Weight measurements may also be conducted by veterinary technicians or other trained clinical professionals as part of routine evaluation and monitoring. These measurements may be obtained using standard digital and analog scales, and weights may be recorded daily within the computer-based system.

[0073] The model trainer 140 determines hyperparameters for the predictive model 120 and performs an iterative process of inputting examples from training data 145a into the predictive model 120 to find a set of model parameters (e.g., weights and / or biases) that minimize a cost function, such as a loss function or error function, for the predictive model 120. The model trainer 140 is part of a machine learning operational framework that includes hardware such as one or more processors (e.g., CPUs, GPUs, TPUs, FPGAs, etc., or any combination thereof), memory, and storage devices that run software or computer program instructions (e.g., TensorFlow, PyTorch, Keras, etc.) to execute operations, logic, and input / output commands to train the predictive model 120. In some cases, the model trainer 140 performs training using at least a GPU. Since the input data size is on the order of several gigabytes, a GPU can provide better computing performance to further improve computing cost and efficiency.

[0074] Hyperparameters are settings that can be tuned or optimized to control the behavior of the predictive model 120. Most models explicitly define hyperparameters that control different features of the model, such as memory or execution cost. However, additional hyperparameters may be defined to adapt the predictive model 120 to specific scenarios. For example, hyperparameters may include the number of hidden units of the model, the learning rate of the model, the convolution kernel width, the number of kernels for the model, the number of graph connections made during the lookback period, the maximum depth of trees in the random forest, the minimum sample splits, the maximum number of leaf nodes, the minimum number of leaf nodes, etc. A cost function may be constructed to measure the difference between the output predicted using the predictive model 120 and the ground truth, which is annotated to samples with labels.

[0075] In some cases, the model trainer 140 can generate a weak learner or ensemble model. Initially, the model trainer 140 can create a first model (e.g., a decision tree) from the training data 145a. Generating the first model can involve the model trainer 140 identifying which input variables in the training data 145a can best separate the target variable (e.g., unscheduled death or upcoming veterinary call), or more precisely, identifying how easy it is to distinguish between possible target variables when the input variables are split at a particular value. For example, the location where an animal subject is located may have less influence on predicting the probability of veterinary call than the animal subject's food consumption cataloged in clinical observation data, so food consumption is likely to be a better predictor. The model trainer 140 may determine the split based on a purity measure. For example, a Gini impurity score or other purity score may be determined, which, if randomly selected, may measure the likelihood that a data point will be associated with a given class (in this case, the target label is veterinary call or unscheduled death). The split is determined by the value of the variable that provides the purest split. A first model is created, and the error from the first model is confirmed by the boosting algorithm's loss function (e.g., a logarithmic loss function). The misclassified training data is used as input for the subsequent model. Subsequent models are generated from a previous model (e.g., the first model) by attempting to reduce the error from the previous model. Subsequent models are added that reduce the loss (i.e., follow the gradient of the error) to perform a gradient descent procedure. Subsequent models may be parameterized, and the parameters may then be modified to move in a direction that reduces the residual loss. Models are added sequentially, each correcting its predecessor, until the training data 145a is perfectly predicted or a maximum number of models (e.g., 100) have been added to the ensemble. Essentially, boosting attempts to reduce bias errors that occur when a model fails to identify relevant trends in the data.This occurs by assessing the error between the model's predicted values ​​and the actual or ground truth values ​​assigned to the training data 145a. The output of each model is added to the outputs of the other models to correct or improve the final output of the predictive model 120, which includes all ensemble models.

[0076] Once the set of model parameters is identified, the predictive model 120 is trained, and the model trainer 140 performs an additional testing or validation process using a subset of the test data 145b (a test or validation data set). The testing or validation process involves iteratively inputting utterances from the subset of the test data 145b into the predictive model 120 using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, and nested cross-validation to adjust the hyperparameters and ultimately find an optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, a reserved test set from the test data subset 145b may be input into the predictive model 120 to obtain an output (in this example, a prediction of veterinary care or unplanned death), which may be evaluated against ground truth entities using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient. Additionally, performance metrics may be calculated, such as error, precision, precision, recall, receiver operating characteristic curves (ROC), etc. The metrics may be used to analyze the performance of the predictive model 120 to provide recommendations.

[0077] The model training subsystem or service 110 outputs trained models, including one or more trained predictive models 160. The one or more trained predictive models 160 may be deployed and used in the implementation subsystem or service 115 (as described in detail with respect to FIG. 3 ) to provide predictions 165 to a user. For example, the trained predictive model 160 may receive input data 170 including clinical observation data for an animal subject over a period of time (e.g., 3 weeks), weight measurement data for the animal subject over that period of time, outcome status data for the animal subject over that period of time, or any combination thereof, and provide predictions 165 to a user based on the likelihood of veterinary care in a future period of time (e.g., 5 days), treatment needed in a future period of time (e.g., 5 days), or unscheduled death for the animal subject. The input data 170 may include data stored in the same data structures as used in the dataset 145. For example, the input data 170 is stored in an adjacency graph that includes sub-graphs that store clinical observation data over a period of time, weight measurement data over that period of time, outcome status data for animal subjects over that period of time, and / or veterinary medical record data for animal subjects over that period of time.

[0078] The implementation subsystem or service 115 includes a deployment tool 175 that is part of a machine learning operational framework that includes hardware such as one or more processors (e.g., CPU, GPU, TPU, FPGA, etc., or any combination thereof), memory, and storage that runs software or computer program instructions (e.g., application programming interfaces (APIs), cloud infrastructure, Kubernetes, Docker, TensorFlow, Kuberflow, Torchserve, etc.) to execute the computations, logic, and input / output commands to execute the predictive model 120 in a production environment. In some cases, the deployment tool 175 implements the deployment of the predictive model 120 using cloud platforms such as Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. Cloud platforms make machine learning more accessible, flexible, and cost-effective while enabling developers to build and deploy predictive models 120 faster.

[0079] FIG. 2 shows a flowchart illustrating a process 200 for training a machine learning model according to various embodiments. The process depicted in FIG. 2 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof (e.g., an intelligent selection machine). The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 2 and described below is intended to be exemplary and non-limiting. While FIG. 2 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, steps may be performed in several different orders, and some steps may also be performed in parallel. In certain embodiments, the process depicted in FIG. 2 or portions of the process may be performed by a computing device such as a computer (e.g., computing device 520 of FIG. 5).

[0080] In block 205, a set of data for a plurality of animal subjects is obtained. The set of data includes (i) clinical observation data, (ii) weight measurement data, (iii) outcome status data, (iv) veterinary treatment record data, or (v) any combination thereof. The clinical observation data, weight measurement data, veterinary treatment record data, and outcome status data can include free text entries, categorical variable entries, floating point entries, or any combination thereof. In some cases, all of the historical data for the animal subjects is used as input for training the machine learning model. In other cases, in block 215, a predetermined amount of data for the animal subjects (e.g., data falling within a given time frame or window) is selected and used as the set of input data for training the machine learning model. In particular cases, the predetermined amount of data is all of the historical data collected for the animal subjects within a look-back window (e.g., three weeks prior to a health event such as needing veterinary attention or an unscheduled death), or, if there is no health event, all of the historical data collected for the animal subjects within a most recent look-back window (e.g., the last three weeks of data). The predetermined amount of data is an adjustable hyperparameter (for example, it can be set based on conversations with veterinary staff and how they typically review an animal's history to determine any health issues).

[0081] At block 210, the data set is processed into a training set of numeric values. Free text entries may be converted into vectors of numeric values ​​using word embedding vectors, and categorical variable entries may be converted into numeric values ​​using a predefined mapping between categorical variables and numeric values. Vectors for free text entries may additionally be reduced in size. Floating-point numeric values ​​may be left unchanged.

[0082] At block 215, a machine learning model is trained on the training set. Training may involve generating a first decision tree based on values ​​for the clinical observation data, weight measurement data, veterinary treatment record data, and outcome status data, and then determining an error associated with the first decision tree. A second decision tree may then be generated based on the error and values ​​for the clinical observation data, weight measurement data, veterinary treatment record data, and outcome status data. Additional decision trees may be generated until an acceptable level of error is reached or an iteration number is reached. A machine learning model includes each of the generated decision trees and is trained for multi-class prediction.

[0083] In some cases, training can involve generating a multi-layer graph neural network. A time-oriented, forward-in-time graph may be created for each animal's clinical observation, weight measurement, and treatment administration data. Information is propagated forward over time between observations for each animal and aggregated. The aggregated animal history is then passed through a series of hidden neural network layers, returning a predicted probability of treatment for each animal. The most likely predicted class is subsequently compared to the actual class to calculate an error. The learnable parameters may then be updated, and another iteration of updating may be performed. This process continues until the model converges to a best-fit, generalizable (i.e., not overfitted) model.

[0084] In some cases, training can be further optimized based on new or expanded training data. For example, more time point entries corresponding to training animal subjects can be collected over time and input into a partially trained model for model optimization. In some cases, as discussed with respect to FIG. 1A, the adjacency matrix / graph used to store training data can advantageously accommodate the addition of more input entries.

[0085] The machine learning model ultimately attempts to predict: 0 = all health statuses are good; 1 = veterinary attention is likely in the future (e.g., the animal subject is ill); 2 = the animal subject is likely to experience unscheduled death in the future; or 3 = therapeutic treatment is likely to be administered to the animal subject in the future. The likelihood or probability for a given class refers to the model's confidence in making that prediction. For example, a 70% probability of veterinary attention means that the model is 70% confident that the observation indicates that veterinary attention is needed in the future. This is different from classification accuracy, which tells us the actual correctness of the prediction as verified by humans. Therefore, the machine learning model is (70%) confident but may be wrong (inaccurate). Confidence is equal to the likelihood or probability for a given class. The machine learning model does not distinguish when exactly the event will occur; it only indicates that the event is likely to occur within a future period (e.g., a 0-5 day window). This is because the model is trained on a data set with future veterinary requests or attention labels. As a result, the training set of values ​​may take into account observational data from a look-back window period, and then if a veterinary request is opened any time within the future period (e.g., 0-5 days in the future), this is labeled as a positive example for learning (e.g., an opened veterinary request may be defined as any time within a 0-5 day window).

[0086] In block 220, the machine learning model is output to predict whether the animal subject's health is normal, whether the animal subject is likely to require veterinary attention in the future, whether the animal subject is likely to have an unplanned death outcome in the future, or whether a therapeutic treatment is likely to be administered to the animal subject in the future. Currently, veterinary technicians and staff must painstakingly review all clinical observations, weight measurements, and other details for each animal to determine which animals need to be observed first. Alternatively, another strategy is to simply go room by room. In either case, when the number of animal subjects is large, veterinary technicians and staff are uncertain about which animals may need further attention until they complete the assessment. This wastes resources, including human resources, and in some cases can result in a highly inaccurate means of assessing the health of animal subjects. Predictions from machine learning can aid in the triage process and increase the overall accuracy of health assessments without requiring veterinary technicians and staff to first review every animal's history.

[0087] After a machine learning model is trained, model artifacts are created that constitute the fully trained model. Machine learning models can be used to make predictions on new data by exposing them through some kind of service. This is typically done by loading the machine learning model artifact from disk, storing it in memory on some computerized system, and then creating a REST API endpoint that allows other applications the ability to pass data to the model. The model endpoint returns class predictions as well as probabilities. Making predictions from a trained machine learning model is called inference. Because the goal is to predict outcomes before they occur, only the clinical observation table (Table 1) and weight table (Table 2) are needed for inference. The clinical observation table and weight table follow the same preprocessing and merging pipeline described for training.

[0088] FIG. 3 is a flowchart illustrating a process 300 for predicting an animal's health outcome using a machine learning model, according to various embodiments. The process depicted in FIG. 3 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof (e.g., an intelligent selection machine). The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 3 and described below is intended to be exemplary and non-limiting. While FIG. 3 depicts various process steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, steps may be performed in several different orders, and some steps may also be performed in parallel. In certain embodiments, the process depicted in FIG. 3 or portions of the process may be performed by a computing device such as a computer (e.g., computing device 520 of FIG. 5).

[0089] In block 305, a set of data spanning a period of time is obtained for an animal subject. The set of data may include (i) clinical observation data, (ii) weight measurement data, (iii) outcome status data, (iv) veterinary treatment record data, or (v) any combination thereof. The clinical observation data, weight measurement data, veterinary treatment record data, and outcome status data may include free text entries, categorical variable entries, floating-point entries, date and time entries, numeric entries, Boolean entries, or any combination thereof. The set of data may be preprocessed before being input into the machine learning model in block 310. For example, free text entries may be converted into vectors of numeric values ​​using word embedding vectors, and categorical variable entries may be converted into numeric values ​​using a predefined mapping between categorical variables and numeric values. Vectors for free text entries may be additionally reduced in size using dimensionality techniques such as principal components. Floating-point values, numeric entries, or Boolean entries may be left unchanged. The set of data may be further preprocessed, for example, by data preparation module 130 using similar preprocessing techniques disclosed in FIG. 1A. The set of data, or the preprocessed set of data, is stored in a data structure (e.g., a list, graph, table, matrix, etc.) suitable for input to a machine learning model at block 310. For example, the set of data may be stored in an adjacency graph used by a multi-layer graph neural network.

[0090] At block 310, the set of data is input to a machine learning model having a learned set of model parameters for predicting an outcome for the animal subject. In some cases, the machine learning model is obtained via the process described with reference to FIGS. 1A and 2. The machine learning model may include an ensemble of classifiers, where the learned set of parameters is associated with a relationship calculated by a boosting algorithm. In particular cases, the boosting algorithm is an additive gradient boosting algorithm. The machine learning model may include a neural network classifier, where the learned set of parameters is associated with a relationship calculated by a gradient descent algorithm. In some cases, the neural network classifier is a multilayer graph neural network. In particular cases, the multilayer graph neural network is constructed based on a time-oriented forward-in-time graph. The set of data may be input to the machine learning model via a graphical user interface (GUI).

[0091] At block 315, an outcome for the animal subject is predicted using the machine learning model. The machine learning model may provide a multi-class prediction, where a first outcome is associated with a first predicted class (e.g., healthy animal subject), a second outcome is associated with a second predicted class (e.g., veterinary referral), a third outcome is associated with a third predicted class (e.g., unscheduled death), and / or a fourth outcome is associated with a fourth predicted class (e.g., therapeutic treatment administered). In some cases, the results are stored in a data structure suitable for output or display on a GUI.

[0092] At block 320, a classification is output based on the result for the animal subject. The classification can involve comparing the result for the animal subject to a determined threshold and classifying the animal subject as having a veterinary consultation in a future time period or as having an unplanned death outcome based on the comparison. For example, the result can include a confidence level of the prediction, and the classification can be based on a comparison of the confidence level to a threshold. For example, if the result indicates an 80% confidence level that the animal subject will have a future veterinary consultation and the threshold is 75%, the classification can be that the animal subject is predicted to have a veterinary consultation. However, if the result indicates a 50% confidence level, the classification can be that the animal subject is not predicted to have a veterinary consultation. Based on the classification, a probability of requiring attention or a recommendation for veterinary consultation can be provided. Additionally, if the classification indicates a likelihood of unplanned death for the animal subject, a recommendation regarding a course of action can be provided, or a treatment recommendation can be provided, allowing a veterinary professional to determine which animal subject to examine first to assess the appropriate course of treatment. In some cases, the probabilities and / or recommendations are provided to a user, such as a veterinary technician (e.g., a healthcare worker associated with the animal subject). The classification probabilities and / or recommendations can promote the health and well-being of the animal by consistently and accurately detecting when an animal subject is expected to experience an undesirable outcome and taking action to improve the expected outcome.

[0093] Optionally, in block 325, the classification and / or recommendation may be provided to a user, such as a veterinary technician, via a graphical user interface (GUI). Exemplary GUIs are shown in FIGS. 4A and 4B. As can be seen in FIG. 4A, each icon may represent a different species, e.g., a dog, a monkey, and a pig. Clicking on each icon may present a secondary user interface (UI) demonstrating the classification and / or probability of requiring attention for the animal subject. FIG. 4B shows a secondary UI showing the probability of requiring attention for a monkey. As shown in FIG. 4B, the probability of an animal likely requiring attention based on the machine learning model classification is presented. The presented probability may be related to the sum of all probabilities associated with the unhealthy class, such as the probability of veterinary consultation, treatment, or unscheduled death in the next few days. Animals with scores above a predetermined threshold may be color-coded based on the threshold to facilitate identification of animals flagged by the model. Historical animal health scores may also be presented in context to enable veterinary technicians and staff to identify animal subjects whose health may be deteriorating over time. For example, a darkly shaded cell in FIG. 4B represents a probability greater than 70% that the animal subject requires attention, while a lightly shaded cell represents a probability between 40% and 70%. When the probability is less than 40%, the corresponding cell may not be shaded. As shown in FIG. 4B, the secondary UI may include multiple sheets (e.g., a scoreboard, AI scores, and / or animals) and may be easily switched between different sheets based on user needs.

[0094] In some cases, a complete clinical report for the animal subject may also be provided by the GUI or secondary UI. The complete clinical report may include the set of data obtained for the animal subject in block 305, the predicted results generated by the machine learning model in block 315, and the classification probabilities (and / or recommendations) output in block 320 for the animal subject's entire history. By reviewing the complete report, veterinary technicians and staff can better triage which animals require attention first based on the animals' specific clinical observations.

[0095] In some cases, notifications are displayed using the GUI or a secondary UI. For example, when a user clicks on an animal species icon on the GUI, the secondary UI will display a health score or AI score associated with the animal subject of the animal species. In some cases, the displayed score or similar notification is color-coded. For example, a red notification may indicate that the animal subject needs attention, yellow indicates monitoring, and a white background means that nothing is predicted to be noteworthy. The color coding may be predetermined with associated thresholds.

[0096] In some cases, the species measure is also displayed using the GUI. For example, the proportion or percentage of monkey subjects whose health score exceeds a predetermined threshold may be displayed in association with the monkey icon in FIG. 4A. In some cases, a notification to the user may be automatically pushed to the user when the proportion or percentage exceeds a predetermined threshold. The notification may be pushed using the GUI or via a communication system, such as an instant messaging system.

[0097] FIG. 5 illustrates a non-limiting example of a computing environment 510 in which various systems, methods, processes, and data structures described herein may be implemented. The computing environment 510 is merely one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the systems, methods, and data structures described herein. The computing environment 510 should not be interpreted as having any dependency or requirement relating to any one component or combination of components illustrated in the computing environment 510. A subset of the systems, methods, and data structures illustrated in FIG. 5 may be utilized in particular embodiments. The systems, methods, and data structures described herein operate with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of known computing systems, environments, and / or configurations that may be suitable include, but are not limited to, personal computers, server computers, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0098] The computing environment 510 includes a computing device 520 (e.g., a computer or other type of machine, such as a sequencer, photocell, photomultiplier tube, optical reader, sensor, etc.) that includes a processing unit 521, a system memory 522, and a system bus 523 that operatively couples various system components, including the system memory 522, to the processing unit 521. There may be only one processing unit 521, or there may be more than one processing unit 521, such that the processor of the computing device 520 includes a single central processing unit (CPU) or multiple processing units, commonly referred to as a parallel processing environment. The computing device 520 may be a conventional computer, a distributed computer, or any other type of computer that may include a graphical processing unit (GPU) used for computation.

[0099] The system bus 523 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory 522, which may also be referred to simply as memory, includes read-only memory (ROM) 524 and random access memory (RAM) 525. A basic input / output system (BIOS) 526, containing the basic routines that help to transfer information between elements within the computing device 520, such as during start-up, is stored in ROM 524. The computing device 520 may further include a hard disk drive interface 527 for reading from and writing to a hard disk (not shown), a magnetic disk drive 528 for reading from or writing to a removable magnetic disk 529, and an optical disk drive 530 for reading from or writing to a removable optical disk 531, such as a CD-ROM or other optical media.

[0100] The hard disk drive, magnetic disk drive 528, and optical disk drive 530 are connected to the system bus 523 by a hard disk drive interface 532, a magnetic disk drive interface 533, and an optical disk drive interface 534, respectively. The drives and their associated computer-readable media provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing device 520. Any type of computer-readable medium capable of storing data accessible by a computer may be used in the operating environment, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memory (RAM), read-only memory (ROM), etc.

[0101] Some program modules include an operating system 535, one or more application programs 536, other program modules 537, and program data 538, which may be stored on the hard disk, magnetic disk 529, optical disk 531, ROM 524, or RAM 525. A user may enter commands and information into the computing device 520 through input devices such as a keyboard 540 and a pointing device 542. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 521 through a serial port interface 546 that is coupled to the system bus, but may also be connected by other interfaces such as a parallel port, game port, or universal serial bus (USB). A monitor 547 or other type of display device is also connected to the system bus 523 via an interface, such as a video adapter 548. In addition to a monitor, computers typically include other peripheral output devices (not shown), such as speakers and printers.

[0102] The computing device 520 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 549. These logical connections may be achieved by or part of a communications device coupled to the computing device 520, or in other ways. The remote computer 549 may be another computer, a server, a router, a network PC, a client, a peer device, or other common network node, and typically includes many or all of the elements described above for the computing device 520. The logical connections depicted in FIG. 5 include a local area network (LAN) 551 and a wide area network (WAN) 552. Such networking environments are commonplace in office networks, enterprise-wide computer networks, intranets, and the Internet, which are all types of networks.

[0103] When used in a LAN networking environment, the computing device 520 is connected to the local area network 551 through a network interface or adapter 553, which is one type of communications device. When used in a WAN networking environment, the computing device 520 often includes a modem 554, some type of communications device, or any other type of communications device for establishing communications over the wide area network 552. The modem 554, which may be internal or external, is connected to the system bus 523 via a serial port interface 546. In a networked environment, program modules, such as the application program 536 depicted for the computing device 520, or portions thereof, may be stored in a remote memory storage device. It will be understood that the network connections shown are non-limiting examples and that other communications devices may be used to establish a communications link between computers.

[0104] III. Additional Considerations Specific details are given in the above description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits may be shown in block diagrams so as not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0105] The implementation of the techniques, blocks, steps, and means described above can be done in various ways. For example, these techniques, blocks, steps, and means can be implemented in hardware, software, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions described above, and / or combinations thereof.

[0106] Also, it is noted that the embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. A flowchart may describe operations as a sequential process, where many of the operations may be performed in parallel or concurrently. Additionally, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

[0107] Furthermore, embodiments may be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scripting languages, and / or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable medium such as a storage medium. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or any combination of instructions, data structures, and / or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, ticket passing, network transmission, etc.

[0108] For a firmware and / or software implementation, the methodologies may be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. Any machine-readable medium tangibly embodying instructions may be used in implementing the methodologies described herein. For example, software code may be stored in a memory. The memory may be implemented within the processor or external to the processor. As used herein, the term "memory" refers to any type of long-term, short-term, volatile, non-volatile, or other storage medium and is not limited to any particular type of memory or memories or type of media on which memory is stored.

[0109] Additionally, as disclosed herein, the terms "storage medium," "storage," or "memory" can refer to one or more memories for storing data, including read-only memory (ROM), random-access memory (RAM), magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices, and / or other machine-readable media for storing information. The term "machine-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and / or various other storage media capable of containing or carrying instructions and / or data.

[0110] While the principles of the present disclosure have been described above in connection with specific apparatus and methods, it is to be clearly understood that this description is made only by way of example and not as a limitation on the scope of the disclosure.

Claims

1. 1. A method comprising: acquiring sets of data for a plurality of animal subjects over a period of time, the sets of data including: (i) clinical observation data; (ii) weight measurement data; (iii) outcome status data; (iv) veterinary medical record data; or (v) any combination thereof; processing said data set into a training set of numbers; training a machine learning model on the training set to predict whether an animal subject is in normal health, whether the animal subject is likely to require veterinary attention in a future period, whether the animal subject is likely to have an unplanned death outcome in a future period, or whether a therapeutic treatment is likely to be administered to the animal subject in a future period; and outputting the machine learning model.

2. Processing the set of data into the training set of values ​​includes: determining free text entries in the set of data; applying an embedding model to the free text entry to generate a vector of the free text entry; reducing the size of the vector using principal component analysis reduction; and including the vector in the training set.

3. Processing the set of data into the training set of values ​​includes: determining categorical variable entries in the set of data; converting said categorical variable entries to numeric values ​​using a mapping between numeric values ​​and categorical variable entries; and including the numerical value in the training set.

4. Before training the machine learning model, The method of claim 1 , further comprising labeling the values ​​in the training set with an unscheduled death index, a veterinarian request index, or a veterinarian treatment index.

5. Labeling the values ​​in the training set includes: determining that the clinical observation data for an animal subject among the plurality of animal subjects includes the veterinarian's request indicator; and labeling the training set with the veterinarian's request index for the animal subject.

6. Labeling the values ​​in the training set includes: determining that the outcome status data for an animal subject of the plurality of animal subjects includes the unscheduled mortality indicator; and labeling the training set with the unscheduled mortality indicator for the animal subject.

7. Labeling the values ​​in the training set includes: determining that the veterinary treatment record data for an animal subject among the plurality of animal subjects includes the veterinary treatment indicator; and labeling the training set with the veterinary treatment indication for the animal subject.

8. Training the machine learning model includes: generating a first decision tree based on values ​​for the clinical observation data, the weight measurement data, the veterinary medical record data, and the outcome status data; determining an error associated with the first decision tree; generating a second decision tree based on the error and the values ​​for the clinical observation data, the weight measurement data, the veterinary treatment record data, and the outcome status data, wherein the machine learning model comprises the first decision tree and the second decision tree.

9. Training the machine learning model includes: generating a chronologically ordered time graph for each animal subject based on the values ​​for the clinical observation data, the weight measurement data, the veterinary medical record data, and the outcome status data; converting the chronologically ordered time graph into a preprocessed table for classification; and automatically adjusting the weights based on predetermined conditions.

10. The method of claim 1 , wherein the machine learning model comprises an additive gradient boosting algorithm or a multi-layer graph neural network algorithm.

11. 1. A method comprising: acquiring a set of data about an animal subject over a period of time, the set of data including (i) clinical observation data, (ii) weight measurement data, (iii) outcome status data, (iv) veterinary medical record data, or (v) any combination thereof; inputting the set of data into a machine learning model trained to predict an outcome for the animal subject, the outcome comprising a likelihood of veterinary care in a future time period, an unplanned mortality outcome for the animal subject, or a likelihood that a therapeutic treatment will be administered to the animal subject in a future time period; using the machine learning model to predict the outcome for the animal subject; and outputting a classification based on the results for the animal subject.

12. The method of claim 11 , wherein the machine learning model is an additive gradient boosting algorithm or a multi-layer graph neural network algorithm.

13. 12. The method of claim 11, wherein the classification comprises comparing the outcome for the animal subject to a determined threshold and, based on the comparison, classifying the animal subject as having veterinary needs in the future period, as having a high probability of the unplanned death outcome for the animal subject in the future period, or as having a therapeutic treatment that should be administered to the animal subject in the future period.

14. 14. The method of claim 13, further comprising providing a recommendation based on the classification of the animal subject.

15. The method of claim 14 , further comprising providing the classification and / or the recommendation to a user through a graphical user interface (GUI).

16. prior to receiving the set of data about the animal subject, acquiring sets of data for a plurality of animal subjects over a period of time, the sets of data including the clinical observation data, the weight measurement data, the outcome status data, the veterinary medical record data, or any combination thereof; processing said data set into a training set of numbers; training the machine learning model on the training set to predict whether an animal subject is in normal health, whether the animal subject is likely to require veterinary attention in a future period, whether the animal subject is likely to have an unplanned death outcome in a future period, or whether a therapeutic treatment is likely to be administered to the animal subject in a future period; The method of claim 11 , further comprising: outputting the machine learning model.

17. Processing the set of data into the training set of values ​​includes: determining free text entries in the set of data; applying an embedding model to the free text entry to generate a vector of the free text entry; reducing the size of the vector using principal component analysis reduction; and including the vector in the training set.

18. Processing the set of data into the training set of values ​​includes: determining categorical variable entries in the set of data; converting said categorical variable entries to numeric values ​​using a mapping between numeric values ​​and categorical variable entries; and including the numerical value in the training set.

19. Before training the machine learning model, 17. The method of claim 16, further comprising labeling the values ​​in the training set with an unscheduled death index, a veterinary request index, or a veterinary treatment index.

20. Labeling the values ​​in the training set includes: determining that the clinical observation data for an animal subject among the plurality of animal subjects includes the veterinarian's request indicator; and labeling the training set with the veterinarian's request index for the animal subject.

21. Labeling the values ​​in the training set includes: determining that the outcome status data for an animal subject of the plurality of animal subjects includes the unscheduled mortality indicator; and labeling the training set with the unscheduled mortality indicator for the animal subject.

22. Labeling the values ​​in the training set includes: determining that the veterinary treatment record data for an animal subject among the plurality of animal subjects includes the veterinary treatment indicator; and labeling the training set with the veterinary treatment indication for the animal subject.

23. Training the machine learning model includes: generating a first decision tree based on values ​​for the clinical observation data, the weight measurement data, the veterinary medical record data, and the outcome status data; determining an error associated with the first decision tree; generating a second decision tree based on the error and the values ​​for the clinical observation data, the weight measurement data, the veterinary treatment record data, and the outcome status data, wherein the machine learning model comprises the first decision tree and the second decision tree.

24. Training the machine learning model includes: generating a chronologically ordered time graph for each animal subject based on the values ​​for the clinical observation data, the weight measurement data, the veterinary medical record data, and the outcome status data; converting the chronologically ordered time graph into a preprocessed table for classification; and automatically adjusting the weights based on predetermined conditions.

25. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform the method of any one of claims 1 to 24.

26. 1. A system comprising: one or more data processors; a non-transitory computer-readable storage medium comprising instructions that, when executed on said one or more data processors, cause said one or more data processors to perform the method of any one of claims 1 to 24.

Images