Loop thresholds in machine learning for predicting infection counts

Abstract

A method for predicting case counts for future dates in one or more geographic areas of a person infected with a disease is disclosed. The presence of the disease in a biological sample can be tested by a polymerase chain reaction (PCR) test. The load of the one or more pathogens associated with the disease is associated with a PCR cycle that indicates the presence of the one or more pathogens, referred to as a threshold cycle (Ct). Data related to predicting a case count including Ct data and other data is received. The Ct data includes Ct values from a PCR test of a biological sample of a person within the one or more geographic regions. An array of feature data processed by the trained machine learning model is generated, the feature data including Ct features obtained from the data and other features. A predicted number of infected persons is generated by processing the array using machine learning.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 391,740, filed on July 23, 2022, and claims priority to U.S. Non-Provisional Application Serial No. 18 / 225,065, filed on July 21, 2023. To the extent permitted by applicable jurisdictions, the entire contents of this application are incorporated herein by reference. Background Art

[0003] The present disclosure generally relates to techniques for predicting case counts during a disease outbreak. Summary of the invention

[0004] Predicting future case counts is an important tool when managing a disease outbreak. While national case counts can be predicted fairly well using relatively simple models applied to historical national case count data, effective logistical planning at the local level requires the ability to better predict future case counts in local geographic areas. One recent advance in addressing this problem is the β-AR model described in the following paper, the entire contents of which are incorporated herein by reference: Matthew Le et al., Neural Relational Autoregression for High-Resolution COVID-19 Forecasting, published by FB Data for Good, October 1, 2020 (available at: https: / / ai.meta.com / research / publications / neural-relational-autoregression-for-high-resolution-covid-19-forecasting) (the "β-AR Paper").

[0005] Polymerase chain reaction (PCR) tests are widely used to determine pathogens, such as specific viruses or bacteria, or other pathogens, such as fungi, protozoa, worms or prions. PCR tests perform thermal cycles on biological samples. If the target sequence is present in the sample, the DNA corresponding to the target sequence is amplified cyclically. If the target sequence can be detected by the PCR instrument before a given cycle (for example, before the 38th cycle of a 40-cycle assay), then the test can be considered "positive" for the corresponding person infected with the virus corresponding to the sequence. However, the information provided by the PCR test is not just whether a person is positive or negative. It also provides a cycle threshold (Ct), i.e., the PCR cycle at which the relevant sequence is fully amplified for the first time to be detected. Because the PCR process amplifies DNA, on average, the cycle of the first detectable sequence is inversely proportional to the amount of the given DNA sequence initially present in a given sample volume. In other words, the amount of a given DNA sequence indicated by a small Ct value is much higher than the amount of a given DNA sequence indicated by a high Ct value. It has been shown that this is related to viral load, i.e., the amount of virus in an infected person.

[0006] Although PCR tests provide Ct data, only binary "positive" or "negative" result data (rather than Ct data) are usually used to predict incidence and epidemic trajectories. Hay et al. have shown that, on average, because Ct data are related to viral load, they can improve incidence estimates and epidemic growth reproduction rate estimates. See Hay et al., "Estimating epidemiologic dynamics from cross-sectional viral load distributions", published in Science 373, eabh0635 (2021) July 16, 2021, the entire contents of which are incorporated herein by reference ("Hay paper").

[0007] However, neither the β-AR model nor other existing models utilize Ct data to improve predictions of future case counts.

[0008] Embodiments of the present disclosure provide methods, systems, and computer program products to improve high-resolution case count predictions by generating and using features derived from Ct data from PCR testing. Specifically, Ct data is used to generate Ct features to improve the performance of machine learning models in case count prediction.

[0009] Additional details of these embodiments are more fully disclosed herein and in: Sharmin et al., “Cross-sectional Ct distributions from qPCR tests can provide an early warning signal for the spread of COVID-19 in communities,” medRxiv preprint doi: https: / / doi.org / 10.1101 / 2023.01.12.23284489 , published on January 14, 2023, the entire contents of which are incorporated herein by reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A high-level view of a computerized system according to an exemplary embodiment of the present disclosure is shown.

[0011] Figure 2 yes Figure 1 Framework diagram of the case count prediction system cited in .

[0012] Figure 3 is a flow chart illustrating a method for generating a Ct feature according to an embodiment of the present disclosure.

[0013] Figure 4 An exemplary computer system that can be configured by a computer program product to perform embodiments of the present disclosure is shown.

[0014] Although the present disclosure is described with reference to the above drawings, these drawings are intended to be illustrative and other embodiments are consistent with the spirit of the invention and are within the scope of the invention. DETAILED DESCRIPTION

[0015] Various embodiments will now be described more fully below with reference to the accompanying drawings, which form a part of the present invention and are used as illustrations to show specific examples of practical embodiments. However, this specification can be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein; on the contrary, these embodiments are provided to make this specification more thorough and complete and to fully convey the scope of the invention to those skilled in the art. In addition, this specification can be embodied as a method or an apparatus. Therefore, any of the various embodiments herein may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Therefore, the following description should not be considered to be limiting.

[0016] Figure 1 A system 1000 according to an exemplary embodiment of the present disclosure is shown. The system 1000 includes a data source computer 101, one or more computers 103, and a user device 107.

[0017] Instructions for implementing the case count prediction system 102 reside in a computer program product 104 stored in a storage device 105, and these instructions can be executed by a processor 106. When the processor 106 is executing the instructions of the computer program product 104, the instructions or a portion thereof are typically loaded into a working memory 109, from which the processor 106 can easily access the instructions. In the illustrated embodiment, the computer program product 104 is stored in a storage device 105 or another non-transitory computer-readable medium (which may include media distribution across different devices and different locations). In an alternative embodiment, the storage medium is transient.

[0018] In one embodiment, processor 106 actually includes multiple processors, which may include additional working memory (additional processors and memory are not separately shown), including a graphics processing unit (GPU), which includes at least thousands of arithmetic logic units that support large-scale parallel computing. GPUs are commonly used for deep learning applications because GPUs can perform related processing tasks more efficiently than typical general-purpose processors (CPUs). Other embodiments include one or more special-purpose processing units, which include systolic arrays and / or other hardware arrangements that support efficient parallel processing. In some embodiments, such special-purpose hardware works with CPUs and / or GPUs to perform various processes described herein. In some embodiments, such special-purpose hardware includes application-specific integrated circuits, etc. (which may refer to a part of an application-specific integrated circuit), field programmable gate arrays, etc., or a combination thereof. However, in some embodiments, processors such as processor 106 can be implemented as one or more general-purpose processors (preferably with multiple cores) without departing from the spirit and scope of the present invention.

[0019] The user device 107 includes a display 108 for displaying the results of the processing performed by the case count prediction system 102. Such user devices may include mobile devices, such as mobile phones, smart phones, smart watches, or tablet computers, and / or laptop or desktop computers including a display 108. In some embodiments, alerts of impending epidemic waves in one or more communities of interest detected by the case count prediction system 102 are routed to one or more users in real time or near real time via their respective user devices 107. Such alerts may be displayed on the user device 107 via application notifications to one or more mobile applications that are configured to receive the results of the processing performed by the case count prediction system 102. Such application notifications may be automatically displayed on the display 108 of the user device 107 and may alert the user via an audible sound or vibration.

[0020] In some embodiments, alerts of upcoming epidemic waves detected by the case count prediction system 102 may be sent to users via email alerts displayed on the user device 107. In other embodiments, these alerts may be displayed on a user dashboard of the case count prediction system 102, which is shown on the display 108 of the user device 107.

[0021] In a typical embodiment, data source computer 110 is connected to a network such as the Internet or another public or private network ( Figure 1 The computer 103 communicates with one or more computers in the computer 103 through a computer network such as a computer network (not shown separately), which may be a wide area network or a local area network.

[0022] Figure 2 According to the embodiments of the present disclosure Figure 1 . The system 102 includes a preprocessing block 201 and a machine learning model 203. In the example shown, the machine learning model 203 includes a recurrent neural network (RNN) 204, an autoregressive model 205, and an output multiplier 206. In the example shown, the RNN 204 includes two long short-term memory (LSTM) layers with a hidden state size of 2. The outputs from the autoregressive model 205 and from the RNN 204 are multiplied by the output multiplier 206, which outputs a case count prediction for one or more future dates in each of one or more geographic regions, which in this example are counties.

[0023] In one example, the operation of the case count prediction system 102 is performed as follows. The preprocessing block 201 preprocesses Figure 1 The data source computer 110 is shown receiving cycle threshold (Ct) data and other data and generating a feature array 202 for input into a machine learning model 203. The feature array 202 includes three-dimensional data including feature values ​​and corresponding geographic regions (e.g., counties, states, etc.) and dates associated with the data from which the feature values ​​were derived.

[0024] The feature data 202-2 includes Ct features, which will be described below. Figure 3 Detailed description in the context of .

[0025] Feature data 202-1 includes other features. In this example, the other features include features cited in the β-AR paper cited in the above overview section. Specifically, the β-AR features include features obtained from the following datasets: confirmed cases (data collected by the New York Times), Facebook Permanent Data (FBDG) symptom survey, FBDG movement range map, Google community mobile data, doctor visits (CMU COVIDcast), testing (COVID tracking project) and weather (including average, minimum, maximum temperature and rainfall for each county (from NOAA GHCN). See the β-AR paper at 6.

[0026] In this example, the Ct feature data 202 - 2 and most of the β-AR feature data 202 - 1 are input into the RNN 204 , except for the β-AR confirmed case features which are input into the autoregressive model 205 .

[0027] In an alternative embodiment, additional features are used in addition to the features included in the β-AR feature data 202-1 and the Ct feature data 202-2. For example, in addition to the β-AR features and the Ct features, features associated with disease variants are used. In one example, the time-varying prevalence values ​​of each of the one or more current top variants are used as additional features. For example, in one embodiment, five variant features can be obtained for use by selecting the top five variants from GISAID, for example, available at: https: / / www.gisaid.org / epiflu-applications / influenza-genomic-epidemiology / , and the time-varying prevalence values ​​calculated from the GISAID site for each of the five selected variants can be used as features.

[0028] In the example shown, the machine learning model 203 includes the neural relational autoregressive model (β-AR model) described in the β-AR paper. However, in alternative embodiments, other machine learning models capable of generating case count predictions from data including Ct data can be used. For example, the machine learning model 203 can also be updated over time, where machine learning model enhancements can be considered and incorporated into the machine learning model 203.

[0029] Figure 3 Shown by Figure 2 Method 300 of generating Ct features 202-2 by preprocessing block 201. Method 300 operates on Ct data 320 to generate Ct features 341-347, which fill Ct feature array 202-2 and are provided as part of feature array 202 Figure 2 Machine learning model 203.

[0030] Step 301 uses the Ct data 320 to generate features 341, 342, 343, and 344 by determining the mean, smoothed mean, skewness, and smoothed skewness of the Ct value vector, respectively. Specifically, a corresponding set of features 341-344 is calculated for each corresponding date (e.g., each calendar day) on which a sample corresponding to a corresponding Ct value was collected. And this is done for each of the one or more geographic regions (e.g., each county) for which Ct data is provided (the geographic region data dimension is not included in the Figure 3 shown separately in ).

[0031] Features 341 and 343 (mean and skewness) are calculated based on all Ct values ​​collected on a given date in a given geographic area. Features 342 and 344 (smoothed mean and smoother skewness) are calculated based on Ct values ​​collected in a moving date window around a given date. In one example, the moving window is 14 days, which means, for example, that the smoothed mean is based on Ct values ​​collected seven days before and seven days after a given date. In addition, in one example, for each date in the rolling window, the daily average Ct value is used for smoothed mean and smoothed skewness determination.

[0032] Step 302 uses the weekly Ct data to estimate the incidence rate and generates estimated incidence rate data 340. In this example, an estimated incidence rate for each county per day is generated. In one example, this is done using a Gaussian process model from the Hay virosolver R package, using recommended parameters. The package is available at https: / / jameshay218.github.io / virosolver / index.html, and the entire contents are incorporated herein by reference.

[0033] Step 302 uses the estimated morbidity data to generate an estimated effective reproductive rate (Rt) curve. In one example, this is done by first calculating a smoothed moving average of the estimated morbidity using a 14-day window. The resulting smoothed morbidity is then used to estimate the Rt curve using EpiEstim, which can be obtained from

[0034] EpiEstim is available at https: / / cran.r-project.org / web / packages / EpiEstim / index.html, and the entire contents are incorporated herein by reference. Each estimated Rt curve is a time series of estimated Rt values, for example, a series of daily estimated Rt values. In one example, additional data in addition to the Ct-derived incidence estimates can also be submitted to EpiEstim to enrich the estimated Rt curve determination. For example, case count data can also be submitted. In one example, this data can also be smoothed using, for example, a moving average calculation with a 14-day moving window. In one example, the parameters recommended by EpiEstim for an average series interval of 6.14 and a standard deviation of 3.96 can be used.

[0035] Step 304 then uses the estimated Rt curve to determine features 345, 346, and 347. Specifically, it determines the median estimated Rt value for each day and the upper and lower confidence limits.

[0036] In some embodiments, machine learning model performance will be automatically evaluated over time, and features that show reduced utility will be excluded and reconsidered if they appear valuable again. In other embodiments, new features may be considered through test runs of the machine learning model 203. If new features are determined to be useful in predicting case counts, such new features may be manually added to the machine learning model. In some embodiments, the feature may be manually added using the user interface of the case count prediction system 102.

[0037] Figure 4 An exemplary computer system is shown that can be configured by a computer program product to perform embodiments of the present invention.

[0038] In this example, the computer system 400 may provide one or more of the components of the automated case count prediction system, which is configured to implement one or more logic modules and artificial neural networks and associated components for a computer-implemented case count prediction system and an associated interactive graphical user interface. The computer system 400 executes the instruction code contained in the computer program product 460. The computer program product 460 includes executable code in an electronically readable medium, which can instruct one or more computers (such as the computer system 400) to perform the processing of the exemplary method steps performed by the embodiments cited herein. The electronically readable medium can be any non-transitory medium that stores information electronically and can be accessed locally or remotely, for example, via a network connection. In an alternative embodiment, the medium can be transient. The medium may include a plurality of geographically dispersed media, each of which is configured to store different parts of the executable code at different locations or at different times. The executable instruction code in the electronically readable medium instructs the computer system 400 shown to perform various exemplary tasks described herein. The executable code for instructing to perform the tasks described herein will usually be implemented in software. However, it will be appreciated by those skilled in the art that a computer or other electronic device may utilize code implemented in hardware to perform many or all of the identified tasks without departing from the present invention.It will be appreciated by those skilled in the art that many variations of the exemplary methods within the spirit and scope of the present invention may be found for implementing executable code.

[0039] The code or a copy of the code contained in the computer program product 460 may reside in one or more permanent storage media (not separately shown) communicatively coupled to the computer system 400 for loading and storage in the permanent storage device 470, and / or reside in the memory 410 for execution by the processor 420. The computer system 400 also includes an I / O subsystem 430 and peripherals 440. The I / O subsystem 430, the peripherals 440, the processor 420, the memory 410, and the permanent storage device 470 are coupled via a bus 450. Similar to the permanent storage device 470 and any other permanent storage device that may contain the computer program product 460, the memory 410 is a non-transitory medium (even if implemented as a typical volatile computer storage device). In addition, those skilled in the art will appreciate that in addition to storing the computer program product 460 for carrying out the processes described herein, the memory 410 and / or the permanent storage device 470 may be configured to store various data elements mentioned and shown herein.

[0040] Those skilled in the art will appreciate that computer system 400 illustrates only one example of a system in which a computer program product according to an embodiment of the present invention may be implemented. To cite only one example of an alternative embodiment, the storage and execution of instructions contained in a computer program product according to an embodiment of the present disclosure, such as, for example, computer program product 460, may be distributed across multiple computers, such as, for example, computers of a distributed computing network.

Claims

1. A method, implemented by one or more computers, for predicting case counts for a future date in one or more geographic areas of humans infected with a disease associated with one or more pathogens, the presence of the one or more pathogens in a biological sample being testable by a polymerase chain reaction (PCR) test, such that the load of the one or more pathogens is generally correlated to a PCR cycle at which the PCR test of the biological sample indicates the presence of the one or more pathogens, such PCR cycle being referred to as a threshold cycle (Ct), the method comprising: receiving, at one or more computers, data related to predicting the case count, the data comprising Ct data and other data, the Ct data comprising Ct values ​​from PCR tests of biological samples from people within the one or more geographic regions; generating, by the one or more computers, an array of feature data for processing by a trained machine learning model implemented by the one or more computers, the feature data comprising Ct features obtained from the Ct data and other features obtained from the other data; as well as The array of feature data is processed by the one or more computers using the machine learning model to generate at least one predicted case count, the at least one predicted case count comprising a predicted number of infected persons at the future date in the one or more geographic regions.

2. The method of claim 1, wherein the Ct data comprises corresponding sets of Ct values ​​from PCR tests performed on corresponding dates, the PCR tests corresponding to people in the one or more geographic regions.

3. The method of claim 2, wherein generating comprises determining a mean and a skewness for each of the corresponding sets of Ct values.

4. The method of claim 3, wherein generating further comprises determining a smoothed mean and a smoothed skewness for each of the corresponding sets of Ct values ​​using Ct values ​​from a rolling date window around the date of each corresponding set of Ct values.

5. The method according to any one of claims 2 to 4, wherein generating further comprises: determining a corresponding set of estimated incidence rates using the corresponding set of Ct values; determining a corresponding set of estimated effective reproductive rate (Rt) time series values ​​using the corresponding set of estimated disease incidences; as well as Determine the mean and skewness for each corresponding set of Rt time series values.

6. The method of claim 5, wherein generating further comprises: Determine the smoothed mean and smoothed skewness for each corresponding set of Rt time series values.

7. A method according to any one of claims 1-6, wherein the machine learning model comprises a recurrent neural network.

8. The method of claim 7, wherein the machine learning model further comprises an autoregressive model and an output multiplication function, the output multiplication function being configured to multiply the output of the recurrent neural network with the output of the autoregressive model to provide the output of the machine learning model, wherein: Some features including the Ct feature are processed by the recurrent neural network; and At least one of the other features is processed by the autoregressive model.

9. A method according to any one of claims 7-8, wherein the recurrent neural network comprises two long short-term memory (LSTM) layers.

10. The method of claim 9, wherein the two LSTM layers have a hidden state size of 2.

11. The method of any one of claims 2-10, wherein the respective date corresponding to the respective set of Ct data is the date on which the sample for the corresponding PCR test was collected.

12. The method of any one of claims 1-11, wherein the one or more geographic regions include a plurality of corresponding geographic regions, and further wherein the at least one case count includes a plurality of corresponding case counts, each corresponding case count corresponding to a different one of the corresponding geographic regions.

13. The method of claim 12, wherein the corresponding geographic area is a county.

14. The method of any one of claims 5-13, wherein using the corresponding set of estimated disease incidences to determine a corresponding set of estimated effective reproductive rate (Rt) time series values ​​comprises using an EpiEstim process.

15. The method of any one of claims 5-14, wherein using the corresponding set of Ct values ​​to determine the corresponding set of estimated incidence rates comprises using Hay model processing.

16. A computer program product, comprising executable code stored in a non-transitory computer readable medium, the executable code being executable on one or more computer processors to perform the method according to any one of claims 1-15.

17. A computer system comprising one or more computers, wherein the one or more computers are configured by the computer program product according to claim 16 to execute the method according to any one of claims 1-15.

18. A non-transitory computer readable medium storing one or more executable instructions that, when executed by at least one processor coupled to the non-transitory computer readable medium, perform a method for predicting case counts for a future date in one or more geographic areas of humans infected with a disease associated with one or more pathogens, the presence of the one or more pathogens in a biological sample being testable by a polymerase chain reaction (PCR) test, such that the load of the one or more pathogens is generally correlated to a PCR cycle at which the PCR test of the biological sample indicates the presence of the one or more pathogens, such PCR cycle being referred to as a threshold cycle (Ct), the method comprising: receiving, at one or more computers, data related to predicting the case count, the data comprising Ct data and other data, the Ct data comprising Ct values ​​from PCR tests of biological samples from people within the one or more geographic regions; generating, by the one or more computers, an array of feature data for processing by a trained machine learning model implemented by the one or more computers, the feature data comprising Ct features obtained from the Ct data and other features obtained from the other data; as well as The array of feature data is processed by the one or more computers using the machine learning model to generate at least one predicted case count, the at least one predicted case count comprising a predicted number of infected persons at the future date in the one or more geographic regions.

19. A system for predicting case counts at a future date in one or more geographic areas of humans infected with a disease associated with one or more pathogens, the presence of the one or more pathogens in a biological sample being testable by a polymerase chain reaction (PCR) test, such that the load of the one or more pathogens is generally correlated to the PCR cycle at which the PCR test of the biological sample indicates the presence of the one or more pathogens, such PCR cycle being referred to as a threshold cycle (Ct), the system comprising: one or more processors configured to receive data from one or more data source computers, the data relevant to predicting the case count and including Ct data and other data, the Ct data including Ct values ​​from PCR tests of biological samples from people within the one or more geographic regions; as well as One or more computer-readable memories for storing a plurality of computer-readable instructions that, when executed by the one or more processors, perform the following operations: generating an array of feature data for processing by a trained machine learning model implemented by the one or more processors, the feature data comprising Ct features obtained from the Ct data and other features obtained from the other data; as well as The array of feature data is processed using the machine learning model to generate at least one predicted case count comprising a predicted number of infected persons at the future date in the one or more geographic regions.

20. The system of claim 19, wherein the plurality of computer-readable instructions, when executed by the one or more processors, further perform the step of providing a real-time or near real-time notification of the predicted case count to a user device.

Images