Systems and methods to ensure machine learning model results can be audited

Abstract

In one embodiment, a method for auditing results of a machine learning model includes: retrieving, from an audited database, a set of state estimates for original time series data values; reversing, for each state estimate, a state estimate calculation to produce reorganized time series data values for each state estimate; retrieving, from the audited database, the original time series data values; pairwise comparing the original time series data values to the reorganized time series data values to determine whether the original time series matches the reorganized time series; and generating a signal indicating that the audited database (i) has not been modified if the original time series matches the reorganized time series, and (ii) has been modified if the original time series does not match the reorganized time series.

Description

Background Technology

[0001] As the cost of sensor and processor technologies decreases, sensors are being added to or associated with components in electrical, mechanical, power distribution, and other systems to enable sensor-based automation. These sensors generate vast amounts of information describing component behavior, which is stored in databases for at least a period of time. Machine learning algorithms can be applied to the sensed information to enable prediction, anomaly detection / discovery, and predictive maintenance of sensor-monitored system components.

[0002] Examples of sensor-based automation can be found in power utilities, oil and gas, environmental and water quality monitoring, data processing, manufacturing, passenger and freight transport, and even the financial services sector. In these areas, the behavior of equipment or systems and / or decisions made by machine learning processes can be subject to regulatory scrutiny. This scrutiny can be based on stored sensor information, and substantial fines for violating entities can be imposed due to non-compliance revealed by stored sensor information. Therefore, it is worthwhile for regulated entities and regulators to ensure that stored sensor information is not compromised or tampered with.

[0003] For this and other reasons, there is a need for an effective and efficient technique to ensure that the results guaranteed by machine learning models can be audited. Summary of the Invention

[0004] In one embodiment, a computer-implemented method for auditing the results of a machine learning model includes: retrieving from an audited database a set of state estimates for original time series data values, wherein each state estimate is generated by a state estimate computed for one of the time series data values; reversing the state estimate computed for each state estimate to produce a recombined time series data value for each state estimate; retrieving the original time series data values ​​from the audited database; comparing the original time series data values ​​with the recombined time series data values ​​in pairs to determine whether the original time series and the recombined time series match; and generating a signal indicating that the audited database (i) has not been modified if the original time series and the recombined time series match, and (ii) has been modified if the original time series and the recombined time series do not match.

[0005] In one embodiment, a computer-implemented method for auditing the results of a machine learning model further includes: training a multivariate state estimation model with a set of training values ​​selected from raw time series values; decomposing a matrix associated with the multivariate state estimation module into a set of eigenvectors; selecting a subset of principal eigenvectors from the set of eigenvectors; and creating a restricted multivariate state estimation model from the subset of principal eigenvectors; wherein the reversal of the state estimation computation further includes the reversal of the generative computation, thereby forming a state estimate from the restricted multivariate state estimation model.

[0006] In one embodiment, a computer-implemented method for auditing the results of a machine learning model further includes: omitting training data values ​​from raw time-series data values ​​in an audited database; preprocessing the remaining raw time-series data values ​​to mitigate the effects of sensor interference and generating a model that records changes to the remaining raw time-series data; and performing state estimation on the preprocessed remaining raw time-series data values ​​using a restricted multivariate state estimation model to create a compressed time-series database.

[0007] In one embodiment, a computer-implemented method for auditing the results of a machine learning model further includes: preprocessing raw time series values ​​to mitigate the impact of sensor interference on the quality of the raw time series values; and generating a data preprocessing model that records one or more changes to the raw time series values ​​during preprocessing; wherein the reversal of state estimation computation further includes: retrieving the data preprocessing model; and reversing the preprocessing described by the data preprocessing model for each state estimation.

[0008] In one embodiment, a computer-implemented method for auditing the results of a machine learning model, the method further comprising generating an electronic data report data structure comprising: a preprocessing model that records one or more changes to the original time series values ​​during preprocessing to mitigate the impact of sensor interference on the quality of the original time series values; a compressed time series database generated by performing state estimation with a restricted multivariate state estimation model and excluding values ​​of parameters from the compressed time series database that are not from the restricted multivariate state estimation model; and (i) a set of training values ​​selected from the original time series values ​​and used to train the restricted multivariate state estimation model and (ii) one or more of the restricted multivariate state estimation models.

[0009] In one embodiment, a computer-implemented method for auditing the results of a machine learning model, wherein the reversal of a state estimation computation further includes: identifying a reversed state estimation computation that reverses the steps performed by the state estimation computation to form a state estimate from the original time series data values; generating a set of reversed state estimates for the original time series data from the set of state estimates, wherein each reversed state estimate is generated by performing a reversed state estimate on one of the set of state estimates; wherein the reconstructed time series data values ​​for each state estimate are based on the reversed state estimate.

[0010] In one embodiment, a computer-implemented method for auditing the results of a machine learning model further includes: generating an electronic verification report message indicating that the audited database has been proven to be undamaged and untampered in response to a signal that the audited database has been modified (i) and (ii) modified; generating an electronic verification report message indicating that the audited database has been damaged or tampered with; and transmitting the generated electronic verification report to a computing device so that the verification report message is stored by the computing device or displayed by the computing device.

[0011] In one embodiment, a non-transitory computer-readable medium storing computer-executable instructions for auditing the results of a machine learning model, the instructions, when executed by at least one processor of a computer, cause the computer to: retrieve a set of state estimates for original time-series data values ​​from an audited database, wherein the state estimates are generated by calculating the state estimate for each time-series data value; reverse the state estimate calculation for each state estimate to produce a reconstituted time-series data value for each state estimate; retrieve the original time-series data values ​​from the audited database; compare the original time-series data values ​​in pairs with the reconstituted time-series data values ​​to determine whether the original time series and the reconstituted time series match; and generate a signal indicating that the audited database (i) has not been modified if the original time series and the reconstituted time series match, and (ii) has been modified if the original time series and the reconstituted time series do not match.

[0012] In one embodiment, a non-transitory computer-readable medium, wherein the instructions further cause a computer to: train a multivariate state estimation model with a set of training values ​​selected from the original time series values; decompose a matrix associated with the multivariate state estimation module into a set of eigenvectors; select a subset of eigenvectors with the largest eigenvalue from the set of eigenvectors; create a restricted multivariate state estimation model from the subset of eigenvectors; preprocess the original time series values ​​to mitigate the impact of sensor interference on the quality of the original time series values; create a data preprocessing model that records one or more changes to the original time series values ​​during preprocessing; perform state estimation on the original time series data values ​​using the restricted multivariate state estimation model to create a compressed time series database; and generate a report including the preprocessing model, the compressed time series database, and one or more of (i) the set of training values ​​and (ii) the restricted multivariate state estimation model; wherein the set of state estimates is the set of state estimates included in the compressed time series database in the report, and wherein the reconstructed time series data is generated based on the report.

[0013] In one embodiment, a computational system for auditing the results of a machine learning model includes: a processor; a memory operatively connected to the processor; a sensor interface operatively connected to the processor and the memory; and a non-transitory computer-readable medium operatively connected to the processor and the memory and storing computer-executable instructions, which, when executed by at least the processor of the computer, cause the computer to: retrieve from an audited database a set of state estimates of raw time-series data values ​​received via the sensor interface, wherein the state estimates are generated by calculating a state estimate for each time-series data value; reverse the state estimate calculation for each state estimate to produce a reconstituted time-series data value for each state estimate; retrieve the raw time-series data values ​​from the audited database; compare the raw time-series data values ​​in pairs with the reconstituted time-series data values ​​to determine whether the raw time-series and the reconstituted time-series match; and generate a signal indicating that the audited database (i) has not been modified if the raw time-series and the reconstituted time-series match, and (ii) has been modified if the raw time-series and the reconstituted time-series do not match.

[0014] In one embodiment, a computational system for auditing the results of a machine learning model, wherein the non-transitory computer-readable medium further includes instructions that, when executed by at least a processor, cause the computational system to: train a multivariate state estimation model with a set of training values ​​selected from the original time series values; decompose a matrix associated with the multivariate state estimation module into a set of eigenvectors; select a subset of principal eigenvectors from the set of eigenvectors; and create a restricted multivariate state estimation model from the subset of principal eigenvectors; wherein the reversal of the state estimation computation further includes the reversal of the generative computation, thereby forming a state estimate from the restricted multivariate state estimation model.

[0015] In one embodiment, a computational system for auditing the results of a machine learning model, wherein the non-transitory computer-readable medium further includes instructions that, when executed by at least a processor, cause the computational system to: omit training data values ​​from raw time-series data values ​​in the audited database; preprocess the remaining raw time-series data values ​​to mitigate the effects of sensor interference and generate a model that records changes to the remaining raw time-series data; and perform state estimation on the preprocessed remaining raw time-series data values ​​using a restricted multivariate state estimation model to create a compressed time-series database.

[0016] In one embodiment, a computational system for auditing the results of a machine learning model, wherein instructions for reversing state estimation computation further cause the computational system to generate an electronic data report data structure comprising: a preprocessing model that records one or more changes to the original time series values ​​during preprocessing to mitigate the impact of sensor interference on the quality of the original time series values; a compressed time series database generated by performing state estimation with a restricted multivariate state estimation model and excluding values ​​of parameters from the compressed time series database that are not from the restricted multivariate state estimation model; and (i) a set of training values ​​selected from the original time series values ​​and used to train the restricted multivariate state estimation model and (ii) one or more of the restricted multivariate state estimation models; wherein instructions for reversing state estimation computation further cause the computer to: undo the steps performed by the state estimation computation using the reverse state estimation computation; and reverse the preprocessing described by the preprocessing model for each state estimation.

[0017] In one embodiment, a computational system for auditing the results of a machine learning model, wherein the non-transitory computer-readable medium further includes instructions that, when executed by at least a processor, cause the computational system to: identify a reversal of a state estimation computation performed by a state estimation computation to form a state estimate from original time series data values; generate a set of reversal state estimates for the original time series data from a set of state estimates, wherein each reversal state estimate is generated by performing a reversal state estimate on one of the set of state estimates; wherein the reconstructed time series data values ​​for each state estimate are based on the reversal state estimate.

[0018] In one embodiment, a computing system for auditing the results of a machine learning model, wherein the non-transitory computer-readable medium further includes instructions that, when executed by at least a processor, cause the computing system to: generate an electronic verification report message indicating that the audited database has been proven to be undamaged and untampered in response to a signal that the audited database has been modified (i) and (ii) modified; and generate an electronic verification report message indicating that the audited database has been damaged or tampered with; and transmit the generated electronic verification report to a computing device so that the verification report message is stored by the computing device or displayed by the computing device. Attached Figure Description

[0019] Various systems, methods, and other embodiments of this disclosure are illustrated in conjunction with the accompanying drawings, which form a part of this specification. It will be appreciated that the element boundaries (e.g., boxes, groups of boxes, or other shapes) illustrated in the figures represent one embodiment of a boundary. In some embodiments, one element may be implemented as multiple elements, or multiple elements may be implemented as one element. In some embodiments, an element shown as an inner component of another element may be implemented as an outer component, and vice versa. Furthermore, elements may not be drawn to scale.

[0020] Figure 1 The illustration shows an example of a system associated with ensuring that the results of a machine learning model can be audited.

[0021] Figure 2 The illustration shows an example of a method associated with ensuring that the results of a machine learning model can be audited.

[0022] Figure 3 The diagram illustrates a flowchart of one embodiment of the process associated with ensuring that the results of a machine learning model can be audited.

[0023] Figure 4 The illustration shows a schematic diagram of an embodiment of storing change records from intelligent data preprocessing in a sample IDP model.

[0024] Figure 5The illustration shows a schematic diagram of one embodiment of the MSET model training process associated with ensuring that the results of the machine learning model can be audited.

[0025] Figure 6 The illustration shows a schematic diagram of one embodiment of the MSET model constraint process associated with ensuring that the results of a machine learning model can be audited.

[0026] Figure 7 The illustration shows a schematic diagram of one embodiment of the data compression process associated with ensuring that the results of a machine learning model can be audited.

[0027] Figure 8A The illustration shows a first example data report format generated by an embodiment of the data compression process.

[0028] Figure 8B The illustration shows a second example data report format generated by another embodiment of the data compression process.

[0029] Figure 9 The illustration shows a schematic diagram of an embodiment of a data reconstruction process using a first data report format.

[0030] Figure 10 The illustration shows a schematic diagram of an embodiment of a data reconstruction process using a second data report format.

[0031] Figure 11 An example computing device is illustrated, which is configured and / or programmed with one or more and / or equivalents of the example systems and methods described herein. Detailed Implementation

[0032] This article describes systems and methods for ensuring that the results of machine learning models can be audited.

[0033] Sensors, such as Internet of Things (IoT) sensors, can be added to physical devices to monitor their operation. These sensors can be numerous, especially in sensor-intensive industries like utilities, oil and gas, and manufacturing. For example, an oil refinery might include over a million sensors. The number of sensors in a utility power grid can be far greater, especially considering the monitoring and data acquisition (SCADA) sensors on utility assets such as power plants and substations, as well as sensors in advanced metering infrastructure (AMI). Data from these sensors can be stored as time series—a series of time-indexed data points, or pairs of values ​​and associated times. Time series data can be stored in time series databases—database systems optimized for storing and serving time series. Thus, the utility industry can generate very large time series databases of sensor readings, on the order of petabytes (PB) or larger.

[0034] It is important to note that systems acquiring large amounts of time-series signals from systems with physical transducers may collect erroneous time-series values ​​from faulty physical transducers / sensors. It is possible that a proportion of sensors are degrading and / or may be "stuck" in a fault, permanently emitting a single signal regardless of the input. Sensors or upstream data acquisition and / or data aggregation electronics may exhibit intermittent problems, and individual time series may contain missing values. When these types of anomalies are not detected as the time-series data is captured into a time-series database, they can affect the subsequent use of the data stored in the database. For example, the accuracy of data analysis, and in some cases the reasonableness of conclusions drawn, will be negatively impacted when the time-series data is used for product development or other scientific purposes. The systems and methods described herein also address this issue by providing signal validation and sensor operability validation for time-series databases derived from sensors monitoring critical assets.

[0035] Machine learning (ML) algorithms can be applied to sensor data stored in time-series databases to enable prediction, anomaly detection / detection, and predictive maintenance of sensor-monitored equipment. This can be termed automated prognostic monitoring. In the sensor-intensive industry, particularly in utilities, an increasingly important aspect is the ability to explain to regulators, insurance investigators, and end customers why a predictive ML algorithm flags one or more signals as anomalies, and equally importantly, why the ML did not issue an alert. Therefore, the systems and methods described in this paper provide an "auditability guarantee" for pattern recognition ML determination.

[0036] Industries that use automated prognostic monitoring systems, such as utilities, must comply with legal requirements and are largely subject to government regulations. For example, U.S. utility companies are bound by local, county, state, and federal regulations, including additional federal regulations from the U.S. Nuclear Regulatory Commission (USNRC) if the utility operates a nuclear power plant, and additional regulations from the North American Electric Reliability Corporation (NERC) and the Federal Energy Regulatory Commission (FERC). In principle, such requirements and regulations could involve submitting data for legal or regulatory purposes. However, submitting the entire petabyte-scale data upon which decisions made by these large automated systems are based is impractical, not only because of the sheer size of the datasets, but also because very little of the data is persistently stored in a way that would make such submissions possible.

[0037] Techniques for extracting data for legal and regulatory purposes often suffer from some or all of the following drawbacks: (i) the technique can use ad hoc, on-demand, or arbitrary criteria to select data without a rigorous theoretical basis; (ii) the dataset selected for the technique is insufficient to recover the original source data; and (iii) because the original source data cannot be recovered, the technique can potentially conceal violations of regulatory requirements. In fact, regarding the third drawback, this has reportedly occurred in practice.

[0038] In particular, the lack of recoverability of raw source data leads to unnecessary adversarial relationships between industry and government. For example, power companies operating in California must maintain a minimum 10% overhead buffer each month—meaning at least 10% of overhead or "reserve" capacity regardless of demand on any given date and time. Therefore, if a generating asset or grid transmission asset suddenly fails, a complete blackout is unlikely due to the overhead capacity. Utilities must submit monthly time-series data collected by the California Public Utilities Commission (PUC) to demonstrate that they maintain a minimum 10% overhead buffer each month. Whenever a utility's generating capacity falls below 10% of its overhead buffer, the PUC imposes hefty fines (which can occur multiple times per month). The PUC examines time-series signals very carefully because it automatically assumes that utilities have an incentive to manipulate some time-series values ​​so that generating capacity never appears to fall below the 10% threshold. Similarly, utilities automatically assume that PUCs have an incentive to adjust some time-series values ​​to make generation capacity appear to have dropped by a few percent below the 10% threshold, thereby incurring huge fines from utilities.

[0039] These drawbacks (including mutual distrust) are eliminated by the systems and methods described herein for ensuring that the results of machine learning models can be audited. In one embodiment of ensuring auditability, the following information should be retained, for example, in the log of changes to the database:

[0040] • Time-series data sensed (e.g., including timestamped sensor readings and users associated with sensor inputs, if any);

[0041] • Description of the sensor (e.g., the asset sensed by the sensor, the sensor's operating status at the time of measurement);

[0042] • Description of the sensed assets (e.g., the version of the asset at the time of measurement, a record of the relationship between the asset and other assets—such as a time instance graph or a time list of materials);

[0043] • The state of "knowledge" at the time of measurement (e.g., one or more specific machine learning models, specific conditions, and specific data used in the analysis); and

[0044] • The version of the machine learning model creation software and the identifier of the dataset used to develop a specific machine learning model and analyze sensor readings.

[0045] It is important to note that the sensed time-series data can be very large. In one embodiment, instead of extracting large amounts of data, a relatively small but significant (or most prominent or most important) dataset from which approximations of the original data can be recovered is sufficient for legal and regulatory purposes. A collection of saved data possessing this recoverability characteristic—that is, the recoverability from approximate original data—is referred to herein as a “tamper-proof” dataset. In one embodiment, creating a tamper-proof dataset may involve one or more techniques described in “Intelligent Preprocessing of Multi-Dimensional Time-Series Data,” U.S. Patent Application Serial No. 15 / 925,427, filed March 19, 2018, by inventors D. Gawlick, K. C. Gross, Z. H. Liu, and A. Ghoneimy, which is incorporated herein by reference in its entirety. While not necessarily preventing malicious actors from altering the original data, or even the tamper-proof dataset, the systems and methods disclosed herein actively ensure that such tampering is detected during audits conducted according to the procedures described herein.

[0046] In one embodiment, the systems and methods described herein are thus able to create approximations of the original source data using a relatively small amount of processed data—a tamper-proof dataset—which can be used to meet legal and regulatory requirements. Furthermore, the tamper-proof dataset can be used to meet these requirements while still being compatible with machine learning signal verification and sensor operability verification techniques.

[0047] In one embodiment, the processing technique used to extract tamper-proof datasets exhibits three advantageous characteristics:

[0048] • Determinism: The processed data in the tamper-proof dataset is uniquely determined by the input data and has no randomness.

[0049] • Compression: Compared to the input data, the processed data in a tamper-proof dataset is relatively small. In practice, the processed data in a tamper-proof dataset can be several orders of magnitude smaller.

[0050] • Reversibility: It is possible to recover a relatively close approximation of the original source data from the processed data in the tamper-proof dataset. In practice, the degree of approximation is determined by legal and regulatory requirements and can be adjusted to meet these requirements.

[0051] Processing techniques such as Multivariate State Estimation (MSET) exhibit deterministic, compressible, and reversible properties, and are therefore used in one embodiment of the invention. However, any processing technique used to extract a dataset that satisfies the above three properties can be used to create a tamper-proof dataset according to the systems and methods described herein.

[0052] In one embodiment, in addition to the three characteristics mentioned above, there is an automated technology for determining whether legal and regulatory requirements have been met based on tamper-proof datasets and historical provenance information (which tracks changes to the database but does not guarantee data integrity). If the requirements are not met, the system can determine appropriate legal remedies or government fines. By making legal and regulatory processes fully transparent to all parties, the motivation to conceal violations and the resulting adversarial relationships can be significantly reduced or even eliminated.

[0053] In one embodiment, the systems and methods described herein improve upon existing ML monitoring systems to which they are applied, thereby increasing the accuracy of alerts and reducing the false alarm rate for ML prognostic anomaly detection. It is important to note that these improvements can be achieved by implementing the systems and methods described herein without requiring hardware upgrades anywhere within the systems in which they are implemented. Therefore, the systems and methods described herein are immediately backward compatible with any existing IoT system. This is particularly advantageous in the power utilities, oil and gas, manufacturing, and aerospace industries, where legacy sensor data collection systems are already in place and require significant labor for upgrades.

[0054] The systems and methods described herein are based on those described in the power utility sector, but are clearly applicable anywhere that collects and uses IoT sensor time-series data, such as in the oil and gas, manufacturing, and aerospace sectors. In one embodiment, the systems and methods described herein can be applied to the processing of streaming digitized data from utility assets within power generation facilities (e.g., coal-fired power plants, oil-fired power plants, nuclear power plants, wind turbines, geothermal generators, gas turbine power plants, and others, as well as critical assets in distribution networks, such as transformers, substations, and SCADA systems).

[0055] —Auditability of time series processes used for utility forecasting—

[0056] In one embodiment, the methods and processes for ensuring that the results of a machine learning model can be audited include features such as (i) tamper-proofing, (ii) snapshot isolation, (iii) logging, (iv) activity logging, and (v) data provenance recording.

[0057] In one embodiment, tamper protection is a process or system configuration that ensures the detection of malicious data modification (tampering) or accidental data modification (corruption). In one embodiment, a compact record of any changes to the original data that would be exposed may be included in an audit report.

[0058] Outliers in time series data can be identified through machine learning (ML) processing of the raw time series data to generate estimates of the "should" values ​​within the context of the surrounding data. Neural networks (NNs) and support vector machines (SVMs) can be used for anomaly detection. MSET (and variants such as Oracle's proprietary Advanced MSET Pattern Recognition "MSET2") can also be used for anomaly detection. At the black-box level, all three methods (NN, SVM, and MSET) are nonlinear nonparametric (NLNP) regression algorithms. NLNP regression is used for prediction, anomaly detection / outcome, and predictive maintenance in time series data streams, primarily because NLNP machine learning techniques do not make assumptions about linear or nonlinear relationships between time series "signals," but rather learn these relationships empirically.

[0059] In these three NLNP machine learning methods, both NN and SVM employ a stochastic (or obviously stochastic) process to optimize the weights. For NN, stochastic optimization of the weights occurs between perceptron layers. For SVM, stochastic optimization occurs in the convex quadratic programming optimization of the regularization parameters to maintain a balance between bias and variance in the SVM estimate. For example, in both cases (NN and SVM), if pattern recognition is trained on Monday's data, the relationship between the output estimate and the original input signal will be very similar compared to if it were trained on Tuesday's data. However, if the black box is "looked" at the intermediate weights calculated for Monday versus those calculated for Tuesday, the intermediate weights will be very different. For empirically practical applications of pattern recognition only, it doesn't matter if the weights within the black box can be very different each time the black box is run, as long as the output of the black box is an accurate estimate of the underlying time series. However, for auditable applications, the stochasticity introduced by NN and SVM makes them unsuitable for extracting tamper-proof datasets because they are not deterministic and therefore irreversible.

[0060] Applying MSET to anomaly detection (and associated prediction and predictive maintenance) produces estimates that can be stored along with the raw, unprocessed time-series telemetry values. Compared to NN and SVM, MSET is a deterministic (but complex) mathematical algorithm, and as mentioned above, MSET estimates are reversible, which is crucial for tamper-proofing in auditability assurance. Therefore, in one embodiment, MSET is applied to anomaly detection in a time-series data process to make tamper-proofing part of the auditability assurance. If any raw, unprocessed data stream generated by this anomaly detection process has been modified, altered, replaced, or otherwise altered by a user for malicious purposes or due to any data corruption errors in the storage medium, then the alteration to the raw data values ​​can be detected based on the accompanying MSET estimate. Alternatively, if the raw, unprocessed data stream has not been tampered with or corrupted, the raw data values ​​can be verified or confirmed to have remained unchanged based on the accompanying estimate. This tamper-proofing authentication can be performed at any time after the MSET estimate is created. This tamper-proofing authentication is based on incorporating the deterministic, reversible MSET algorithm into the anomaly detection process described herein.

[0061] Therefore, a compact dataset used to estimate the raw values ​​of time series data can be captured at any point in time and stored along with other information about changes made during the lifecycle of the time series database, and used in auditability processes.

[0062] In one embodiment, snapshot isolation forms part of the auditability process. A temporal database provides the ability to store and retrieve any version of a record. These versions are identified by strictly incremental transaction times—the time a version is available or visible for querying and subsequent processing. Each specific version can be referred to as a snapshot. In one embodiment, a tamper-proof dataset created at a specific time can be used to verify the data values ​​of a snapshot that was available at that creation time.

[0063] In one embodiment, the specific creation time of the tamper-proof dataset can be the creation of a log entry. The logging process is configured to track all changes to the database. Each change in the database results in the creation of a log entry in the log associated with the database. Logs are generally highly resistant to data loss. For example, in a log database, data changes on the database's permanent media are not permitted, and external notifications about the database are not allowed until log data describing the changes is stored. This makes the database auditable even in the event of a failure. Logs allow the system to reconstruct any snapshot of the database without temporal support, but at the cost of a significant performance burden in snapshot queries. Each database has a copy of the most recent snapshot, while temporal databases provide many snapshots over time. For immutable data such as sensor readings, snapshots and logs can be stored as a single copy.

[0064] However, in one embodiment, auditing can use additional knowledge about user activity, such as answering questions like, "Who saw what information (in response to a query) when?", or, "Who inserted, updated, or deleted what information, when?" and, "What actions were performed within the same transaction (and how are these actions related)?" This type of information can be logged in activity log entries. Activity log entries are synchronized with standard log entries. Therefore, the activity log provides additional contextual information describing the environment that led to database changes and the subsequent creation of log entries.

[0065] In one embodiment, information describing data origination may also be included in or synchronized with standard log entries, providing further contextual information describing the environment that led to database changes and the creation of accompanying log entries. Data origination information associates derived data with corresponding inputs, processing steps, and physical processing environments. For example, origination information identifies the data that forms the basis of query results. In one embodiment, the origination process may be configured to rewrite the query to determine this data. In the context of auditing, origination metadata significantly reduces the amount of data that must be considered because it retains a record of all data that forms the basis of query results. In one embodiment, origination information may include further information such as the sensor providing the data, the operational status of the sensor, the asset associated with the sensor, the version of the asset, and a description of the relationships between the assets.

[0066] In one embodiment, the tamper-proof dataset is created in response to changes to the database, simultaneously with the creation of log entries. In one embodiment, the tamper-proof dataset is captured in the log, just like other metadata associated with the change (including activity logs and change metadata for data origination metadata). In one embodiment, the tamper-proof dataset is captured for each change to the database, thereby ensuring that any subsequent data changes are detected during auditing according to the systems and methods described herein (and incidentally, also ensuring that the configuration of the machine learning software running at the time of the change is captured and audited along with other log information). Therefore, the systems and methods disclosed herein enable auditing of raw data in any snapshot of the data, regardless of the subsequent temporal evolution of the data in the time-series database. Thus, the auditability of machine learning results in a time-series database is ensured by capturing all necessary information—log entries, activity log entries, data origination, and tamper-proof machine learning records—at each change to the database, thereby tracing the entire historical origin of the signal in the database.

[0067] —Example Environment—

[0068] Figure 1An embodiment of a system 100 associated with ensuring that the results of a machine learning model can be audited is illustrated.

[0069] In one embodiment, system 100 includes a time-series data service 105 and an enterprise network 110 connected via a network 115, such as the Internet. The time-series data service 105 is directly connected to a sensor (such as sensor 120) or a remote terminal unit (RTU) via network 125, or indirectly connected to a sensor (such as sensor 130) or an RTU via one or more upstream devices 135. In one embodiment, networks 115 and 125 are the same network, while in another embodiment, networks 115 and 125 are separate networks.

[0070] In one embodiment, the time-series data service 105 includes various systems that may include a machine learning audit assurance system 140, a sensor interface server 145, a prediction, anomaly detection, and predictive maintenance system 150, a network interface server 155, and a data repository 160. Each of these systems 140-160 is interconnected via a server-side network 165. Each of these systems 140-160 is configured with logic, such as through various software modules, for performing the functions they are described to perform. In one embodiment, systems 140-160 are implemented using dedicated computing devices. In another embodiment, one or more of systems 140-160 may be implemented using public (or shared) computing devices, even if... Figure 1 The unit is represented as a discrete unit.

[0071] In one embodiment, the time series data service 105 may be hosted by a third party and / or operated by a third party for the benefit of multiple account owners / tenants, each of whom operates a business and each of whom has an associated enterprise network 110. In one embodiment, the time series data service 105 is associated with a utility entity such as an electricity utility or with a major utility asset such as a power generation facility, substation, or other major grid component. In one embodiment, according to the systems and methods described herein, the time series data service 105 is configured with logic such as software modules to operate the time series data service 105 to (i) create and export time series databases and / or (ii) audit time series databases.

[0072] In one embodiment, sensors 120, 130 may be attached to or otherwise configured to detect the performance of one or more components of a device or system. The device or system generally includes any type of machine or facility having components that perform measurable activities. Sensors 120, 130 may include (but are not limited to): voltage sensors, current sensors, temperature sensors, pressure sensors, speed sensors, flow meter sensors, vibration sensors, microphones, electromagnetic radiation sensors, proximity sensors, gyroscopes, inclinometers, accelerometers, Global Positioning System (GPS) sensors, torque sensors, bending sensors, nuclear radiation detectors, or any of a variety of other sensors or transducers used to generate electrical signals describing the detected or sensed physical behavior.

[0073] In one embodiment, sensors 120 and 130 are connected to a sensor interface server 145 via a network 125. In one embodiment, the sensor interface server 145 is configured with logic, such as software modules, to collect readings from sensors 120 and 130 and store them as observations in a time series in, for example, a data repository 160. The sensor interface server 145 is configured to interact with the sensors, such as by exposing one or more application programming interfaces (APIs) configured to accept readings from the sensors using sensor data formats and communication protocols suitable for various sensors 120 and 130. The sensor data format will generally be specified by the sensor device. The communication protocol can be a custom protocol (such as a legacy protocol earlier than IoT implementations) or any of various IoT or machine-to-machine (M2M) protocols, such as CoAP (Co-Restricted Application Protocol), Data Distribution Service (DDS), Device Profile for Web Services (DPWS), Hypertext Transfer Protocol / Representation of State Transfer (HTTP / REST), MQ Telemetry Transport (MQTT), Universal Plug and Play (UPnP), Extensible Messaging and Presence Protocol (XMPP), ZeroMQ, and other communication protocols that the Transmission Control Protocol can carry—Internet Protocol or User Datagram Protocol (TCP / IP or UDP) transport protocols. When extended to operate via TCP / IP or UDP, SCADA protocols such as OLE for Unified Architecture for Process Control (OPC UA), Modbus RTU, RP-570, Profibus, Conitel, IEC 60870-5-101 or 104, IEC 61850, and DNP3 can also be used. In one embodiment, sensor interface server 145 polls sensors 120 and 130 to retrieve sensor readings. In one embodiment, the sensor interface server passively receives sensor readings actively transmitted by sensors 120 and 130.

[0074] In one embodiment, enterprise network 110 may be associated with a utility entity such as an electricity utility. In another embodiment, enterprise network 110 may be associated with a regulatory entity such as a government. For simplicity and clarity, enterprise network 110 is represented by a field local area network 170 to which one or more personal computers 175 or servers 180 are operatively connected, and by one or more remote user computers 185 connected to enterprise network 110 via network 115 or other suitable communication network or combination of networks. Personal computers 175 and remote user computers 185 may be, for example, desktop computers, laptop computers, tablet computers, smartphones, or other devices capable of connecting to local area network 170 or network 115 or having other synchronization capabilities. The computers of enterprise network 110 interface with time-series data service 105 via network 115 or another suitable communication network or combination of networks.

[0075] In one embodiment, remote computing systems (such as those of enterprise network 110) can access information or applications provided by time series data service 105 via web interface server 155. For example, computers 175, 180, and 185 of enterprise network 110 can request a time series database from time series data service 105. Alternatively, for example, computers 175, 180, and 185 of enterprise network 110 can audit the time series database according to the systems and methods described herein. In one embodiment, the remote computing system can send requests to and receive responses from web interface server 155. In one example, access to information or applications can be achieved using a web browser on personal computer 175 or remote user computer 185. In one example, these communications can be exchanged between web interface server 155 and server 180, and can take the form of a Remote Representation State Transfer (REST) ​​request using JavaScript Object Notation (JSON) as the data exchange format, or a Simple Object Access Protocol (SOAP) request between an XML server and a JavaScript object notation server.

[0076] In one embodiment, the data repository 160 includes one or more time-series databases configured to store and provide time-series data received by the sensor interface server 145 from the sensors 120, 130. In one embodiment, the time-series database is configured to store and provide time-series data. Database. In some example configurations, network-attached storage (NAS) devices and / or other dedicated server devices can be used to implement one or more data storage repositories 160.

[0077] In one embodiment, upstream device 135 may be a third-party service for managing IoT-connected devices. Alternatively, in one embodiment, upstream device 135 may be a gateway device configured to enable sensor 130 to communicate with sensor interface server 145 (e.g., in cases where sensor 130 does not support IoT and therefore cannot communicate directly with sensor interface server 145).

[0078] —Example methods for ensuring auditing of ML models—

[0079] In one embodiment, each step of the computer-implemented method described herein can be transmitted via the processor of one or more computing devices (such as reference 1). Figure 11 The processor 1110 shown and described performs the following operations: (i) accessing memory (such as memory 1115 and / or reference memory). Figure 11 (ii) Other computing device components shown and described) and are configured with logic (such as reference) to enable the system to perform steps of the method. Figure 11 The machine learning audit assurance logic shown and described is 1130. For example, the steps of a processor accessing and reading from or writing to memory to perform the computer-implemented methods described herein. These steps may include (i) retrieving any necessary information, (ii) calculating, determining, generating, classifying, or otherwise creating any data, and (iii) storing any calculated, determined, generated, classified, or otherwise created data. References to storage or storing refer to memory or storage devices / disks as computing devices (such as memory 1115, or storage device / disk 1135 of computing device 1105, or references). Figure 11 The storage of data structures in the remote computer 1165 shown and described.

[0080] In one embodiment, each subsequent step of the method begins in response to parsing a received signal or retrieved stored data indicating that the previous step has been performed at least to the extent necessary for the subsequent step to begin. Generally, the received signal or retrieved stored data indicates the completion of the previous step.

[0081] Figure 2 The illustration depicts one embodiment of a method 200 associated with ensuring that the results of a machine learning model can be audited. In one embodiment, the steps of method 200 are performed by any one of the computers 175, 180, 185 in the machine learning audit assurance system 140 or the enterprise network 110 (as shown in the reference). Figure 1(As shown and described). In one embodiment, the machine learning audit ensures that any one of the system 140 or computers 175, 180, 185 is a dedicated computing device (such as computing device 1105) configured with machine learning audit ensure logic 1130.

[0082] Method 200 may be initiated based on various triggers, such as receiving a signal over the network or parsing stored data indicating that: (i) a user (or administrator) of the time series data service 105 or computers 175, 180, 185 has initiated method 200; (ii) method 200 is scheduled to be initiated at a defined time or time interval; (iii) a user of the time series data service 105 or computers 175, 180, 185 (associated with a utility or regulatory entity) has requested an audit of the time series database; or (iv) some other trigger indicating that method 200 should begin. Method 200 begins at “Start” block 205 in response to parsing the received signal or retrieving the stored data and determining that the signal or stored data indicates that method 200 should begin. Processing continues to process block 210.

[0083] At process block 210, the processor retrieves a set of state estimates for the raw time series data values ​​from the audited database. The state estimates are calculated for each time series data value. Processing at process block 210 is complete, and processing continues to process block 215.

[0084] At process block 215, the processor reverses the state estimate computation for each state estimate to produce recombined time-series data values ​​for each state estimate. Processing at process block 215 is complete, and processing continues to process block 220.

[0085] At process block 220, the processor retrieves the raw time series data values ​​from the audited database. Processing at process block 220 is complete, and processing continues to process block 225.

[0086] At process block 225, the processor compares the original time series data values ​​with the reconstructed time series data values ​​in pairs to determine whether the original time series and the reconstructed time series match. Processing at process block 225 is complete, and processing continues to decision block 230.

[0087] At decision block 230, the processor evaluates whether the original time series matches the reconstructed time series. If the original time series matches the reconstructed time series (yes), then the processing at decision block 230 is complete and processing continues to process block 235. If the original time series does not match the reconstructed time series (no), then the processing at decision block 230 is complete and processing continues to process block 240.

[0088] At process block 235, the processor generates a signal indicating that the audited database has not been modified because the original time series matches the reconstructed time series. Processing at process block 235 is complete, and processing continues to "End" block 245, where process 200 ends.

[0089] At process block 240, the processor generates a signal indicating that the audited database has been modified due to a mismatch between the original time series and the reconstructed time series. Processing at process block 240 is complete, and processing continues to "End" block 245, where process 200 ends.

[0090] Each of the procedures outlined in the foregoing steps of method 200 is described in further detail elsewhere in this document.

[0091] —Preparing to reconstruct the original time series data from MSET estimates—

[0092] Figure 3 The diagram illustrates a flowchart of one embodiment of an auditability process 300 associated with ensuring the auditability of the results of a machine learning model. MSET is incorporated into the data flow process, and the MSET operation is reversed to reconstruct the original time-series data for auditing. This diagram demonstrates the auditability of the MSET application.

[0093] In one embodiment, at a high level, the auditability process 300 begins when the system retrieves the MSET estimate from the time series database. Next, the system reverses the MSET calculation to generate reconstructed time series data from the MSET estimate. As discussed above, MSET is a reversible process, meaning that performing a reverse MSET calculation on the MSET estimate will reconstruct the original time series data. The original time series data is then fed to the system from the audited database. Finally, the system compares the reconstructed time series data with the original time series data to demonstrate that if the comparison indicates a match, then there was no intentional tampering (or accidental data corruption) in the original time series data.

[0094] As discussed in more detail above, the actions described in the auditability procedure 300 can be accessed by the reference. Figure 1 And executes with the processor of one or more computing devices, including the memory, storage and / or other computing device components shown and described in 11.

[0095] In one embodiment, an archived time series database 305 is presented for auditing to determine whether the original time series data in the database is intact or has been corrupted or tampered with. The archived time series database 305 may include one or more time series signals, which are sequences of time series values ​​for all observations of the time series. The archived time series database 305 will be presented with a pairwise difference analysis 310 between the original time series data values ​​from the archived time series database 305 and the reconstructed time series values ​​derived from MSET modeling and MSET estimation of the values ​​from the archived database.

[0096] In one embodiment, the archived time-series database 305 is a snapshot of the database taken at the time changes were made to it. Furthermore, when changes are made to the database, log entries describing the changes are created, and a set of state estimates is created for later pairwise difference analysis 310 in the context of auditing the archived time-series database 305. In one embodiment, the set of state estimates is stored in association with the log entries for the audited database and the archived time-series database 305. For example, the set of state estimates may have been generated in response to one or more commands instructing changes to the audited database and stored in a data structure associated with the log entries describing the changes. Other metadata may also be stored in association with the log entries, including source metadata identifying the data on which the query results (which caused the log changes) are based.

[0097] Sensor interference includes decalibration bias, intermittent jamming failures, gain shift drift, and occasional spikes that degrade signal quality and are major causes of false positives (Type I errors) and false negatives (Type II errors) in machine learning predictions for IoT applications. The raw time-series data values ​​in the archived time-series database 305 may include some of these sensor interferences. In one embodiment, the archived time-series database 305 undergoes a series of intelligent data preprocessing steps 315 to clean the data and prepare it for MSET modeling and estimation. Preprocessing is used to mitigate the adverse effects of sensor interference. In one embodiment, intelligent data preprocessing 315 iterates over one or more time series included in the archived time-series database 305.

[0098] In one embodiment of intelligent data preprocessing 315, all observations of all signals from the archived time-series database 305 are first preprocessed, and then, for example, by using... An analytical resampling process (ARP) is optimally resampled and "coordinated" to produce an updated database of clean and optimally resampled / synchronized signals. In one embodiment, ARP may involve one or more techniques described in "Automated Analytic Resampling Process for Optimally Synchronizing Time-Series Signals," inventors K. C. Gross and GC. Wang, U.S. Patent Application Serial No. 16 / 168,193, filed October 23, 2018, which is incorporated herein by reference in its entirety.

[0099] In one embodiment, the archived time series database 305 undergoes a missing value imputation process 320 during intelligent data preprocessing 315. For each time series signal in the archived time series database 305, the missing value imputation process 320 parses the original time series signal with missing value checks. In response to missing values ​​in the parsed time series, the missing value imputation process 320 fills the missing values ​​with estimated values. In one embodiment, the estimated value is a simple interpolation. In another embodiment, the estimate is a highly accurate estimate based on the sequence correlation derived from MSET and the cross-correlation with existing values, rather than a simple interpolation. The updated, imputed time series is stored for future processing. The missing value imputation process 320 may also store a record of the positions of missing values ​​in the time series that have been filled with estimates for later reversal. In one embodiment, ARP may involve one or more techniques described in “Missing Value Imputation to Facilitate Prognostic Analysis of Time-Series Sensor Data”, inventors GC Wang, KCGross, and D. Gawlick, U.S. Patent Application Serial No. 16 / 005,495, filed June 11, 2018, which is incorporated herein by reference in its entirety.

[0100] In one embodiment, intelligent data preprocessing 315 includes a despiking process 325. In one example, the despiking process 325 may occur after missing values ​​in the original time series have been replaced by a missing value imputation process 320. In the despiking process 325, the updated time series signal is parsed through outlier checking to detect and remove data “peaks,” i.e., sudden, transient changes that do not represent accurate sensor readings. Outlier checking detects peaks in the signal by iteratively characterizing various statistical distributions of the signal (generating descriptive parameters to describe its characteristics and behavior). Time series data values ​​that are outliers based on these characterizations are marked as data value peaks. The captured peaks are temporarily replaced by the signal average. The updated time series is stored for later processing. The despiking process 325 may also store records of the values ​​and locations within the time series of the detected peaks. In one embodiment, the despiking process 325 may involve one or more techniques described in “Synthesizing High-Fidelity Signals with Spikes for Prognostic Surveillance Applications”, U.S. Patent Application Serial No. 16 / 215,345, filed December 10, 2018, by inventors GC Wang and KCGross, which is incorporated herein by reference in its entirety.

[0101] In one embodiment, intelligent data preprocessing 315 includes a dequantization process 330. For example, the dequantization process 330 may occur after data spikes have been detected by a de-spiking process 325 and removed from the time series. In the dequantization process 330, the updated signal is analyzed by a “quantization” check to determine whether data quantization—a lossy data compression technique in which data intervals are grouped or merged into a single representative value—has caused signal values ​​to switch rapidly and repeatedly between adjacent representative values. In other words, the quantization check identifies segments in the time series where observations bounce back and forth between a certain number of observation limits. Data quantization may be caused by a low-resolution “quantized” transducer or sensor. The dequantization process 330 then converts the quantized values ​​detected in the time series into high-accuracy continuous signals. These continuous signals are very close to the signal values ​​detected using a higher-resolution sensor. The updated, unquantized time series is stored for later processing. The dequantization process 330 may also store a record of the values ​​and positions of the detected quantized values ​​within the time series. In one embodiment, the dequantization process 330 may involve one or more techniques described in “Dequantizing Low-Resolution IoT Signals to Produce High-Accuracy Prognostic Indicators”, U.S. Patent No. 10,496,084, issued December 3, 2019, by inventors M. Li and K. C. Gross, which is incorporated herein by reference in its entirety.

[0102] In one embodiment, intelligent data preprocessing 315 includes a destepping process 335. For example, the destepping process 335 can occur after quantized values ​​have been detected by the dequantization process 330 and replaced in the time series. Stepping is caused by a sampling rate mismatch between the recording and detection systems, where a slower sampling rate signal simply repeats its last measurement at a higher sampling rate, resulting in all measured signals having a uniform sampling rate. The time series values ​​from the slower sampling rate sensor have a sequence of flat segments, resembling a step. Stepping is a common problem in business data archives where low-sampling-rate data is collected into higher-sampling-rate time series using simple algorithms. In one embodiment, the destepping process 335 parses the time series to identify any stepped values ​​present. The destepping process 335 "fills" the stepped portions of the signal with the higher-sampling-rate signal. In one embodiment, MSET estimation can be used to derive the higher-sampling-rate signal values. The updated, filled time series is stored for future processing. The destepping process 335 can also store a record of the values ​​and positions within the time series of detected stepped values ​​for later reversal. In one embodiment, the destairing process 335 may involve one or more techniques described in “Replacing Stair-Stepped Values ​​in Time-Series Sensor Signals With Inferential Values ​​to Facilitate Prognostic Surveillance Operations”, inventors K. C. Gross and GC. Wang, U.S. Patent Application Serial No. 16 / 128,071, filed September 11, 2018, which is incorporated herein by reference in its entirety.

[0103] In one embodiment, intelligent data preprocessing 315 includes a uniform sampling process 340. In one embodiment, the uniform sampling process 340 can occur after stepped values ​​are detected by a non-stepped process 335 and replaced in the time series. In the uniform sampling process 340, the signal is analyzed by a sampling rate check to identify whether the signal's sampling rates are different. If the signal exhibits different sampling rates (e.g., having a different number of observations within the same time period), then observations of the slower signal are resampled to match the signal's highest sampling rate. The updated, resampled time series is stored for future processing. The uniform sampling process 340 can also store records of the original, unresampled values ​​and their positions in the time series for later reversal.

[0104] In one embodiment, intelligent data preprocessing 315 includes a phase synchronization process 345. For example, the phase synchronization process 345 may occur after the uniform sampling process 340. Alternatively, in one embodiment, the phase synchronization process 345 may occur in parallel with the uniform sampling process 340. In the phase synchronization process 345, the updated signal is parsed through a correlation check to detect out-of-phase observations. Out-of-phase observations (or time signal values ​​associated with incorrect time indices in the time series) may be caused by clock synchronization differences in measuring instruments, such as sensors. The phase synchronization process 345 shifts the out-of-phase values ​​in the time domain to align them with the correct time indices. The updated phase-synchronized time series is stored for future processing. The phase synchronization process 345 may also store a record of the original placement of out-of-phase values ​​within the time series for later reversal.

[0105] In one embodiment, the unified sampling process 340 and one or more uses in the phase synchronization process An analytical resampling process (ARP) is performed, and may involve one or more of the techniques described above in “Automated Analytic Resampling Process for Optimally Synchronizing Time-Series Signals,” which is incorporated by reference above. Note that the signal sampling between time-series signals in the archived time-series database 305 may differ. For example, one time-series may have a high but regular sampling rate or intervals between observations. Another time-series may have a low but regular sampling rate or intervals between observations. Yet another time-series may have irregular or non-uniform intervals between observations. In one embodiment, the phase of the time-series signals may be adjusted such that the set of time-series signals in the archived time-series database 305 is aligned relative to the observation time. In one embodiment, the data values ​​of the time-series signals can be resampled at a new sampling interval by interpolating estimated data values ​​of observations within the time-series signals at the new sampling interval. This can be performed for multiple synchronized time-series signals to produce a shared sampling interval for the time-series signals.

[0106] The six intelligent data preprocessing processes described above (315) produce a high-quality, "cleaned," or "enhanced" version of the archived time-series database (305), which can be stored, retrieved, and used in subsequent machine learning processes. Note that other data preprocessing techniques may also be applied, or fewer than all six techniques may be performed in the intelligent data preprocessing (315). All observations from all signals are now preprocessed and optimally resampled and "coordinated" using ARP.

[0107] The Intelligent Data Preprocessing 315 process corrects common problems in typical machine learning datasets. However, each step modifies some part of the original data. A record should be maintained for each modification so that the original data can be reconstructed. As mentioned in the discussion of each of the six Intelligent Data Preprocessing 315 processes above, each process will store a record indicating the changes to the dataset. In one embodiment, the change records for all Intelligent Data Preprocessing 315 processes applied to the archived time-series database 305 will be stored in a single electronic data structure called the Intelligent Data Preprocessing (IDP) model.

[0108] Now for reference Figure 4 , Figure 4 The illustration shows a schematic 400 of an embodiment storing change records from intelligent data preprocessing 315 in example IDP model 405. IDP model 405 is a complete, ordered record of each operation performed on an archived time-series database, preparing for machine learning operations.

[0109] The missing value imputation process 320 stores records (also called tags or markers) of the locations of the estimated missing values ​​in the time series in a missing value tag data structure 410 within the IDP model 405. In one embodiment, the missing value tag data structure includes a collection of arrays associated with time series in the archived time series database 305. In the archived time series database 305, there exists a missing value tag array associated with each time series having missing values. The missing value tag array may be an array of observation index (such as time) values, where each index or time value indicates an observation in the associated time series where the missing values ​​were filled with estimated values.

[0110] The de-spiking process 325 stores records of the values ​​and locations of the detected spikes within the time series in a spike data structure 415 in the IDP model 405. In one embodiment, the spike data structure includes a collection of arrays associated with time series in the archived time series database 305. There exists a spike array associated with each time series in which spikes are detected. The array may be an array of tuples, including an observation index value (such as time) and the associated amplitude value of the spike at that observation (i.e., the erroneous sensor reading).

[0111] The records of the values ​​and positions of the quantized values ​​detected in the dequantization process 330 within the time series are stored in the smoothing model data structure 420 in the IDP model 405. In one embodiment, the smoothing model 420 includes a collection of quantized observation arrays associated with time series in the archived time series database 305. There exists a quantized observation array associated with each time series in which quantized values ​​were detected. The array may be an array of tuples, including observation index values ​​(such as time) and the associated quantized value at that observation.

[0112] The de-stepping process 335 stores records of the values ​​and positions of the detected stepped values ​​within the time series in a smoothing model data structure 420. In one embodiment, the smoothing model 420 includes stepped observation arrays associated with time series in an archived time series database 305. There exists a stepped observation array associated with each time series in which stepped values ​​are detected. The array can be an array of tuples, including observation index values ​​(such as time) and the stepped (raw or unstepped) value associated with that observation. In one embodiment, the quantized observation array and the stepped observation array are a single array containing all the raw quantized and stepped values ​​indexed by their positions in the time series. In this case, the de-stepping process can transform the stepped value tuples into a quantized observation array created by the non-quantized process 330.

[0113] The uniform sampling process 340 stores records of the original, unresampled values ​​and their placement within the time series in a timestamp sequence data structure 425 within the IDP model 405. Similarly, the phase synchronization process 345 stores records of the original placement of out-of-phase values ​​within the time series in the timestamp sequence data structure 425. In one embodiment, the timestamp sequence data structure 420 includes a collection of timestamp arrays associated with time series in the archived time series database 305. There exists a timestamp array observation array associated with each time series in which out-of-phase observations are detected. The array includes observation index values ​​(such as time) for each observation of the time series. In one embodiment, timestamp arrays are created for both the uniform sampling process 340 and the phase synchronization process 345. This can be a case where the processes are executed sequentially. When the uniform sampling process 340 and the phase synchronization process 345 are executed together, the results can be stored together in a single timestamp array.

[0114] Therefore, in one embodiment, the processor preprocesses the raw time series values ​​to mitigate the impact of sensor interference on the quality of the raw time series values, as shown in intelligent data preprocessing 315. During that intelligent data preprocessing, the processor generates a data preprocessing model that records one or more changes to the raw time series values ​​during preprocessing, as shown, for example, by IDP model 405. In one embodiment, reference... Figure 2 The process block 215 describes the reversal of the state estimation computation, which also includes retrieving a data preprocessing model, such as the IDP model 405, from a storage device or memory; and reversing the preprocessing described by the data preprocessing model for each state estimation.

[0115] Each of the six intelligent data preprocessing processes 315 described above can be reversed or otherwise undone by retrieving the original time series values ​​retained in the IDP model 405 and replacing the corresponding augmented values ​​in the augmented time series database with the original values. In one embodiment, the reversal of the various processes of intelligent data preprocessing 135 is performed in the reverse order of the order in which the intelligent data preprocessing 135 processes were executed.

[0116] —Model Training—

[0117] Refer again Figure 3 After intelligent data preprocessing 315, the enhanced time series signal was used to train the MSET model 350. Figure 5 The illustration shows a schematic diagram of one embodiment of the MSET model training process 500 associated with ensuring that the results of the machine learning model can be audited. An archived time-series database 305 is provided to a training data selection process 505. In the training data selection process 505, data is selected from the time-series data to form a training dataset. In one embodiment, a subset of the observation vectors of the time series is selected to form the training set. The selected training dataset is then stored in a training data structure 510.

[0118] Once the training data 505 has been selected and stored in the training data structure 510, intelligent data preprocessing 315 begins, and the IDP model 405 is created and stored, as referenced above. Figure 3 and Figure 4 In one embodiment, training data 505 is selected before intelligent data preprocessing 315, and the training data is selected from un-augmented data. In another embodiment, training data 505 is selected after intelligent data preprocessing 315, and the training data is selected from augmented data.

[0119] Once the selection of training data 505 and intelligent data preprocessing 315 are completed, the training vector selection process 515 and the MSET model training process 520 begin in a loop. During vector selection, a set of training vectors is chosen from the training data to provide to the MSET model for training. For example, 50 or 100 vectors can be selected. In one embodiment, the preprocessed training data is split into two parts to improve the model. For example, even-numbered observations may form the first part of the training set, and odd-numbered observations may form the second part. During the iteration of the selection process, even / odd "hopscotch" vector selection selects a set of training vectors from odd-numbered observations and then from even-numbered observations.

[0120] As mentioned above, training an MSET model, such as MSET model 350, is a deterministic mathematical process. MSET training 520 uses time-series signals representing time series records in an archived time-series database to learn correlations between time-series signals. The MSET model training process 520 uses augmented signals to identify all signals in the archived time-series database 305 that have any degree of correlation with any other signal in the archived time-series database 305. In one embodiment, the identification of correlations between signals is performed both for the entire signal domain in the archived time-series database 305 as a whole and for individual clusters of signals. This empirical clustering approach recognizes that the archived time-series database 305 of signals can originate from individual systems within a utility entity facility or individual assets within a fleet of utility equipment and systems of a utility entity. The final output of the MSET training process 520 is the trained MSET model 350. In one embodiment, the training process is an MSET2 training process, and the MSET model is an MSET2 model.

[0121] In one embodiment, vector selection 515 and the MSET training process 520 can be repeated in a loop (one or more iterations) until the MSET model 350 is stable, as indicated in decision box 525. If the MSET model 350 does not change significantly with changes in the training dataset, then the model 350 is stable (yes), training is complete, and the trained MSET model 350 is stored as a data structure for further use. If the MSET model 350 still changes significantly when the training dataset changes, then the model 350 is not yet stable (no), and an additional set of training vectors is selected at 515, and the MSET model 350 is further trained with the additional set of training vectors at 520.

[0122] Refer again Figure 3Based on the empirical correlation patterns learned between each signal and other signals in the archived time series database 305 during model training 520, the trained MSET model 350 is used to compute the MSET state estimate 355 for each signal in the archived time series database 305. The MSET estimates are very accurate, although the level of accuracy may vary depending on the training level of the MSET model.

[0123] At process block 360, the MSET estimate, MSET parameters, and training vectors are saved to a report data structure. This report data structure may be associated with a log entry in a database or data structure describing changes to the archived time-series database 305 (e.g., changes that result in a change in the archived state of the archived time-series database). In one embodiment, processes 320 through 355 are initiated and executed in response to changes to the archived time-series database to be included along with log entries describing the changes, ensuring that data tampering in the archived time-series database 305 is detectable. In one embodiment, activity log metadata describing one or more queries prior to the change and source metadata describing the underlying data of the query results are stored along with the log entries and the report data structure.

[0124] Note that reports can be generated in response to any changes to the audited database, creating a trace of a tamper-proof dataset associated with each change to the database. Therefore, historical provenance information (including the changes, activity logs describing the activities prior to the changes, and provenance information describing the basis of the information presented in response to queries associated with the changes) and records used to reconstruct the original data are logged at each log entry. Thus, the complete history of the audited database is recorded and can be reviewed for auditing purposes. Furthermore, the records are much more compact than maintaining additional snapshots of the database, resulting in significant performance improvements and enhanced portability of the data used for auditing.

[0125] In one embodiment, the MSET estimate 355 is stored together with the raw, unprocessed signal (MSET parameters) in an archived time-series database 305. Also stored in the archived time-series database 305 are “sensor operability flags,” where “1” indicates complete verification, or “0” indicates an anomalous signal was found in the sensor measuring the raw, unprocessed signal. In one embodiment, these sensor operability flags are determined based on the results of a fault detection estimate, which uses a Sequential Probability Ratio Test (SPRT) to analyze the residual between the MSET estimate and the raw, unprocessed signal value and marks that value as either anomalous or non-anomalous. When a threshold number of outliers appear in the time-series data for a particular sensor, the sensor can be marked as partially or completely inoperable (“0”) in the time series. Inoperability flags can indicate signals of fault or degradation from the sensor / transducer, jamming faults, or intermittent problems with the sensor / transducer or upstream data collection electronics or network. If the number of outliers in the time-series data for a particular sensor does not exceed a threshold number, the sensor can be marked as verified as operable (“1”) in the time series. In one embodiment, the threshold can be as low as meeting or exceeding an outlier reading in the time series, or it can be appropriately higher. In one embodiment, the threshold can be set through machine learning analysis of the time series, such as MSET analysis. In one embodiment, these flags can be set in the data structure that includes the time series.

[0126] —Model Constraints—

[0127] In some examples, the trained MSET model 350 may not be very compact. Therefore, for portability, it may be desirable to limit the size of the MSET model 350. Now refer to Figure 6 , Figure 6 The illustration shows a schematic diagram of one embodiment of the MSET model constraint process 600 associated with ensuring that the results of a machine learning model can be audited.

[0128] In one embodiment, the system initially trains a multivariate state estimation model, such as the MSET model 350, with a set of training values ​​selected from the original time series values, for example, as referenced. Figure 5As shown and described, the complexity of a trained multivariate state estimation model can be reduced by performing principal component analysis on the matrix of the trained multivariate state estimation model and restricting the trained multivariate state estimation model to the principal components of the matrix. For example, MSET model 350 can be simplified by performing principal component analysis on the MSET matrix of model 350 and restricting model 350 to the principal components of the matrix. Note that the MSET model is not only composed of the MSET matrix, but the MSET matrix is ​​the core of the MSET algorithm. As mentioned above, MSET model 350 can be an MSET2 model.

[0129] In one embodiment, the system decomposes the matrix associated with the multivariate state estimation module into a set of eigenvectors. For example, the complexity of the MSET model 350 is reduced by performing a singular value decomposition (SVD) 605 on the MSET matrix included in the MSET model 350. In one embodiment, SVD 605 consists of a sequence of eigenvectors and their associated eigenvalues. In another embodiment, the eigenvectors generated by SVD 605 are sorted in descending order by their respective associated eigenvalues. The system then stores the sorted sequence 610 of eigenvectors and their associated eigenvalues, for example, as an SVD data structure for further processing.

[0130] In one embodiment, the system selects a subset of principal eigenvectors from the set of eigenvectors stored in the SVD data structure. For example, the system selects principal eigenvectors 615 from SVD 605 based on the sorted eigenvectors and eigenvalues ​​610. The eigenvector with the largest eigenvalue is the principal eigenvector. These principal eigenvectors explain the greatest variability in the sensor data in the time series and are therefore the most useful information for state estimation modeling. In one embodiment, the processor selects the “top” N eigenvectors with the largest associated eigenvalues ​​as principal eigenvectors with their associated principal eigenvalues ​​620. Then, apart from the top N eigenvectors, the processor removes all eigenvectors and eigenvalues ​​from the SVD data structure, thereby removing all eigenvectors that are not principal eigenvectors (along with their associated eigenvalues).

[0131] In one embodiment, the system creates a “restricted” multivariate state estimation model from a subset of the principal eigenvectors. For example, the top N principal eigenvectors and their associated principal eigenvalues ​​620 are provided to the MSET model restrictor 625. The MSET model 350 is also provided to the MSET model restrictor 625. The MSET model restrictor 625 operates to restrict the MSET model 350 to the top N principal eigenvectors and their eigenvalues ​​620. This significantly reduces the amount of data required to encode the results of the MSET algorithm. In one embodiment, the MSET model restrictor 625 constructs a restricted MSET matrix from the principal eigenvectors and their associated eigenvalues ​​620. The model restrictor 625 replaces the original MSET matrix in the MSET model 350 with the restricted MSET matrix to create a restricted MSET model 630. The restricted MSET model 630 is stored as a data structure in memory or a storage device.

[0132] Refer again Figure 2 In one embodiment, the reversal of the state estimation computation also includes generating a restricted multivariate state estimation model (such as a restricted MSET model 630) through the reversal of the computation of the state estimation.

[0133] —Data Compression—

[0134] In some examples, a fully archived time-series database 305 can be very large. Therefore, for portability, it may be desirable to limit the size of the database to the parameters of the constrained MSET model 630. Figure 7 The illustration shows a schematic diagram of one embodiment of a data compression process 700 associated with ensuring that the results of a machine learning model can be audited. In the data compression process 700, the archived time series database 305 is reduced to a minimum size suitable for auditing the uncompressed or raw archived time series database 305.

[0135] In one embodiment, at process block 705, the system omits training data values ​​from the original time-series values ​​in the audited database, archived time-series database 305. For example, in one embodiment, the system creates a copy of the archived time-series database 305 that does not include the training data used to train the MSET model. In one embodiment, training data for the MSET model (such as training data 510 for MSET model 350) is removed from the copy of the archived time-series database 305. For example, all observations used to train MSET model 350 may be deleted from the copy of the archived time-series database 305. In another embodiment, the copy of the archived time-series database 305 was initially created without training data, thus avoiding the need to delete it. The system stores a reduced copy of the archived time-series database for subsequent processing.

[0136] In one embodiment, at process block 707, the system preprocesses the remaining raw time-series data values ​​to mitigate the effects of sensor interference and generates a model of how the records change the remaining raw time-series data. For example, intelligent data preprocessing 315 (as described above) can be performed on a reduced copy of the archived time-series database 305. The system forms an IDP operation model 710 similar to IDP model 405, where the IDP operation model 710 applies only to the remaining observation records retained in the reduced copy of the archived time-series database 305 after training observations have been omitted. The system stores the preprocessed, reduced copy of the archived time-series database for subsequent processing.

[0137] In one embodiment, at MSET operation block 715, the system performs state estimation (such as MSET state estimation) on the preprocessed remaining raw time series data values ​​using a restricted multivariate state estimation model to create a compressed time series database. A restricted MSET model 630 and a copy of the archived time series database 305, with training data omitted, are used to perform MSET operation 715. In one embodiment, as described above, MSET operation 715 and the restricted MSET model 630 are an MSET2 operation and an MSET2 model. MSET operation 715 (i) forms a state estimate for each observation based on the remaining parameters, and (ii) removes time series variables from the copy of the archived time series database that are not parameters of the restricted MSET model 630. Therefore, data values ​​that do not affect the MSET state estimate (data values ​​that do not significantly affect the value of the MSET state estimate) are removed from the copy of the archived time series database, further reducing the data size. The MSET state estimate formed for each observation and the remaining parameter values ​​for each observation form compressed data 720. Compressed data 720 is stored as a data structure in memory or a storage device.

[0138] Therefore, in one embodiment, once the archived time series database has been preprocessed, the MSET algorithm is performed using a restricted MSET model 630. The restricted MSET model 630, along with its MSET parameters, constitutes a compressed version of the original time series database (compressed data 720). The MSET estimate, along with the MSET model, is stored to represent the original time series data in a reduced size. If it is necessary to identify whether any original, unprocessed data stream has been modified, altered, or replaced, the system will be able to reverse the stored MSET estimate and MSET model to reconstruct the original time series data, which can then be used to verify the purported original time series data or to invalidate it, as described below.

[0139] —Data Format—

[0140] Figure 8A and Figure 8B Two example data report formats are illustrated. Each of these formats contains sufficient information to perform an audit on the database being audited (such as the original archived time-series database 305). In one embodiment, the system generates an electronic data report data structure that follows one of the two example data report formats.

[0141] In one embodiment, the system generates an electronic data report data structure comprising: a preprocessing model that records one or more changes to the original time series values ​​during preprocessing to mitigate the impact of sensor interference on the quality of the original time series values; a compressed time series database generated by performing state estimation with a restricted multivariate state estimation model and excluding values ​​of parameters that are not part of the restricted multivariate state estimation model from the compressed time series database, the restricted multivariate state estimation model being trained with a set of training values ​​selected from the original time series values; and (i) one or more of the set of training values ​​and (ii) the restricted multivariate state estimation model.

[0142] Comparing the two formats, the first data report format 800 specifies (i.e., included in the data report) all training data 510 (instead of the restricted MSET2 model 630), while the second data report format 850 specifies the restricted MSET2 model 630 (instead of training data 510). Both the first data report format 800 and the second data report format 850 specify the IDP operating model 710 and the compressed data 720.

[0143] Therefore, in one embodiment, the electronic data report data structure includes a preprocessing model (IDP operating model 710) that records one or more changes to the original time series values ​​during preprocessing to mitigate the impact of sensor interference on the quality of the original time series values ​​(as referenced). Figure 3 , Figure 4 and Figure 7 (As shown and described). The electronic data report data structure also includes a compressed time series database (compressed data 720), which is generated by performing state estimation using a restricted multivariate state estimation model and excluding values ​​of parameters from the compressed time series database that are not from the restricted multivariate state estimation model (as referenced). Figure 7 (As shown and described). The electronic data report data structure also includes one or more of the following: (i) a set of training values ​​selected from the original time series values ​​and used to train the restricted multivariate state estimation model, such as in the first data report format 800, and (ii) the restricted multivariate state estimation model, such as in the second data report format 850.

[0144] An exemplary advantage of the first data report format 800 is that it does not reveal the details of the MSET (or MSET2) auditability algorithm. An example advantage of the second data report format (850) is that it reduces the amount of data that must be provided in the report. Note that in both the first data report format 800 and the second data report format 850, the amount or volume of compressed data 720 will generally dominate the size of the report, so in practice the two formats may not differ significantly in size.

[0145] Refer again Figure 3 In one embodiment, the system reverses the MSET calculation to reconstruct the raw data, as shown in process block 365. For example, a data report conforming to one of data report formats 800 and 850 is created and can be used to reconstruct the data for auditing of the archived time series database 305. Note that the report is stored, for example, along with log entries, awaiting evaluation in the context of auditing the archived time series database 305. In some embodiments, the report may be stored for a considerable period before being used to reconstruct the raw data. In some embodiments, auditing of the archived time series database 305 may not occur, and the data report is never retrieved and used to reconstruct the raw data at process block 365. Moreover, the report may be distributed to third parties for external auditing of copies of the archived time series database held by those third parties.

[0146] —Data reconstruction using the example first data report format—

[0147] Figure 9 The illustration shows a schematic diagram of an embodiment of a data reconstruction process 900 using a first data report format 800, which is further associated with ensuring that the results of a machine learning model can be audited.

[0148] In one embodiment, during the data reconstruction process 900, a restricted MSET (or MSET2) model 630 is first trained using the IDP model 405 and training data 510 (e.g., as referenced). Figure 5 (As shown and described) are then limited (e.g., as referenced) Figure 6 (As shown and described), as indicated in process block 905. IDP model 405 is retrieved from storage and provided to training and constraint process 905. Training data 510 is read from data report 800. Since a constrained MSET model 630 is missing in the first data report format 800, a constrained MSET model 630 needs to be created.

[0149] In one embodiment, the system identifies a reversed state estimation computation that cancels the steps performed by the state estimation computation to form a state estimate from the original time-series data values. For example, the MSET (or MSET2) algorithm is reversed and applied to compressed data 720, as shown in process block 910. To reverse the MSET algorithm, the steps performed by the training model 630 to form the MSET estimate are parsed, and a sequence of discrete operations is recorded. For each discrete operation in the sequence, a reverse operation that cancels that discrete operation is identified and recorded in the sequence of reverse operations. In one embodiment, the order of the reverse operations should be the reverse of the order of the discrete operations. For example, if a discrete operation is performed first in the sequence of discrete operations, its reverse operation is performed last in the sequence of reverse operations, the second-to-last in the sequence of reverse operations is the reverse operation of the second discrete operation, and so on, so that the reverse operations of discrete operations are performed in the reverse order of the discrete operations. The system stores the sequence of reverse operations for subsequent processing.

[0150] In one embodiment, the system then generates a set of inverted state estimates for the original time series data from the set of state estimates. Each inverted state estimate is generated by performing an inverted state estimate on one of the set of state estimates, for example, as indicated by a sequence of inverted operations. For example, the system performs an inversion of the MSET calculation on those state estimates used to create state estimates stored in compressed data 720. The inverted MSET estimate for the original time series data values ​​at each observation is created from the state estimates of those values ​​and observations of other parameters stored in compressed data 720. A sequence of inverted operations is performed on the estimated data at each observation to create an inverted MSET estimate for each estimated data value.

[0151] In one embodiment, intelligent data preprocessing 315 is also reversed and applied to reverse the MSET estimate, as shown in process block 915. In one embodiment, IDP operating model 710 is read from data report 800, and the original time series values ​​retained in IDP operating model 710 are replaced with any corresponding values ​​in the reversed MSET estimate, thereby reversing the intelligent data preprocessing 315 used to compress the data and forming reconstructed data that approximates the original data 920. The reconstructed data 920 is stored for subsequent use in the audit process, including, for example, referencing... Figure 3 The verification process is shown and described in blocks 310 and 370-380. Therefore, the recombined time-series data values ​​used for each state estimate are based on the reversed state estimate.

[0152] It should be noted that the resulting reconstructed time-series data approximates the original data in the archived time-series database 305, but may differ slightly from the original data. However, this approximation can be used to verify that the archived time-series database 305 has not been tampered with or corrupted. Therefore, this approximation can be used to verify that the archived time-series database 305, which indicates compliance or violation of regulations, truly indicates such compliance or violation.

[0153] —Data reconstruction using the second example data report format—

[0154] Figure 10 The illustration shows a schematic diagram of one embodiment of a data reconstruction process 1000 using a second data report format 850, which is further associated with ensuring that the results of a machine learning model can be audited. Except that the restricted MSET model 630 does not need to be computed first (as it is already stored in the second data report format 850), the reconstruction process 1000 generally follows the same process steps as process 900. Note that in one embodiment, in process 1000, the order in which intelligent data preprocessing 315 reversal 1010 and MSET reversal 1015 are performed can be switched relative to the order in which MSET reversal 910 precedes intelligent data preprocessing 315 reversal 915 in process 900.

[0155] In one embodiment, intelligent data preprocessing 315 is first reversed and applied to the MSET estimate in compressed data 720, as shown in process block 1010. In another embodiment, intelligent data preprocessing 315 for compressed data 720 is reversed by reading the IDP operating model 710 from data report 800 and replacing the original time-series values ​​retained in the IDP operating model 710 with any corresponding values ​​in the MSET estimate in the compressed data. Thus, the reconstructed time-series data values ​​for each state estimate are based on the reversed state estimates, such as those described with reference to process block 1015.

[0156] In one embodiment, the MSET (or MSET2) algorithm is reversed and applied to compressed data 720, as shown in process block 1015, in a manner similar to that described above with reference to process block 910. For example, the system performs the inversion of MSET calculations on which state estimates are used to create state estimates stored in compressed data 720. The inverted MSET estimate of the original time-series data values ​​at each observation is created from the observed values ​​of those values' state estimates and other parameters stored in compressed data 720. The inverted MSET estimate and the replacement intelligent data preprocessing values ​​from process block 1010 form reconstructed data that approximates the original data 1020. The reconstructed data 1020 is stored for subsequent use in an audit process, which may include, for example, referencing… Figure 3The verification process is shown and described in boxes 310 and 370-380. As in the above process 900, the resulting time series data approximates the original data in the archived time series database 305, and may differ slightly from the original data, but can be used to verify that the archived time series database 305 has not been tampered with or corrupted.

[0157] —Report Tampering—

[0158] If there is no deliberate tampering (or accidental data corruption) in the "raw" time series data (the comparison indicates a statistically significant match), then the system proves that the raw data is intact; otherwise, the raw data is corrupted.

[0159] Refer again Figure 3 The system continues to audit the status (intact or corrupted / tampered) of the original time series in the archived time series database 305. In one embodiment, within the context of this audit, the correctness of the reported time series is verified using reconstructed data: the original data is compared with the reconstructed (or rebuilt) data, such as reconstructed data 920 generated from process 900 or reconstructed data 1020 generated from process 1000. The two data streams (i.e., the original time series and the reconstructed time series) are compared pairwise, as shown at the pairwise difference analyzer in process block 310. The pairwise difference analyzer 310 compares the original time series data values ​​and the reconstructed time series data values ​​at each observation of the two time series to see if the reconstructed values ​​match (or are very close to) the original values ​​within a threshold.

[0160] In one embodiment, a pairwise difference analyzer compares the original time-series data value for each observation with the reconstructed time-series data value to determine whether its variation exceeds a preset threshold amount (e.g., a percentage amount). In another embodiment, the pairwise difference analyzer compares each original time-series data value for each observation with the reconstructed time-series data value to determine whether it triggers fault detection using a trained fault detection model included in a trained restricted MSET2 model 630. The fault detection model may employ a sequential probability ratio test (SPRT) to analyze the residuals between the original time-series data value and the reconstructed time-series data value for each observation to determine whether the claimed original time-series data is anomalous.

[0161] The results of the pairwise comparisons are evaluated at decision box 370. If the two data streams match (yes), the validation passes and processing proceeds to process box 375. If the two data streams do not match (no), the validation fails and processing proceeds to process box 380.

[0162] At process block 375, a "pass" verification report indicating that the original time series data is intact is generated. In one embodiment, the pass verification report is a signal indicating that the original time series data is intact during parsing. In another embodiment, the pass verification report is a human-readable document indicating that the original time series data is intact. In yet another embodiment, in response to the pass verification report, the system generates and, either executes or transmits instructions to display an indication that the original time series data is intact on a graphical user interface (GUI). The GUI may be associated with a public utility entity or a regulatory entity. Processing in process 300 then concludes.

[0163] At process block 380, a “failed” verification report indicating that the original time series data has been corrupted or tampered with is generated. In one embodiment, the failed verification report is a signal indicating that the original time series data has been corrupted or tampered with when parsed. In another embodiment, the failed verification report is a human-readable document indicating that the original time series data has been corrupted or tampered with. In yet another embodiment, in response to the failed verification report, the system generates and, or executes or transmits instructions to display an indication that the original time series data has been corrupted or tampered with on a GUI. The GUI may be associated with a utility entity or a regulatory entity. The processing in process 300 then ends.

[0164] Therefore, in response to a signal that the audited database has not been modified, the system generates an electronic verification report message indicating that the verified database has been proven to be undisturbed and unaltered. In response to a signal that the audited database has been modified, the system generates an electronic verification report message indicating that the audited database has been corrupted or tampered with. The system can then transmit the generated electronic verification report to a computing device so that the verification report message is stored or displayed by the computing device. In one embodiment, the computing device may be associated with a utility entity or a regulatory entity.

[0165] —Software Module Examples—

[0166] Generally, software instructions are designed to be executed by a properly programmed processor. These software instructions can include, for example, computer executable code and source code that can be compiled into computer executable code. These software instructions can also include instructions written in interpreted programming languages ​​such as scripting languages.

[0167] In complex systems, such instructions are typically arranged as program modules, each of which performs a specific task, process, function, or operation. The entire set of modules can be controlled or coordinated in their operation by an operating system (OS) or other form of organizational platform.

[0168] In one embodiment, one or more of the components, functions, methods, or processes described herein are configured as modules stored in a non-transitory computer-readable medium. The modules are configured with stored software instructions that, when executed by at least one processor accessing memory or a storage device, cause a computing device to perform one or more corresponding functions as described herein.

[0169] —Cloud Systems, Multitenancy, and Enterprise Implementation Examples—

[0170] In one embodiment, the system is a computing / data processing system comprising a collection of applications or distributed applications for an enterprise organization. The applications and computing system may be configured to operate with or implemented as cloud-based networked systems, Software-as-a-Service (SaaS) architectures, or other types of networked computing solutions. In one embodiment, the system is a centralized server-side application that provides at least the functionality disclosed herein and is accessed by a number of users via a computer network through computing devices / terminals that communicate with the computing system (acting as a server).

[0171] —Computing Device Examples—

[0172] Figure 11 An example computing device 1100 is illustrated, configured and / or programmed with one or more and / or equivalents of the example systems and methods described herein. The example computing device may be a computer 1105, which includes a processor 1110, a memory 1115, and an input / output port 1120 operably connected via a bus 1525. In one example, computer 1105 may include machine learning logic 1130, configured to ensure that the results of a machine learning model can be audited, similar to... Figures 1 to 10 The logic and system shown and described herein. In various examples, logic 1130 may be implemented in hardware, a non-transitory computer-readable medium having stored instructions, firmware, and / or a combination thereof. While logic 1130 is shown as a hardware component attached to bus 1125, it should be appreciated that in other embodiments, logic 1130 may be implemented in processor 1110, stored in memory 1115, or stored in disk 1135. In one embodiment, logic 1130 or a computer is a component (e.g., structure: hardware, non-transitory computer-readable medium, firmware) for performing the described actions. In some embodiments, the computing device may be a server operating in a cloud computing system, a server configured in a Software as a Service (SaaS) architecture, a smartphone, a laptop computer, a tablet computing device, etc.

[0173] This component can be implemented, for example, as an ASIC programmed to ensure that the results of the machine learning model can be audited. It can also be implemented as stored computer-executable instructions, which are presented to the computer 1105 as data 1140 temporarily stored in memory 1115 and then executed by processor 1110.

[0174] The Logic 1130 may also provide components (e.g., hardware, non-transitory computer-readable media storing executable instructions, firmware) to ensure that the results of machine learning models can be audited.

[0175] Generally describing an example configuration of computer 1105, processor 1110 can be a variety of different processors, including dual-microprocessor and other multiprocessor architectures. Memory 1115 can include volatile memory and / or non-volatile memory. Non-volatile memory can include, for example, ROM, PROM, etc. Volatile memory can include, for example, RAM, SRAM, DRAM, etc.

[0176] Storage disk 1135 can be operatively connected to computer 1100 via, for example, an input / output (I / O) interface (e.g., a card, device) 1145 and an input / output port 1120. Disk 1135 can be, for example, a disk drive, solid-state drive, floppy disk drive, tape drive, Zip drive, flash memory card, memory stick, etc. Furthermore, disk 1135 can be a CD-ROM drive, CD-R drive, CD-RW drive, DVD ROM, etc. For example, memory 1115 can store processes 1150 and / or data 1140. Disk 1135 and / or memory 1115 can store an operating system that controls and allocates resources of computer 1105.

[0177] Computer 1105 can interact with input / output (I / O) devices via I / O interface 1145 and input / output port 1120. Input / output devices may include, for example, a keyboard 1180, microphone 1184, pointing and selection device 1182, camera 1186, video card, monitor 1170, scanner 1188, printer 1172, speaker 1174, disk 1135, network device 1155, and so on. Input / output port 1120 may include, for example, a serial port, a parallel port, and a USB port.

[0178] Computer 1105 can operate in a network environment and therefore can be connected to network device 1155 via I / O interface 1145 and / or I / O port 1120. Through network device 1155, computer 1105 can interact with network 1160. Through network 1160, computer 1105 can logically connect to remote computer 1165. Networks that computer 1105 can interact with include, but are not limited to, LANs, WANs, and other networks.

[0179] —Definitions and Other Examples—

[0180] In another embodiment, the described methods and / or their equivalents may be implemented using computer-executable instructions. Thus, in one embodiment, a non-transient computer-readable / storage medium is configured to have stored computer-executable instructions of an algorithm / executable application that, when executed by one or more machines, cause the machines (and / or associated components) to perform the methods. Example machines include, but are not limited to, processors, computers, servers operating in cloud computing systems, servers configured with a Software as a Service (SaaS) architecture, smartphones, and the like. In one embodiment, the computing device is implemented using one or more executable algorithms configured to perform any of the disclosed methods.

[0181] In one or more embodiments, the disclosed methods or their equivalents are performed by any of: computer hardware configured to perform the methods; or, computer instructions embodied in a module stored in a non-transient computer-readable medium, wherein the instructions are configured to execute an algorithm that is configured to perform the methods when executed by at least one processor of a computing device.

[0182] While the methods illustrated in the figures are shown and described as a series of boxes representing the algorithm for illustrative purposes, it should be understood that these methods are not restricted by the order of the boxes. Some boxes may appear in a different order than those shown and described, and / or may appear simultaneously with other boxes. Furthermore, example methods may be implemented using fewer boxes than are shown in all the figures. Boxes may be combined or divided into multiple actions / components. Additionally and / or alternative methods may employ additional actions not illustrated in the boxes.

[0183] The following includes definitions of the selected terms used herein. Definitions include various examples and / or forms of components that fall within the scope of the term and can be used to implement it. Examples are not intended to be restrictive. Both singular and plural forms of the terms may be included within the definitions.

[0184] References to "an embodiment," "an embodiment," "an example," "an example," etc., indicate that one or more embodiments or examples as described may include a particular feature, structure, characteristic, property, element, or limitation, but not every embodiment or example must include that particular feature, structure, characteristic, property, element, or limitation. Furthermore, repeated use of the phrase "in one embodiment" does not necessarily refer to the same embodiment, but may refer to the same embodiment.

[0185] ASIC: Application-Specific Integrated Circuit.

[0186] CD: Optical disc.

[0187] CD-R: CD is recordable.

[0188] CD-RW: CDs are rewritable.

[0189] DVD: Digital multifunction disc and / or digital video disc.

[0190] HTTP: Hypertext Transfer Protocol.

[0191] LAN: Local Area Network.

[0192] RAM: Random Access Memory.

[0193] DRAM: Dynamic RAM.

[0194] SRAM: Synchronous RAM.

[0195] ROM: Read-only memory.

[0196] PROM: Programmable ROM.

[0197] EPROM: Erasable PROM.

[0198] EEPROM: Electrically erasable PROM.

[0199] USB: Universal Serial Bus.

[0200] XML: Extensible Markup Language.

[0201] WAN: Wide Area Network.

[0202] As used herein, a “data structure” is an organization of data stored in memory, storage devices, or other computerized systems within a computing system. A data structure can be any of, for example, a data field, a data file, a data array, a data record, a database, a data table, a graph, a tree, a linked list, etc. A data structure can be formed from and contain many other data structures (e.g., a database contains many data records). Other examples of data structures are also possible according to other embodiments.

[0203] As used herein, "computer-readable medium" or "computer storage medium" means a non-transient medium that stores instructions and / or data configured to perform one or more of the disclosed functions when executed. In some embodiments, data may be used as instructions. Computer-readable media may take the form of, but is not limited to, non-volatile and volatile media. Non-volatile media may include, for example, optical discs, magnetic disks, etc. Volatile media may include, for example, semiconductor memory, dynamic memory, etc. Common forms of computer-readable media may include, but are not limited to, floppy disks, flexible disks, hard disks, magnetic tapes, other magnetic media, application-specific integrated circuits (ASICs), programmable logic devices, compact discs (CDs), other optical media, random access memory (RAM), read-only memory (ROM), memory chips or cards, memory sticks, solid-state storage devices (SSDs), flash drives, and other media in which computers, processors, or other electronic devices may operate. If each type of media is selected for implementation in one embodiment, it may include stored instructions of an algorithm configured to perform one or more of the disclosed and / or claimed functions.

[0204] As used herein, “logic” means a component implemented using computer or electrical hardware, a non-transient medium having instructions for executable application or program modules stored therein, and / or a combination thereof, to perform any function or action disclosed herein, and / or to cause a function or action from another logic, method, and / or system to be performed as disclosed herein. Equivalent logic may include firmware, a microprocessor programmed with an algorithm, discrete logic (e.g., an ASIC), at least one circuit, analog circuit, digital circuit, programmable logic device, memory device containing instructions for an algorithm, etc., any of which may be configured to perform one or more of the disclosed functions. In one embodiment, logic may include one or more gates, combinations of gates, or other circuit components capable of performing one or more of the disclosed functions. In the case of describing multiple logics, it is possible to combine multiple logics into one logic. Similarly, in the case of describing a single logic, it is possible to distribute that single logic among multiple logics. In one embodiment, one or more of these logics are corresponding structures associated with performing the disclosed and / or claimed functions. The choice of which type of logic to implement may be based on desired system conditions or specifications. For example, hardware implementation of the function would be chosen if higher speed is considered. If lower cost is a consideration, then stored instructions / executable applications will be chosen to implement the functionality.

[0205] An "operable connection," or a connection through which entities are "operably connected," is a connection capable of sending and / or receiving signals, physical communication, and / or logical communication. An operable connection may include physical interfaces, electrical interfaces, and / or data interfaces. An operable connection may include various combinations of interfaces and / or connections sufficient to allow for operable control. For example, two entities may be operably connected to transmit signals to each other directly or through one or more intermediate entities (e.g., processors, operating systems, logic, non-transient computer-readable media). Logical and / or physical communication channels can be used to create an operable connection.

[0206] As used herein, “user” includes, but is not limited to, one or more persons, computers or other devices, or a combination of these.

[0207] While the disclosed embodiments have been illustrated and described in considerable detail, they are not intended to limit the scope of the appended claims or in any way restrict them to such detail. It is certainly impossible to describe every contemplated combination of components or methods in order to describe all aspects of the subject matter. Therefore, this disclosure is not limited to the specific details or illustrative examples shown and described. Consequently, this disclosure is intended to cover changes, modifications, and variations that fall within the scope of the appended claims.

[0208] As to the extent to which the term “comprising” is used in the specific embodiments or claims, it is intended to be inclusive in a manner similar to that interpreted when the term “comprising” is used as a transitional word in the claims.

[0209] As far as the term “or” is used in the specific embodiments or claims (e.g., A or B), it is intended to mean “A or B or both.” When the applicant intends to indicate “only A or B but not both,” then the phrase “only A or B but not both” will be used. Therefore, the use of the term “or” herein is inclusive, not exclusive.

Claims

1. A computer-implemented method for auditing a database, the method comprising: Retrieve a set of state estimates for the raw time series data values ​​from the audited database, where each state estimate is generated by calculating one of the state estimates for the time series data values; A restricted multivariate state estimation model is created using the following method: Train the multivariate state estimation model using a set of training values ​​selected from the original time series values; The matrix included in the multivariate state estimation model is decomposed into a set of eigenvectors; Select a subset of the main eigenvectors from the set of eigenvectors; as well as Create a restricted multivariate state estimation model from a subset of the main eigenvectors; By generating the inverse of computation, the constrained multivariate state estimation model forms state estimates, and the state estimate computation is reversed for each state estimate to produce recombined time series data values ​​for each state estimate. Retrieve raw time series data values ​​from the audited database; The original time series data values ​​are compared in pairs with the reconstructed time series data values ​​to determine whether the original time series and the reconstructed time series match. as well as Generate a signal indicating that the audited database (i) has not been modified if the original time series matches the reconstructed time series, and (ii) has been modified if the original time series does not match the reconstructed time series.

2. The method of claim 1, further comprising: Training data values ​​derived from the original time-series data values ​​are omitted from the audited database; The remaining raw time series data values ​​are preprocessed to mitigate the effects of sensor interference and to generate a model that records the changes to the remaining raw time series data. as well as A restricted multivariate state estimation model is used to perform state estimation on the remaining raw time series data values ​​after preprocessing to create a compressed time series database.

3. The method of claim 1, further comprising: Preprocess the raw time series values ​​to mitigate the impact of sensor interference on the quality of the raw time series values; as well as Generate a data preprocessing model that records one or more changes to the original time series values ​​during preprocessing; The inversion of state estimation calculations also includes: Retrieval data preprocessing model; as well as Reverse the preprocessing described by the data preprocessing model for each state estimate.

4. The method of claim 1, further comprising generating an electronic data report data structure, the electronic data report data structure comprising: A preprocessing model that records one or more changes to the original time series values ​​during preprocessing to mitigate the impact of sensor interference on the quality of the original time series values; A compressed time series database is generated by performing state estimation using a restricted multivariate state estimation model and excluding the values ​​of parameters that are not from the restricted multivariate state estimation model from the compressed time series database. as well as (i) a set of training values ​​selected from the original time series values ​​and used to train the restricted multivariate state estimation model, and (ii) one or more of the restricted multivariate state estimation models.

5. The method of claim 1, wherein the reversal of the state estimation calculation further comprises: Identify the reverse state estimation calculation, which cancels the step performed by the state estimation calculation to form a state estimate from the original time series data values; A set of reverse state estimates is generated from the set of state estimates for the original time series data, wherein each reverse state estimate is generated by performing a reverse state estimate on one of the sets of state estimates; The recombined time series data values ​​for each state estimate are based on the reverse state estimate.

6. The method of claim 1, further comprising: Response to the audited database (i) An unmodified signal generates an electronic verification report message indicating that the audited database has been proven to be undamaged and untampered with, and (ii) The modified signal generates an electronic verification report message indicating that the audited database has been corrupted or tampered with; as well as The generated electronic verification report is transmitted to a computing device so that the verification report message is stored or displayed by the computing device.

7. A non-transitory computer-readable medium storing computer-executable instructions for auditing a database, the instructions causing the computer to: Retrieve a set of state estimates for the raw time series data values ​​from the audited database, wherein the state estimates are generated by calculating the state estimate for each time series data value; A restricted multivariate state estimation model is created using the following method: Train the multivariate state estimation model using a set of training values ​​selected from the original time series values; The matrix included in the multivariate state estimation model is decomposed into a set of eigenvectors; Select a subset of the main eigenvectors from the set of eigenvectors; as well as Create a restricted multivariate state estimation model from a subset of the main eigenvectors; By generating the inverse of computation, the constrained multivariate state estimation model forms state estimates, and the state estimate computation is reversed for each state estimate to produce recombined time series data values ​​for each state estimate. Retrieve raw time series data values ​​from the audited database; The original time series data values ​​are compared in pairs with the reconstructed time series data values ​​to determine whether the original time series and the reconstructed time series match. as well as Generate a signal indicating that the audited database (i) has not been modified if the original time series matches the reconstructed time series, and (ii) has been modified if the original time series does not match the reconstructed time series.

8. The non-transitory computer-readable medium of claim 7, wherein the instructions further cause the computer to: Preprocess the raw time series values ​​to mitigate the impact of sensor interference on the quality of the raw time series values; Create a data preprocessing model that records one or more changes to the original time series values ​​during preprocessing; State estimation is performed on the raw time series data values ​​using a restricted multivariate state estimation model to create a compressed time series database; as well as Generate a report that includes a preprocessed model, a compressed time series database, and one or more of (i) a set of training values ​​and (ii) a restricted multivariate state estimation model; The set of state estimates is the set of state estimates included in the compressed time series database in the report, and the reconstructed time series data is generated based on the report.

9. A computing system for auditing a database, the system comprising: processor; Memory, operatively connected to the processor; A sensor interface that can be operatively connected to a processor and memory; A non-transitory computer-readable medium operatively connected to a processor and memory and storing computer-executable instructions that, when executed by at least the computer's processor, cause the computer to: Retrieve a set of state estimates of raw time series data values ​​received through the sensor interface from the audited database, wherein the state estimates are generated by calculating the state estimate for each time series data value; A restricted multivariate state estimation model is created using the following method: Train the multivariate state estimation model using a set of training values ​​selected from the original time series values; The matrix included in the multivariate state estimation model is decomposed into a set of eigenvectors; Select a subset of the main eigenvectors from the set of eigenvectors; as well as Create a restricted multivariate state estimation model from a subset of the main eigenvectors; By generating the inverse of computation, the constrained multivariate state estimation model forms state estimates, and the state estimate computation is reversed for each state estimate to produce recombined time series data values ​​for each state estimate. Retrieve raw time series data values ​​from the audited database; The original time series data values ​​are compared in pairs with the reconstructed time series data values ​​to determine whether the original time series and the reconstructed time series match. as well as Generate a signal indicating that the audited database (i) has not been modified if the original time series matches the reconstructed time series, and (ii) has been modified if the original time series does not match the reconstructed time series.

10. The computing system of claim 9, wherein the non-transitory computer-readable medium further includes instructions that, when executed by at least a processor, cause the computing system to: Training data values ​​derived from the original time-series data values ​​are omitted from the audited database; The remaining raw time series data values ​​are preprocessed to mitigate the effects of sensor interference and to generate a model that records the changes to the remaining raw time series data. as well as A restricted multivariate state estimation model is used to perform state estimation on the remaining raw time series data values ​​after preprocessing to create a compressed time series database.

11. The computing system of claim 9, wherein the instructions for calculating the reverse state estimation further cause the computing system to generate an electronic data report data structure, the electronic data report data structure comprising: A preprocessing model that records one or more changes to the original time series values ​​during preprocessing to mitigate the impact of sensor interference on the quality of the original time series values; A compressed time series database is generated by performing state estimation using a restricted multivariate state estimation model and excluding the values ​​of parameters that are not from the restricted multivariate state estimation model from the compressed time series database. as well as (i) a set of training values ​​selected from the original time series values ​​and used to train the restricted multivariate state estimation model and (ii) one or more of the restricted multivariate state estimation models; The instructions used for reversing state estimation calculations also enable the computer to: The reverse state estimation computation is used to undo the steps performed by the state estimation computation; as well as For each state estimate, the preprocessing described by the preprocessing model is reversed.

12. The computing system of claim 9, wherein the non-transitory computer-readable medium further includes instructions that, when executed by at least a processor, cause the computing system to: Identify and undo the steps performed by the state estimation computation to form a state estimate from the original time series data values; A set of reverse state estimates is generated from the set of state estimates for the original time series data, wherein each reverse state estimate is generated by performing a reverse state estimate on one of the sets of state estimates; The recombined time series data values ​​for each state estimate are based on the reverse state estimate.

13. The computing system of claim 9, wherein the non-transitory computer-readable medium further includes instructions that, when executed by at least a processor, cause the computing system to: Response to the audited database (i) An unmodified signal generates an electronic verification report message indicating that the audited database has been proven to be undamaged and untampered with, and (ii) The modified signal generates an electronic verification report message indicating that the audited database has been corrupted or tampered with; and The generated electronic verification report is transmitted to a computing device so that the verification report message is stored or displayed by the computing device.

Images