Data quality-based confidence calculations for KPIs derived from time series data

Abstract

A system, computer program product, and method for providing confidence values ​​to replacement data for data having a problem indicating an error are provided, wherein the data problem, the replacement data, and the confidence values ​​are related to one or more KPIs. The method includes identifying one or more potentially erroneous data instances, and determining one or more predicted replacement values ​​for the potentially erroneous data instances. The method also includes determining a confidence value for each predicted replacement value, and resolving the one or more potentially erroneous data instances with one of the one or more predicted replacement values. The method also includes generating an explanatory basis for the resolution of the one or more potentially erroneous data instances.

Description

Technical Field

[0001] The present disclosure relates to calculating confidence values ​​for corrected data within time series data, and more particularly, to providing confidence values ​​for replacement data for data having issues indicating one or more errors, wherein the data issues, replacement data, and confidence values ​​are related to one or more key performance indicators (KPIs). Background Art

[0002] Many known entities, including commercial entities and residential entities, include systems that collect time series data from various sources, such as Internet of Things (IoT) devices, smart home devices, human activities, device activities, etc. The collected data may be analyzed to facilitate energy conservation, occupancy allocation, etc. Sometimes, a portion of the collected time series data may be erroneous due to various reasons, such as failure of a controlled device, failure of a corresponding sensing device, and problems with a data collection system, a data storage system, or a data transmission system. Summary of the invention

[0003] A system, computer program product, and method are provided for calculating confidence values ​​for corrected data within time series data.

[0004] In one aspect, a computer system for calculating confidence values ​​for corrected data within time series data is provided. The system includes one or more processing devices and at least one memory device operably coupled to the one or more processing devices. The one or more processing devices are configured to identify one or more potentially erroneous data instances in a time series data stream and determine one or more predicted replacement values ​​for the one or more potentially erroneous data instances. The one or more processing devices are further configured to determine a confidence value for each of the one or more predicted replacement values ​​and resolve the one or more potentially erroneous data instances with one of the one or more predicted replacement values. The one or more processing devices are further configured to generate an explanatory basis for the resolution of the one or more potentially erroneous data instances.

[0005] In another aspect, a computer program product for calculating a confidence value for corrected data within time series data is provided. The computer program product includes one or more computer-readable storage media and program instructions stored together on the one or more computer storage media. The product also includes program instructions for identifying one or more potentially erroneous data instances in a time series data stream. The product further includes program instructions for determining one or more predicted replacement values ​​for the one or more potentially erroneous data instances. The product also includes program instructions for determining a confidence value for each of the one or more predicted replacement values. The product further includes program instructions for resolving one or more potentially erroneous data instances with one of the one or more predicted replacement values. The product also includes program instructions for generating an explanatory basis for the resolution of the one or more potentially erroneous data instances.

[0006] In yet another aspect, a computer-implemented method for calculating a confidence value for corrected data within time series data is provided. The method includes identifying one or more potentially erroneous data instances in a time series data stream. The method also includes determining one or more predicted replacement values ​​for the one or more potentially erroneous data instances. The method further includes determining a confidence value for each of the one or more predicted replacement values. The method also includes resolving the one or more potentially erroneous data instances with a predicted replacement value in the one or more predicted replacement values. The method further includes generating an explanatory basis for the resolution of the one or more potentially erroneous data instances.

[0007] The present disclosure is not intended to describe every aspect, every implementation and / or every embodiment of the present disclosure. These and other features and advantages will become apparent from the following detailed description of the present embodiment in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings included in this application are incorporated into the specification and form a part of the specification. They illustrate embodiments of the present disclosure and are used to explain the principles of the present disclosure together with the specification. The accompanying drawings are illustrations of certain embodiments rather than limiting the present disclosure.

[0009] Figure 1 is a schematic diagram illustrating a cloud computer environment according to some embodiments of the present disclosure.

[0010] Figure 2 is a block diagram illustrating a set of functional abstraction model layers provided by a cloud computing environment according to some embodiments of the present disclosure.

[0011] Figure 3is a block diagram illustrating a computer system / server that may be used as a cloud-based support system to implement the processes described herein, according to some embodiments of the present disclosure.

[0012] Figure 4 is a schematic diagram illustrating a system for calculating confidence values ​​for corrected data within time series data according to some embodiments of the present disclosure.

[0013] Figure 5A is a flow chart illustrating a process for calculating confidence values ​​for corrected data within time series data according to some embodiments of the present disclosure.

[0014] Figure 5B yes Figure 5A A continuation of the illustrated flowchart showing a process for calculating confidence values ​​for corrected data within time series data according to some embodiments of the present disclosure.

[0015] Figure 5C yes Figure 5A and 5B A continuation of the illustrated flowchart showing a process for calculating confidence values ​​for corrected data within time series data according to some embodiments of the present disclosure.

[0016] Figure 6 is a text diagram illustrating an algorithm for identifying relevant questions according to some embodiments of the present disclosure.

[0017] Figure 7 is a text diagram illustrating an algorithm for observable box KPI analysis according to some embodiments of the present disclosure.

[0018] Figure 8 is a text diagram illustrating an algorithm for unobservable bin KPI analysis according to some embodiments of the present disclosure.

[0019] Fig. 9 is a schematic diagram illustrating a portion of a process for snapshot simulation according to some embodiments of the present disclosure.

[0020] Fig.10 is a schematic diagram illustrating a process for method-based simulation according to some embodiments of the present disclosure.

[0021] Fig.11 is a schematic diagram illustrating a process for point-based simulation according to some embodiments of the present disclosure.

[0022] Fig.12 is a text diagram illustrating an algorithm for a snapshot optimizer according to some embodiments of the present disclosure.

[0023] Fig.13is a graphical diagram illustrating KPI value extrapolation according to some embodiments of the present disclosure.

[0024] Fig.14 is a graphical / textual diagram illustrating the generation of a confidence metric according to some embodiments of the present disclosure.

[0025] Fig.15 is a graphical diagram illustrating a confidence metric according to some embodiments of the present disclosure.

[0026] Fig.16 is a text diagram illustrating a confidence metric interpretation according to some embodiments of the present disclosure.

[0027] Although the present disclosure may be modified into various modifications and alternative forms, the details thereof have been shown by way of example in the accompanying drawings and will be described in detail. However, it should be understood that the purpose is not to limit the present disclosure to the specific embodiments described. On the contrary, the present invention will cover all modifications, equivalents and substitutes that fall within the spirit and scope of the present disclosure. DETAILED DESCRIPTION

[0028] It will be readily appreciated that the components of the present embodiments as generally described herein and illustrated in the accompanying drawings may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the apparatus, system, method, and computer program product of the present embodiments, as presented in the accompanying drawings, is not intended to limit the scope of the embodiments as claimed, but is merely representative of selected embodiments. In addition, it will be appreciated that although specific embodiments have been described herein for purposes of illustration, various modifications may be made without departing from the spirit and scope of these embodiments.

[0029] References throughout the specification to "selected embodiments," "at least one embodiment," "one embodiment," "another embodiment," "other embodiments," or "embodiments" and similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the phrases "selected embodiments," "at least one embodiment," "in one embodiment," "another embodiment," "other embodiments," or "embodiments" appearing in various places in this specification are not necessarily referring to the same embodiment.

[0030] The illustrated embodiments will be better understood by referring to the drawings, wherein like parts are always represented by like reference numerals.The following description is intended to be exemplary only and only illustrates certain selected embodiments of devices, systems, and processes consistent with the claimed embodiments.

[0031] It should be understood that although the present disclosure includes detailed descriptions about cloud computing, the implementation of the teachings recorded herein is not limited to cloud computing environments. Instead, the embodiments of the present disclosure can be implemented in conjunction with any other type of computing environment now known or later developed.

[0032] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a pool of shared, configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, VMs, and services) that can be quickly provisioned and released with minimal management cost or interaction with the service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0033] Features are as follows:

[0034] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capacity (such as server time and network storage) on demand without manual interaction with the service provider.

[0035] Broad network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0036] Resource pooling: The computing resources of a provider are grouped into resource pools to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Generally, consumers cannot control or know the exact location of the provided resources, but can specify the location at a higher level of abstraction (such as country, state, or data center), thus having location independence.

[0037] Rapid elasticity: Capacity can be rapidly and elastically (in some cases automatically) provisioned to scale out quickly and released quickly to scale down quickly. To the consumer, the capacity available for provisioning often appears to be unlimited and can be purchased in any quantity at any time.

[0038] Measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being utilized.

[0039] The service model is as follows:

[0040] Software as a Service (SaaS): The capability provided to consumers is to use the provider's applications running on a cloud infrastructure. Applications can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., web-based email). Aside from limited user-specific application configuration settings, consumers neither manage nor control the underlying cloud infrastructure including networks, servers, operating systems, storage, or even individual application capabilities.

[0041] Platform as a Service (PaaS): The capability provided to consumers is to deploy applications created or acquired by consumers on the cloud infrastructure, which are created using programming languages ​​and tools supported by the provider. Consumers neither manage nor control the underlying cloud infrastructure including networks, servers, operating systems or storage, but have control over the applications they deploy and may also have control over the configuration of the application hosting environment.

[0042] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, network, and other basic computing resources on which consumers can deploy and run arbitrary software including operating systems and applications. Consumers neither manage nor control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and may have limited control over selected network components (e.g., host firewalls).

[0043] The deployment model is as follows:

[0044] Private cloud: Cloud infrastructure is run solely for an organization. It can be managed by that organization or a third party and can exist inside or outside the organization.

[0045] Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist inside or outside the organization.

[0046] Public cloud: Cloud infrastructure is available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0047] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community, or public) that remain distinct entities but are bound together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0048] The cloud computing environment is service-oriented, with characteristics centered on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is the infrastructure consisting of a network of interconnected nodes.

[0049] Reference now Figure 1 , depicts an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which a local computing device used by a cloud consumer can communicate, such as a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automobile computer system 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks (such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above). This allows the cloud computing environment 50 to provide infrastructure as a service, platform as a service, and / or software as a service without the cloud consumer having to maintain resources for them on a local computing device. It should be understood that Figure 1 The various types of computing devices 54A-N shown in the figure are merely illustrative, and the computing node 10 and cloud computing environment 50 can communicate with any type of computing device over any type of network and / or network addressable connection (eg, using a web browser).

[0050] Reference now Figure 2 , showing the cloud computing environment 50 ( Figure 1 ) provides a set of functional abstraction layers. First of all, it should be understood that Figure 2 The components, layers and functions shown are merely illustrative, and embodiments of the present invention are not limited thereto. As shown in the figure, the following layers and corresponding functions are provided:

[0051] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframes 61; servers based on RISC (Reduced Instruction Set Computer) architecture 62; servers 63; blade servers 64; storage devices 65; and networks and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0052] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0053] In one example, the management layer 80 may provide the functionality described below. Resource provisioning functionality 81 provides dynamic acquisition of computing resources and other resources for performing tasks in a cloud computing environment. Metering and pricing functionality 82 provides cost tracking of resources used within the cloud computing environment and provides bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security functionality provides identity authentication for cloud consumers and tasks and provides protection for data and other resources. User portal functionality 83 provides access to the cloud computing environment for consumers and system administrators. Service level management functionality 84 provides allocation and management of cloud computing resources to meet required service levels. Service level agreement (SLA) planning and fulfillment functionality 85 provides pre-scheduling and procurement of cloud computing resources for which future demand is predicted in accordance with the SLA.

[0054] The workload layer 90 provides examples of functions that can utilize the cloud computing environment. In this layer, examples of workloads and functions that can be provided include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom teaching provision 93; data analysis processing 94; transaction processing 95; and calculating confidence values ​​for time series data 96.

[0055] refer to Figure 3 , a block diagram of an example data processing system (referred to herein as computer system 100) is provided. System 100 can be implemented in a computer system / server at a single location, or in at least one embodiment, can be configured in a cloud-based system that shares computing resources. For example, but not limited to, computer system 100 can be used as cloud computing node 10.

[0056] Aspects of computer system 100 can be implemented in a computer system / server at a single location, or in at least one embodiment, can be configured as a cloud-based support system in a cloud-based system that shares computing resources to implement the systems, tools, and processes described herein. Computer system 100 can operate with many other general or special computer system environments or configurations. Examples of known computer systems, environments, and / or configurations suitable for use with computer system 100 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, small computer systems, large computer systems, and file systems (e.g., distributed storage environments and distributed cloud computing environments).

[0057] The computer system 100 may be described in the general context of computer system executable instructions, such as program modules, executed by the computer system 100. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer system 100 may be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in local and remote computer system storage media including memory storage devices.

[0058] like Figure 3 As shown, a computer system 100 is shown in the form of a general-purpose computing device. The components of the computer system 100 may include, but are not limited to, one or more processors or processing devices 104 (sometimes referred to as processors and processing units) (e.g., hardware processors), a system memory 106 (sometimes referred to as a memory device), and a communication bus 102 that couples various system components including the system memory 106 to the processing device 104. The communication bus 102 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. As an example and not limitation, such architectures include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standard association (VESA) local bus, and a peripheral component interconnect (PCI) bus. The computer system 100 typically includes various computer system readable media. Such media can be any available media accessible to the computer system 100, and it includes volatile and non-volatile media, removable and non-removable media. Additionally, computer system 100 may include one or more persistent storage devices 108 , communications unit 110 , input / output (I / O) unit 112 , and display 114 .

[0059] The processing device 104 is used to execute instructions of software that can be loaded into the system memory 106. The processing device 104 can be a plurality of processors, a multi-core processor, or some other type of processor, depending on the specific implementation. As used herein, the number of items means one or more items. Further, the processing device 104 can be implemented using a plurality of heterogeneous processor systems, wherein the main processor and the auxiliary processor exist together on a single chip. As another illustrative example, the processing device 104 can be a symmetric multiprocessor system containing a plurality of processors of the same type.

[0060] System memory 106 and permanent storage device 108 are examples of storage devices 116. A storage device may be any hardware capable of temporarily and / or permanently storing information, such as, but not limited to, data, program code in the form of functions, and / or other suitable information. In these examples, system memory 106 may be, for example, a random access memory or any other suitable volatile or non-volatile storage device. System memory 106 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory.

[0061] Persistent storage 108 may take various forms, depending on the particular implementation. For example, permanent storage 108 may include one or more components or devices. For example, but not limited to, permanent storage 108 may be provided for reading from and writing to a non-removable non-volatile magnetic medium (not shown and commonly referred to as a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media, etc.) may be provided. In this case, each may be connected to communication bus 102 via one or more data media interfaces.

[0062] Communications unit 110 in these examples may provide communications with other computer systems or devices. In these examples, communications unit 110 is a network interface card. Communications unit 110 may provide communications through the use of any or both physical and wireless communications links.

[0063] Input / output unit 112 may allow for input and output of data with other devices that may be connected to computer system 100. For example, input / output unit 112 may provide a connection for user input through a keyboard, a mouse, and / or some other suitable input device. Further, input / output unit 112 may send output to a printer. Display 114 may provide a mechanism for displaying information to a user. Examples of input / output unit 112 that facilitate establishing communications between various devices within computer system 100 include, but are not limited to, network cards, modems, and input / output interface cards. In addition, computer system 100 may communicate with a network adapter (in the form of a network adapter). Figure 3The computer system 100 may be configured to communicate with one or more networks (not shown) such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet). It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the computer system 100. Examples of such components include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

[0064] Instructions for the operating system, applications, and / or programs may be located in storage device 116, which communicates with processing device 104 via communication bus 102. In these illustrative examples, the instructions are in functional form on permanent storage device 108. These instructions may be loaded into system memory 106 for execution by processing device 104. The processes of the various embodiments may be performed by processing device 104 using computer-implemented instructions, which may be located in a memory such as system memory 106. These instructions are referred to as program code, computer usable program code, or computer readable program code, which may be read and executed by a processor in processing device 104. The program code in various embodiments may be embodied on different physical or tangible computer-readable media, such as system memory 106 or permanent storage device 108.

[0065] Program code 118 may be located in a functional form on a selectively removable computer readable medium 120 and may be loaded or transferred to computer system 100 for execution by processing device 104. In these examples, program code 118 and computer readable medium 120 may constitute computer program product 122. In one example, computer readable medium 120 may be a computer readable storage medium 124 or a computer readable signal medium 126. Computer readable storage medium 124 may include, for example, an optical or magnetic disk that is inserted or placed into a drive or other device that is part of permanent storage device 108 for transfer to a storage device (such as a hard drive) that is part of permanent storage device 108. Computer readable storage medium 124 may also take the form of a permanent storage device connected to computer system 100, such as a hard drive, thumb drive, or flash memory. In some cases, computer readable storage medium 124 may not be removable from computer system 100.

[0066] Alternatively, program code 118 may be transmitted to computer system 100 using computer readable signal media 126. Computer readable signal media 126 may be, for example, a propagated data signal containing program code 118. For example, computer readable signal media 126 may be an electromagnetic signal, an optical signal, and / or any other suitable type of signal. These signals may be transmitted via a communication link (e.g., a wireless communication link, a fiber optic cable, a coaxial cable, an electrical wire, and / or any other suitable type of communication link). In other words, in the illustrative examples, the communication link and / or connection may be physical or wireless.

[0067] In some demonstrative embodiments, program code 118 may be downloaded to persistent storage device 108 from another device or computer system over a network via computer readable signal media 126 for use within computer system 100. For example, program code stored in a computer readable storage medium in a server computer system may be downloaded from the server over a network to computer system 100. The computer system providing program code 118 may be a server computer, a client computer, or some other device capable of storing and transmitting program code 118.

[0068] Program code 118 may include one or more program modules that may be stored in system memory 106 (in Figure 3 118) (as an example and not limitation), and an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. The program modules of program code 118 generally perform the functions and / or methods of the embodiments as described herein.

[0069] The different components illustrated for computer system 100 are not meant to provide architectural limitations to the manner in which the different embodiments may be implemented. The different illustrative embodiments may be implemented in a computer system including components in addition to or in place of those illustrated for computer system 100.

[0070] The present invention may be a system, method and / or computer program product at any possible level of technical detail integration. The computer program product may include (one or more) computer-readable storage media having computer-readable program instructions thereon, and the computer-readable program instructions are used to cause a processor to perform various aspects of the present invention.

[0071] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, any suitable combination of electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or the aforementioned storage devices. A non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, such as a punch card or a convex structure in a groove with instructions recorded on it, and any suitable combination of the above devices. Computer readable storage medium as used herein should not be interpreted as a temporary signal itself, such as radio waves or other free propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses by optical fiber cables), or electrical signals transmitted by wires.

[0072] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0073] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0074] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and process programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In certain embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit, so as to perform various aspects of the present invention.

[0075] Various aspects of the present invention are described herein with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to embodiments of the present invention. It will be understood that each frame of the flow chart and / or block diagram and the combination of frames in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0076] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0077] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps will be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0078] Flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in the flow chart or block diagram can represent the module, segment or part of instruction, which includes one or more executable instructions for realizing the specified logical function.In some alternative embodiments, the function noted in the frame may not occur in the order noted in the figure.For example, two frames shown continuously can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the function involved.It will also be noted that the combination of the frames in each frame of the block diagram and / or flow chart illustration and the block diagram and / or flow chart illustration can be realized by a dedicated hardware-based system that performs a specified function or action or performs a combination of special hardware and computer instructions.

[0079] Many known entities, including commercial entities and residential entities, include systems that collect time series data from various sources, such as Internet of Things (IoT) devices, smart home devices, human activities, device activities, etc. The collected data may be analyzed to facilitate energy conservation, occupancy allocation, etc. Sometimes, due to various reasons, such as failure of controlled equipment, failure of corresponding sensing equipment, and problems with the data collection system, data storage system, or data transmission system, a portion of the collected time series data may be erroneous. For example, in one embodiment, an occupancy management system evaluates power usage against a peak load value, and in response to an erroneous occupancy data value, a power outage of predetermined equipment in an associated space may be erroneously initiated in order to avoid peak usage charges.

[0080] In addition, many known entities have one or more key performance indicators (KPIs), where, as used herein, a KPI refers to one or more measurable indicators associated with one or more key objectives. KPIs facilitate the achievement of key objectives by evaluating the success of meeting these key objectives. KPIs are scalable because enterprise-wide KPIs can be used, as well as lower-level sub-organization specific KPIs, such as sales KPIs, marketing KPIs, HR KPIs, IT support KPIs, and maintenance KPIs. In some embodiments, the KPIs are explicitly identified and described in one or more documents, and in some embodiments, the KPIs become explicit based on analysis of the collected data, where "hidden" KPIs can be "discovered" and previously stated KPIs can be verified.

[0081] A system, a computer program product, and a method for collecting time series data from one or more sensor devices are disclosed and described herein. In some embodiments, the system includes a data quality to KPI prediction confidence engine. The collected time series data is referred to as "raw data" and "raw time series data stream" herein. Unless otherwise specified, as discussed herein for various embodiments, each time series data stream originates from a single sensing device or from multiple sensors, wherein the stream is a combination (e.g., clustering) of outputs from each sensor or is the output of a sensor that has been auctioned or otherwise selected from multiple sensors. In addition, unless otherwise specified, the system described herein is configured to analyze multiple data streams simultaneously, without limitation on the number of data streams. Therefore, the system described herein is configured to analyze separate data streams.

[0082] In one or more embodiments, the quality of the data embedded in each data stream is analyzed, and one or more corresponding KPIs related to each data are determined through a two-step process. Initially, the quality of the raw data as data packets is transmitted from the sensor to the data inspection module, wherein the data packets are inspected by the data inspection submodule embedded in the data inspection module. In some cases, one or more data packets may include a problem that identifies the corresponding data packet as containing potential fault data. One such problem may be associated with the sampling frequency. For example, but not limited to, the data inspection submodule checks the sampling frequency of the data sensor to determine whether there are multiple sampling frequencies in the data, for example, whether there are accidental disturbances in the sampling frequency, and whether there are continuous sampling frequency changes. In addition, for example, but not limited to, the data inspection submodule checks the timestamp of the data to determine whether any timestamp is missing in the data, whether the data is missing a continuous extended duration, and whether there are timestamps in a changing format. In addition, for example, but not limited to, the data inspection submodule checks the syntactic value problem to determine whether the so-called numerical data includes an extended duration of "non-digital (NaN)" data and inappropriate numerical rounding and truncation. In addition, for example, but not limited to, the data inspection submodule checks the semantic value problem to determine whether any data includes abnormal events and noise data. Thus, the data check submodule checks the data in the stream and determines whether the data is within predetermined tolerances and whether there are any suspected errors in the data and the nature of the errors.

[0083] In some embodiments, there are two forms, that is, the processing of the raw data that the system attempts to operate to determine the data quality (as described above), and determining one or more KPI formulas that the plan is to be applied to the data. As used herein, the KPI formula includes one or more KPI characteristics, wherein the KPI characteristics also include but are not limited to the details of the formula, such as but not limited to one or more data problems, the algorithm itself, and any parameters and definitions of each KPI that the algorithm of the formula is directed to. In some embodiments, two forms are performed by the data inspection module, that is, data quality is assessed by the data inspection submodule, and the formula evaluation of KPI characterization is performed by the KPI characteristic determination submodule that is operably coupled to the data inspection submodule. In some embodiments, the KPI characteristic determination submodule is a separate module that is operably coupled to the data inspection module. Therefore, the data inspection feature is closely combined with the determination of the relevant KPI formula characteristics.

[0084] In at least some embodiments, at least a portion of such KPI formula characteristics is generally implemented as an algorithm that operates the incoming data stream to provide the user with the necessary output data and functions that support the corresponding KPI. In addition, in some embodiments, the KPI formula is located in the KPI formula submodule embedded in the KPI characteristic determination submodule. Therefore, as previously described, first check the data to verify that it is within a certain tolerance, and then determine whether there is any association between potential error data and one or more specific KPIs. In one or more embodiments, at least a portion of the collected raw data is not associated with any KPI, so for a given KPI, this erroneous raw data has no effect. Therefore, in order to perform the initial identification of related issues, a simple KPI correlation test is performed. For example, but not limited to, if one or more specific KPIs use a formula based on an average value, and the potential error data in each data stream includes an unordered timestamp, it is determined that the unordered timestamp does not have any effect on the corresponding one or more KPIs. Similarly, if one or more specific KPIs are based on a formula based on a median or based on a mode, there is an abnormal value in the corresponding data stream that does not have any effect on the corresponding KPI. Therefore, some erroneous data attributes may have no impact on a particular KPI, and such data is not relevant to the KPI-related analysis described further herein.

[0085] In some embodiments, another mechanism for determining KPI relevance that can be used is to pass at least a portion of the original time series data stream with known non-error data and suspected error data in some embodiments to one or more corresponding KPI formulas in the KPI formula submodule to generate numerical values, that is, to generate original KPI test values. Specifically, data without error values ​​can be processed to change at least one value to a known error value, thereby generating data with estimated errors, which are also passed to corresponding one or more KPIs to generate estimated KPI test values. In some embodiments, the injected errors may include but are not limited to randomly selecting some data in the original data stream and removing such random data to determine whether the missing data problem is relevant, and randomly selecting known non-error data and injecting known values ​​that extend beyond the established tolerance to determine whether the outlier problem is relevant. The estimated KPI test value is compared with the original KPI test value, and if there is enough similarity between them, the original data (that is, the problem associated with the original data) is marked as being relevant to the corresponding KPI. If there is not enough similarity between the estimated KPI test value and the original KPI test value, the original data including the above-mentioned problem is marked as being irrelevant to the corresponding KPI. Therefore, to determine whether there is any correlation between suspected or otherwise identified data errors in the original data stream, the data having the predetermined errors embedded therein is used to determine whether the erroneous data has any relevant and significant impact on the corresponding KPI formula.

[0086] In at least some embodiments, KPI characterization is performed. The basis for each KPI characterization (sometimes referred to as a KPI characterization) includes one or more KPIs, for example, one or more business-specific KPIs for business, one or more residential-specific KPIs for private residences. In some embodiments, KPIs are predetermined and described as explicit measures of success or failure to achieve specific business goals, for example. In some embodiments, KPIs are developed based on the collection and analysis of business data to determine otherwise unidentified measures for achieving business goals, thereby facilitating the identification of one or more additional KPIs. Therefore, regardless of the source, KPIs can be used to match the associated inherent attributes within the KPI with corresponding problems found in the original data, and in some cases, facilitate the identification of related problems.

[0087] In one or more embodiments, when the raw data is sent to the data inspection module, the KPI characteristic determination operation is performed in situ. In addition, since the nature of the raw data problem generated in real time is unknown in advance, data inspection and KPI characteristic determination will be performed in real time and dynamically. Therefore, the determination of the corresponding KPI with the corresponding characteristics embedded in the corresponding formula of the corresponding KPI is performed in conjunction with the determination of the problem affecting the raw data entering. At least a part of the KPI characteristic determination includes determining the nature of the corresponding KPI associated with the raw data. In some embodiments, a part of the raw data entering will not be associated with any KPI, and for the present disclosure, the data will not be further operated, and any embedded problems will be ignored and the data will be processed as is, or problem notifications will be sent to the user in one or more ways. In other embodiments, the relationship between the raw data entering and the associated KPI formula will be further determined.

[0088] In an embodiment, KPI formulas are grouped into one of two types of formulas, namely, "observable box" formulas and "unobservable box" formulas. Observable box KPI formulas are available for inspection, that is, the details are observable, and the KPI characteristic determination submodule includes an observable box module. Unobservable box KPI formulas are opaque to the operations and algorithms contained therein, for example but not limited to, each unobservable box algorithm and operation can be proprietary in nature, and each user requires a certain level of confidentiality and confidentiality for its content. The KPI characteristic determination submodule includes an unobservable box module. In some embodiments with both observable box KPI formulas and unobservable box KPI formulas, the associated algorithm checks whether the appropriate KPI characteristic formula includes one or more analyses of the surrounding raw data values ​​of the data with problems, including but not limited to maximum determination, minimum determination, average determination, median determination, and one or more of other statistical determinations (for example but not limited to standard deviation analysis). As previously described, if there is no relationship between the respective raw data and the KPI formula characteristics, no further action is taken on the data with problems according to the present disclosure. Therefore, for those issues associated with the raw data related to the KPI (both provided by the user), the properties, i.e., characteristics, of the KPI formula, i.e., observable bins or unobservable bins, are determined so that relevant data quality issues that may adversely affect the relevant KPIs can be correctly classified and subsequent optimization can be performed.

[0089] In one or more embodiments, a snapshot generator module receives the output of a data inspection module, which includes erroneous data with known embedded problems and corresponding KPI formula characteristics. The snapshot generator module is configured to simulate each data value through one or more models put into production, thereby generating a snapshot of the simulated data to facilitate the simulation of the original data. In some embodiments, method-based simulation and point-based simulation are used. Although any simulation (including the use of two simulations at the same time) can be used regardless of the nature of the problem in the erroneous data, in some embodiments, which of the two simulations is selected is based on the nature of the original data quality problem, and in some embodiments, the selection can be based on a predetermined instruction generated by a user. However, in general, method-based simulation is better configured to handle missing value problems, and point-based simulation is better configured to handle outlier problems.

[0090] For example, in some embodiments, a user's previous test run may have indicated that missing data can be determined based on whether the data is missing a continuous extended duration or whether a syntactic value problem exists (i.e., the so-called numerical data includes an extended duration of NaN data or inappropriate numerical rounding or truncation is determined). In this way, method-based simulation can provide a better analysis of the aforementioned conditions. If there is a semantic value problem, that is, some data includes abnormal events or consistent or patterned noise data, an outlier problem can be determined. In this way, point-based simulation can provide a better analysis of the aforementioned conditions. Similarly, if the user determines that method-based simulation or point-based simulation provides a better simulation for a specified situation, it may be uncertain, then for those conditions as described above where either can provide a better simulation, both simulation methods can be used.

[0091] The snapshot generator module is configured to analyze one or more remediation methods using method-based simulation, wherein each remediation method may include, for example, but not limited to, an algorithm for determining an average, a median, and the like. In addition, regardless of whether the KPI formula characteristic is an unobservable box or an observable box, the method-based simulation submodule can be used. Each remediation method includes generating one or more estimated values, which, if used with the particular remediation method, are included in the corresponding simulation snapshot as a potential solution or replacement for the erroneous value. Note that the estimated value may or may not be a potential replacement value. Since there is no predetermined concept about which remediation method provides the best or most correct replacement value for a particular current situation, multiple models are used, each of which is used to perform a corresponding remediation method. In some embodiments, the method-based simulation submodule is communicatively coupled to the KPI formula submodule. Moreover, the non-erroneous data portion used to calculate the estimated value for the erroneous data depends on a specific remediation technique. For example, if it is determined that the missing value is replaced with the average of all values, a substantially complete set of corresponding data is used in the remediation module. Alternatively, if only three surrounding values ​​are used to calculate the missing value, the remediation module only uses those surrounding values. Thus, method-based simulation is used to generate one or more simulated snapshots of non-erroneous original data and estimated values ​​for each erroneous original data value, wherein each simulated value indicates what the data value would look like when a particular remediation method is used, thereby generating multiple estimated values, each estimated value being the product of a different remediation method.

[0092] In at least some embodiments, data collection includes the use of heuristic-based features that facilitate determining patterns in data points as they are collected. As used herein, the terms "data point" and "data element" are used interchangeably. In some cases, one or more instances of raw data may appear incorrect because each data point exceeds a threshold value, where the threshold value is based on the probability of what the data point value should be based on the established data pattern. For example, but not limited to, a surface data excursion (i.e., an upward or downward data spike) may be generated by an erroneous data grouping or based on an accurate depiction of what is happening in real time. Therefore, the snapshot generator module is further configured to analyze errors using point-based simulations to determine whether seemingly erroneous data is actually erroneous data.

[0093] In one or more embodiments, data including known correct raw data and suspected potentially erroneous data points are combined into various configurations to initiate a determination of the probability that a potentially erroneous data value is correct or erroneous. Each potentially erroneous data value is individually inferred as a discrete "correct" or discrete "erroneous", and then the potentially erroneous data value is referred to as an "inferred data point" to distinguish it from the known correct raw data. In this way, the inferred data point has the transmitted raw data value and the inferred label of correct or erroneous. The remainder of the analysis focuses exclusively on the inferred data points. Specifically, the full range of all possible combinations of inferred data points collected in the above-mentioned simulation snapshots is evaluated. The generation of all possible combinations of discrete "correct" labels and discrete "erroneous" labels and the subsequent clustering of these facilitates further determination of the "best" actions, whether those actions are to correct the erroneous data or to accept the correct data. These operations take into account the accepted inaccuracies associated with raw data that may or may not be erroneous by determining the probability that one or more suspected potentially erroneous data values ​​are "correct" or "erroneous". For example, but not limited to, generating erroneous data points 2 through point-based simulation 3 or 8 combinations. In each combination, it is assumed that some erroneous values ​​are mistakenly identified as erroneous and some are correctly identified as erroneous. Therefore, for each combination, the erroneous values ​​are replaced with estimated values ​​based on a predetermined remediation method. Therefore, each combination has a different set of correct and incorrect data points and requires different estimated values ​​based on a predetermined remediation technique.

[0094] The total number of possible combinations of discrete “correct” and “wrong” inferred data points grows exponentially with the number of inferred data points (i.e., 2 x, where x = the number of inferred data values), and generating all possible combinations and processing them can be time and resource intensive. Each combination of inferred data points is a potential simulation, and processing each combination as a potential simulation simply increases the processing overhead. Therefore, the possible combinations of inferred data points described are further considered; however, the possible combinations of inferred data points are "pruned" so that only a subset of all possible combinations are further considered. Therefore, the point-based simulation submodule is operably coupled to the snapshot optimization submodule. In such an embodiment, the snapshot optimization feature is adopted by using the KPI formula characteristics determined as described above, regardless of whether the KPI formula characteristics are unobservable boxes or observable boxes. For example, but not limited to, the simulation of inferred data points can be filtered using KPI formula characteristics for maximum, minimum, average and median analysis. Therefore, the snapshot optimization module is communicatively coupled to the KPI formula submodule. Typically, only those combinations of inferred data points with estimated values ​​that successfully pass the pruning process will survive to generate simulations of corresponding suspicious point values ​​through the model, and generate corresponding simulation snapshots with non-erroneous original data and estimated values ​​for the identified erroneous data, wherein a portion of the suspected erroneous point values ​​may not actually be erroneous and will not need to be replaced.

[0095] In at least some embodiments, the simulation snapshots created by the snapshot generator module, whether method-based or point-based, are transmitted to the KPI value inference module. As described above, each simulation snapshot includes non-erroneous original data and estimated values ​​for erroneous data. Each estimated value and associated original data are provided to a corresponding KPI formula to generate a predicted replacement value, i.e., an inferred snapshot value for each estimated value in each simulation snapshot. Each inferred snapshot value is based at least in part on a corresponding KPI formula within the context of non-erroneous original data on a time series data stream. Therefore, for each simulation snapshot transmitted to the KPI value inference module, one or more predicted replacement values, i.e., inferred snapshot values, are generated.

[0096] In some embodiments, the inferred snapshot values ​​are transmitted to a confidence metric module to generate an analysis score in the form of a confidence value for each inferred snapshot value (discussed further below). For each corresponding scored inferred snapshot value for erroneous data, the best confidence value is selected, and the corresponding inferred snapshot value is now promoted to the selected KPI value to replace the erroneous data, where the selected KPI value is referred to as the inferred KPI value. Thus, the inferred KPI value is a value selected from one or more predicted replacement values ​​(i.e., inferred snapshot values) to resolve a potential erroneous data instance.

[0097] In addition, in one or more embodiments, the confidence metric module receives corresponding information to facilitate the selection of the inferred KPI value and additional information for generating an explanation of the selected inferred KPI value. Typically, the confidence metric module compares the inferred snapshot value generated by one or more of the above-mentioned simulations with the corresponding original error data. At least one of the results of the comparison is a corresponding confidence value in numerical form for each inferred snapshot value. Each confidence value applied to each snapshot of the data indicates that the corresponding inferred snapshot value is the correct prediction confidence level. A relatively low confidence value indicates that the corresponding inferred snapshot value including the inferred KPI value should not be used or used with caution. A relatively high confidence value indicates that the corresponding inferred snapshot value including the inferred KPI value should be used. The threshold for the associated confidence value can be established by the user and can also be used to train one or more models, both of which help to fully automate the selection. In addition, subsequent actions can be automatic. For example, but not limited to, for confidence values ​​below a predetermined threshold, the corresponding inferred snapshot value will not be passed for further processing in the native application utilizing the original data stream. Similarly, for confidence values ​​above a predetermined threshold, the corresponding selected inferred KPI value will be passed for further processing within the native application utilizing the raw data. Thus, the systems and methods described herein automatically correct problems with erroneous data in raw data streams in a manner that prevents unintentional actions or initiates appropriate actions based on the situation and indications of correct data.

[0098] In addition, since the confidence value of the inferred KPI value cannot be 100%, the confidence metric module includes an explanation submodule to provide an explanation basis for the solution of one or more potential erroneous data instances by providing details and evidence for selecting a specific simulation snapshot with an inferred KPI value. The explanation submodule provides such details, including but not limited to the type of problems detected in the data set, the number and nature of the simulations generated, the statistical characteristics of the scores obtained from the various simulations, and the comparison of the scores. Therefore, the confidence metric module generates various values ​​and information for the simulation snapshots from the KPI value inference module to facilitate the user to understand the properties of the distribution of the values, thereby further providing clarity in the selection of the corresponding inferred KPI value.

[0099] In some embodiments, the confidence metric module also includes a plurality of additional submodules to facilitate the generation of the above-mentioned confidence values ​​and the details and evidence supporting these values. In some of these embodiments, three confidence metric submodules are used, i.e., a confidence metric submodule based on quantity, a confidence metric submodule based on extension, and a confidence metric submodule based on quantity and extension.

[0100] The quantity-based confidence metric submodule is configured to consider the size of the value obtained from the KPI value inference module and generate associated confidence metric information. For example, whether the size of the KPI value is 50 or 1050, the confidence of the resulting KPI value may be different depending on the additional data and circumstances. The diffusion-based confidence metric submodule considers the range in which the simulated value is located and generates associated confidence metric information. That is, instead of the absolute size of the KPI value, the extended-based confidence metric uses the statistical characteristics of the KPI value, such as the mean, minimum, maximum and standard deviation, and is therefore substantially unaffected by the size. The quantity-based and extended-based confidence metric submodules consider the size and range of the value to generate associated confidence metric information.

[0101] refer to Figure 4 , a schematic diagram illustrating a system (i.e., a time series data replacement confidence system 400) configured to calculate confidence values ​​for corrected data within time series data is provided. The time series data replacement confidence system 400 is referred to herein as "system 400" to refer to any one other than the identified time series data replacement confidence system 400. The system 400 includes one or more processing devices 404 (only one is shown), which are communicatively and operably coupled to one or more memory devices 406 (only one is shown). The system 400 also includes a data storage system 408 that is communicatively coupled to the processing device 404 and the memory device 406 via a communication bus 402. In one or more embodiments, the communication bus 402, the processing device 404, the memory device 406, and the data storage system 408 are respectively similar to Figure 3 10, namely, communication bus 102, processing device 104, system memory 106, and permanent storage device 108.

[0102] In one or more embodiments, the system 400 includes a process control system 410 configured to operate any process that enables operation of the system 400 as described herein, including, but not limited to, electrical processes (e.g., energy management systems), mechanical processes (mechanical management systems), electromechanical processes (industrial manufacturing systems), and financial processes. In some embodiments, the process control system 410 is an external system communicatively coupled to the system 400. As shown and described herein, in some embodiments, the processing device 404, the memory device 406, and the data storage system 408 are communicated through the input / output unit 112 ( Figure 3 ) is communicatively coupled to a process control system 410.

[0103] The process control system 410 includes one or more process devices 412 interfaced with corresponding one or more processes, wherein the process devices 412 execute device / process control commands 414 generated by the interaction of associated programming instructions through the processing device 404 and the memory device 406. The process control system 410 also includes: a sensor suite 416, which includes sensors for monitoring the process devices 412 and corresponding processes and generating feedback 418 (such as, but not limited to, "sensor operation normal" and "sensor failure" signals) to the process devices 412; and one or more raw time series data streams 420, which include data packets representing process measurement outputs of the sensor suite 416 (hereinafter referred to as raw data 422).

[0104] The memory device 406 includes a process control algorithm and logic engine 430 configured to receive a raw time series data stream 420 to generate a device / process control command 414. Additionally, in some embodiments, the memory device 406 includes a data quality to KPI prediction confidence engine 440. In one or more embodiments, the data quality to KPI prediction confidence engine 440 includes one or more models 442 embedded therein. The system 400 also includes one or more output devices 450 communicatively coupled to the communication bus 402 to receive outputs 444 of the data quality to KPI prediction confidence engine 440. The modules and sub-modules in the data quality to KPI prediction confidence engine 440 are discussed with reference to FIG. 5.

[0105] The data storage system 408 stores data quality to KPI prediction confidence data 460, which includes, but is not limited to, raw time series data 462 (captured via the raw time series data stream 420) and confidence values ​​and explanations 464. The data storage system 408 also stores business KPIs 466 including formulas 468, attributes and characteristics 470 (used interchangeably herein), and corresponding metrics 472, wherein the formulas 468 include the characteristics 470 and the metrics 472.

[0106] refer to Figure 5A , a flow chart showing a process 500 for calculating confidence values ​​for corrected data within time series data is provided. Figure 4 , also refer to Figure 5A At least a portion of the modules and sub-modules in the data quality to KPI prediction confidence engine 440 are shown and discussed.

[0107] In one or more embodiments, the quality of the raw data 504 (which is substantially similar to the raw data 422) embedded in each raw time series data stream 420 is analyzed and the corresponding one or more KPIs associated with the corresponding raw data 504 are determined through a two-step process. Initially, the quality of the raw data 504 is transmitted as data packets from the corresponding sensors in the sensor suite 502 (which is substantially similar to the sensor suite 416) to the data inspection module 510 (which resides in the data quality to KI prediction confidence engine 440), wherein the data packets are checked by the data inspection submodule 512 embedded in the data inspection module 510. In at least some embodiments, as further discussed, the data inspection module 510 also includes an integrated KPI characteristic determination feature, thereby avoiding redundancy of the data inspection submodule 512.

[0108] In some embodiments, one or more data packets of the raw data 504 may include a problem that identifies the corresponding data packet as containing potential fault data. One such problem may be associated with the sampling frequency. For example, but not limited to, the data inspection submodule 512 checks the sampling frequency of the sensor suite 502 to determine whether there are multiple sampling frequencies in the raw data 504, for example, whether there are accidental disturbances in the sampling frequency, and whether there are continuous sampling frequency changes. In addition, for example, but not limited to, the data inspection submodule 512 checks the timestamps of the raw data 504 to determine whether any timestamps are missing in the raw data 504, whether the raw data 504 is missing a continuous extended duration, and whether there are timestamps of a changing format. In addition, for example, but not limited to, the data inspection submodule 512 checks the syntactic value problem to determine whether the so-called numerical data includes an extended duration of "non-digital (NaN)" data 504 and inappropriate numerical rounding and truncation. In addition, for example, but not limited to, the data inspection submodule 512 checks the semantic value problem to determine whether any raw data 504 includes abnormal events and noise data. Therefore, the data checking submodule 512 checks the raw data 504 in each raw time series data stream 420 and determines whether the raw data 504 is within a predetermined tolerance and whether there are any suspected errors in the raw data 504 and the nature of the errors.

[0109] In some embodiments, there are two forms, namely, processing of the raw data 504 that the system 400 attempts to operate to determine data quality (as described above), and determining one or more KPI formulas 468 that are intended to be applied to the raw data 504. As used herein, a KPI formula 468 includes one or more KPI characteristics 470, wherein the KPI characteristics 470 also includes, but is not limited to, details of the formula 468, such as, but not limited to, one or more data issues that the algorithm of the formula 468 targets, the algorithm itself, and any parameters and definitions of each KPI 466. The KPI 466 including the formula 468, the characteristics 470, and the metrics 472 are stored in the data storage system 408. In some embodiments, both forms are performed by the data inspection module 510, namely, evaluating data quality by the data inspection submodule 512, and performing the KPI formula characteristic evaluation by the KPI characteristic determination submodule 514 operatively coupled to the data inspection submodule 512. In some embodiments, the KPI characteristic determination submodule 514 is a separate module operatively coupled to the data inspection module 510. Thus, the data inspection feature and the determination of relevant KPI formula feature 470 are tightly coupled.

[0110] In at least some embodiments, at least a portion of such a KPI formula feature 470 is typically implemented as an algorithm that operates on the incoming raw time series data stream 420 to provide the user with the necessary output data and functionality to support the corresponding KPI 466. In addition, in some embodiments, the KPI formula 468 can be easily located within the KPI formula submodule 516 embedded within the KPI feature determination submodule 514, wherein such a KPI formula 468 can be input from the data storage system 408. Therefore, as previously described, the raw data 504 is first checked to verify that it is within a certain tolerance, and then, it is determined whether there is any association of any potential erroneous data with one or more specific KPIs 466. In one or more embodiments, at least a portion of the collected raw data 504 is not associated with any KPI 466, and therefore, for a given KPI 466, such erroneous raw data 504 has no impact. Therefore, in order to perform an initial identification of related issues, a simple KPI correlation test is performed. For example, but not limited to, if one or more particular KPIs 466 use a mean-based formula, and the potentially erroneous data 504 in the respective raw time series data stream 420 includes out-of-order timestamps, then determining the out-of-order timestamps does not have any impact on the corresponding one or more KPIs 466. Similarly, if one or more particular KPIs 466 are based on a median-based or mode-based formula 468, then the presence of outliers in the respective raw time series data stream 420 does not have any impact on the respective KPIs 466. Thus, some erroneous data attributes may have no impact on a particular KPI 466, and such data is not relevant to the KPI-related analysis described further herein.

[0111] In some embodiments, another mechanism for determining KPI relevance that can be used is to transmit at least a portion of the original time series data stream 420 having known non-erroneous data and suspected erroneous data in some embodiments to one or more corresponding KPI formulas 468 in the KPI formula submodule 516 to generate numerical values, that is, to generate original KPI test values. Specifically, data without erroneous values ​​can be processed to change at least one value to a known erroneous value, thereby generating data with estimated errors, which are also passed to the corresponding one or more KPI formulas 468 to generate estimated KPI test values.

[0112] refer to Figure 6 , provides a text diagram illustrating an example algorithm 600 for identifying relevant questions. Figure 4 and 5A, the algorithm 600 resides within the KPI formula submodule 516. The algorithm 600 includes a problem listing operation 602, in which a predetermined set of potential data error problems are listed for selection within the algorithm 600, and each potential data error problem includes one or more corresponding models 442. A data identification operation 604 is performed to identify which portion of the raw data 504 in the raw time series data stream 420 is to be analyzed for potential errors, potential data replacements, and confidence determinations of the replacements. In some embodiments, the data quality to KPI prediction confidence engine 440 is scalable to simultaneously examine multiple streams of the raw time series data stream 420, including but not limited to a small portion of the raw data 504, and to increase to all raw data 504 in all raw time series data streams 420. The KPI formula 468 developed by the user is identified and retrieved in the KPI formula identification and retrieval operation 606, and the selected raw data 504 to be analyzed is passed through the corresponding KPI formula 468 in the raw data to KPI formula operation 608. The influencing questions from question listing operation 602 are looped through the influencing question analysis selection algorithm 610 one at a time or simultaneously in parallel.

[0113] In one or more embodiments, a data-to-problem sub-algorithm 612 is executed, which includes injecting at least a portion of the original data 504 with estimated error data through an estimated data injection operation 614. In some embodiments, the injected errors may include, but are not limited to, randomly selecting some of the original data 504 in the original time series data stream 420 and removing such random data to determine whether the missing data problem is relevant. In addition, the injected errors may include, but are not limited to, randomly selecting known non-erroneous original data 504 and injecting values ​​that are known to extend beyond established tolerances to determine whether the outlier problem is relevant. The estimated data is transmitted through the KPI formula 468 to determine an estimated KPI test value through a KPI test value generation operation 616. The estimated KPI test value from operation 616 is compared with the original KPI test value from operation 608 through a KPI value comparison operation 618, and a problem determination operation 620 is performed based on the comparison operation 618. In some embodiments, if there is sufficient similarity between the estimated KPI test value and the original KPI test value, the original data 504 (i.e., the problem associated with the original data 504) is marked as being relevant to the corresponding KPI 466 via the KPI formula 468. If there is not sufficient similarity between the estimated KPI test value and the original KPI test value, the original data 504 including the above problem is marked as not relevant to the corresponding KPI 466 via the KPI formula 468. After executing the sub-algorithm 612 by exhausting the problems from the problem listing operation 602, the sub-algorithm 612 ends 622 and the algorithm 600 ends 624. Therefore, in order to determine whether there is any relevant relationship between the suspected or otherwise identified data errors in the original time series data stream 420, the data having the predetermined errors embedded therein is used to determine whether the erroneous data has any relevant and significant impact 468 on the corresponding KPI formula.

[0114] Reference again Figure 4 and Figure 5A , in at least some embodiments, KPI characterization is performed. The basis for each KPI characterization (sometimes referred to as a KPI characterization) includes one or more KPIs 466. For example, but not limited to, the basis for a business is one or more business-specific KPIs 466, while for a private residence, the basis is one or more residence-specific KPIs 466. In some embodiments, any entity-based KPI is used that will enable the time series data replacement confidence system 400 disclosed herein.

[0115] In some embodiments, KPIs 466 are predetermined and described, for example, as explicit measures of success or failure to achieve a particular business objective. In some embodiments, KPIs 466 are developed based on the collection and analysis of business data to determine otherwise unidentified measures for achieving a business objective, thereby facilitating the identification of one or more additional KPIs 466. Thus, regardless of the source, KPIs 466 can be used to match associated inherent attributes within corresponding KPI formulas 468 with corresponding issues found in raw data 504, and in some cases, facilitate the identification of related issues.

[0116] In one or more embodiments, the KPI characteristic determination operation is performed in situ when the raw data 504 is sent to the data inspection module 512. In addition, because the nature of the raw data issues generated in real time are not known in advance, the data inspection and KPI characteristic determination will be performed dynamically in real time. Therefore, the determination of the corresponding KPI 466 having the corresponding characteristic 470 embedded in the corresponding formula 468 of the corresponding KPI 466 is performed in conjunction with the determination of the issues affecting the incoming raw data 422. At least a portion of the KPI characteristic determination includes determining the nature of the corresponding KPI 466 associated with the raw data 504.

[0117] In some embodiments, a portion of the incoming raw data 504 will not be associated with any KPI 466, and for the present disclosure, the data will not be further manipulated, and any embedded issues will be ignored and the data processed as is, or a notification of the issue will be sent to the user in one or more ways, such as, but not limited to, via one or more output devices 450. In other embodiments, the relationship of the incoming raw data and the associated KPI formula 468 will be further determined.

[0118] In an embodiment, the KPI formulas 468 are grouped into one of two types of formulas, namely, "observable box" formulas and "unobservable box" formulas. In some embodiments having both observable box KPI formulas and unobservable box KPI formulas, the associated algorithm checks whether the appropriate KPI characteristic formula includes one or more analyses of the surrounding raw data values ​​of the data in question, including but not limited to one or more of maximum value determination, minimum value determination, mean value determination, median value determination, and other statistical determinations (such as, but not limited to, mode value determination and standard deviation analysis).

[0119] In at least some embodiments, the observable box KPI formula 468 is available for inspection, i.e., the details are observable, and the KPI characteristic determination submodule 514 includes an observable box module 518. Figure 7 , provides a text diagram showing an example algorithm 700 for observable box KPI analysis. Also refer to Figure 4 and 5A , the algorithm 700 resides within the observable box module 518. The algorithm 700 includes a KPI formula presentation operation 702, wherein the characteristics of each KPI formula 468 are clearly expressed to the user and the system 400, as described herein. The algorithm also includes a parse tree operation 704, wherein the KPI characteristics 470 are converted into an abstract syntax tree (AST) to generate the KPI characteristics 470 as an AST representation of source code in a corresponding programming language, so that when the details of the KPI 466 are available, the various code blocks can be parsed and understood as nodes in the AST. Figure 7 As shown, algorithm 700 includes a first sub-algorithm, namely, a function analysis operation 706 configured to determine whether a particular node in the AST is a function (eg, but not limited to, a mathematical operation as further discussed).

[0120] exist Figure 7 In the illustrated embodiment, a second sub-algorithm, namely a median determination operation 708, is performed for those KPI formula characteristics 470 that define the median determination of the raw data 504, so that a KPI characteristic assignment operation 710 is performed, in which case the assigned KPI characteristic 470 is the "median" for subsequent portions of the process 500. The median determination operation 708 is then terminated 712. In some embodiments, the algorithm includes one or more other portions of the first sub-algorithm for other types of KPI characteristics, such as, but not limited to, maximum determination, minimum determination, mean determination, and other statistical determinations (such as, but not limited to, mode value determination and standard deviation analysis). Figure 7 In the illustrated embodiment, a third sub-algorithm, namely, an average determination operation 714, is performed for the KPI formula characteristic 470 that defines the average determination of the raw data 504, so that a KPI characteristic assignment operation 716 is performed, in which case the assigned KPI characteristic 470 is the "average" for subsequent portions of the process 500. The average determination operation 714 then ends 718. Similarly, any remaining possible KPI formula characteristics 470 are determined as described above. After the function analysis operation 706 is completed, the process ends 720.

[0121] Further, in one or more embodiments, if Figure 7 As shown, algorithm 700 includes a fourth sub-algorithm, namely, binary operation analysis operation 722, which is configured to determine whether a particular node in the AST is a binary operation, such as, but not limited to, a mathematical operation that uses two elements or operands to create another element. Figure 7In the illustrated embodiment, a fifth sub-algorithm, a division sub-algorithm 724, is executed for a KPI formula property 470 that defines a division operation of the raw data 504. The division operation includes a sixth sub-algorithm, an integral sum operand and a len operand, or integral average sub-algorithm 726, wherein the len operand or operation provides the number of terms being summed, so that a KPI property assignment operation 728 is performed, in which case the assigned KPI property 470 is an "average" for subsequent portions of the process 500. The integral average sub-algorithm 726 ends 730, the division sub-algorithm 724 ends 732, and the binary operation sub-algorithm 722 ends 734. If the user requires other operations in addition to function operations and binary operations, an open sub-algorithm 736 is shown. When all corresponding observable box operations associated with the respective KPI 466 are identified, the parse tree operation 704 ends 738.

[0122] In at least some embodiments, the unobservable bin KPI formula 468 is opaque to the operations and algorithms contained therein, for example, but not limited to, individual unobservable bin algorithms and operations may be proprietary in nature, and individual users may require a certain level of secrecy and confidentiality regarding their contents. In some embodiments, such unobservable bin formulas may take the form of an application programming interface (API). Thus, one mechanism for determining KPI formula characteristics 470 within the unobservable bin KPI formula 468 includes repeatedly sampling the raw data 504 to test the raw data 504 through simulation of the formula. Thus, the KPI characteristic determination submodule 514 includes the unobservable bin module 520.

[0123] refer to Figure 8 , provides a text diagram illustrating an example algorithm 800 for unobservable bin KPI analysis. Figure 4 and 5A, the algorithm 800 resides within the unobservable bin module 520. In at least some embodiments, the algorithm 800 includes a data subset generation operation 802, in which the original data 504 is divided into K data subsets, each having M data points, where M is a predetermined constant. For example, but not limited to, a string of 100 data points can be divided into five subsets, each with 20 points. Generating such subsets facilitates determining whether a particular error recurs or whether it is a single instance of an error (i.e., a one-time error). The algorithm 800 also includes a KPI formula feature listing operation 804, which is configured to identify all potential KPI formula features 470 that can be used within the unobservable bin calculation. As previously described herein, such KPI formula features 470 include, but are not limited to, one or more of the following: mean determination ("mean"), median determination ("median"), mode determination ("mode"), maximum determination ("maximum"), minimum determination ("minimum"), and other statistical determinations (such as, but not limited to, standard deviation analysis). Each of these KPI formula features 470 will be checked through one or more unobservable box model based simulations that are not directly related to simulation modeling discussed further herein with respect to snapshot generation to identify potential issues with erroneous data.

[0124] In one or more embodiments, a raw KPI evaluation operation 806 is performed, wherein each data element in each data subset is processed by using a corresponding unobservable box model, wherein such a model has not yet been determined. As used herein, the term "data point" and the term "data element" are used interchangeably. Therefore, in an embodiment of 100 data points or data elements of the raw data 504, there will be 100 corresponding KPI values, that is, 20 KPI values ​​for each of the 5 subsets of the raw data 504. In this way, 100 process data elements are processed by the unobservable box formula, no matter what they may be, thereby generating 100 raw KPI values ​​by the actual unobservable box formula. In addition, in some embodiments, a correlation operation 808 including a simulation / correlation sub-algorithm 810 is performed. Specifically, in one or more embodiments, a simulated KPI evaluation operation 812 is performed, wherein each data element of the raw data 504 is analyzed by using a corresponding model of each KPI formula feature 470 identified in the KPI formula feature listing operation 804. A raw KPI value to simulated KPI value correlation operation 814 is performed, wherein each of the raw KPI values ​​is compared to each corresponding simulated KPI value generated by each model of the KPI formula features 470 identified in the KPI formula feature listing operation 804. Thus, for an embodiment having 100 data elements, there will be 100 correlations for each KPI formula feature 470 identified in the KPI formula feature listing operation 804. In some embodiments, a statistical evaluation of each set of correlated data elements is performed to determine the strength of the correlation, such as, but not limited to, a weak correlation and a strong correlation, wherein the definition of each correlation can be established by a user. A strong correlation indicates that the simulated KPI formula follows the actual unobservable bin KPI formula 468. A weak correlation indicates that the simulated KPI formula is inconsistent with the actual unobservable bin KPI formula 468. Once the processing of the correlations is completed, the simulation / correlation sub-algorithm 810 is terminated 816. The algorithm 800 for unobservable bin KPI analysis includes a KPI formula feature selection operation 818, wherein the highest correlation feature is selected. Once the unobservable bin KPI formula is determined, the algorithm 800 ends 820 .

[0125] The output 522 of the data checking module 510 includes an analysis of the original data 504 to determine whether there are any data errors therein, and if so, determine the affected KPI formula characteristics 470. If there are no errors, the corresponding data is no longer processed by the process 500, wherein the operation in the KPI characteristic determination submodule 514 is not called, and there is no output 522. If there are data errors in the original data 504, the output 522 is transmitted to the determination operation 524, which determines 524 whether the data problem is related to the identified KPI based on the analysis of the KPI characteristic determination submodule 514. As previously described, if there is no relationship between the corresponding original data 504 and the KPI formula characteristics 470, then according to the present disclosure, a "no" determination is generated, and no further action is taken for the data with the problem. If necessary, the user can choose to take other actions for the data error. For a "yes" determination, i.e., for those data error issues associated with the original data 504 that are relevant to the KPI (both provided by the user), the output 526 of the determination operation 524 is sent for further processing with the corresponding attributes, wherein the output 526 is substantially similar to the output 522. Thus, when the characteristic 470 (whether an observable bin or an unobservable bin) of the KPI formula 468 for the erroneous data is determined to be adversely affecting the relevant KPI, the error is further analyzed so that the error can be correctly classified and subsequent optimization can be performed.

[0126] refer to Figure 5B , provides Figure 5A Continuation of the flowchart shown, which further illustrates a process 500 for calculating confidence values ​​for corrected data within time series data. Figure 4 In one or more embodiments, the process 500 further includes sending the output 526 to a snapshot generator module 530. The snapshot generator module 530 receives the output 526 of the data check module 510, which includes the identification of the erroneous data with known embedded problems and the corresponding KPI formula characteristics 470. The snapshot generator module 530 is configured to simulate each data value through one or more models put into production, thereby generating a snapshot of the simulated data to facilitate the simulation of the original data 504.

[0127] refer to Fig. 9 , provides a schematic diagram illustrating a portion of a process 900 for snapshot simulation using a snapshot generator module 904 that is substantially similar to snapshot generator module 530. Also refer to Figure 5B , further evaluating the raw data 902 substantially similar to the raw data 504 transmitted to the snapshot generator module 904 via output 526. Figure 4The model 442 shown in FIG. 4 is used to process the raw data 902 (in which the erroneous data problem is embedded) to generate a plurality of simulated snapshots 906 including simulated data, as further discussed herein. The simulated data snapshots 906 are then used for KPI inference 908 and confidence measurements 910, which are then used to generate the simulated data snapshots 906. Fig. 9 It is shown for context only.

[0128] Reference again Figure 4 and Figure 5B In some embodiments, method-based simulation and point-based simulation are used. Although any simulation (including the use of both simulations simultaneously) can be used regardless of the nature of the problem in the erroneous data, in some embodiments, the selection of which of the two simulations is based on the nature of the original data quality problem, and in some embodiments, the selection can be based on a predetermined instruction generated by a user. However, in general, method-based simulation is better configured to handle missing value problems, and point-based simulation is better configured to handle outlier problems. For example, in some embodiments, a user's previous trial run may have indicated that missing data can be determined based on whether the data has lost a continuous extended duration or whether a syntactic value problem exists (i.e., the so-called numerical data includes an extended duration of NaN data or determines inappropriate numerical rounding or truncation). In this way, method-based simulation can provide a better analysis of the aforementioned situation. If there is a semantic value problem, i.e., some data includes abnormal events or consistent or patterned noise data, an outlier problem can be determined. In this way, point-based simulation can provide a better analysis of the aforementioned situation. Similarly, if a user is unsure whether method-based simulation or point-based simulation provides a better simulation for a given situation, both simulation methods may be used for those situations as described above where either may provide a better simulation.

[0129] In one or more embodiments, the snapshot generator module 530 is configured to analyze one or more remediation methods using method-based simulation, where each remediation method may include, for example, but not limited to, an algorithm for determining an average, a median, etc. based at least in part on a corresponding KPI 466 affected by a data error. However, the remediation methods need not be limited to the KPI formula characteristics 470. The snapshot generator module 530 includes a method-based simulation submodule 534 that can be used regardless of whether the KPI formula characteristics 470 are unobservable bins or observable bins.

[0130] refer to Fig.10 , provides a schematic diagram showing a process 1000 for generating a simulation based on the method. Figure 4 and 5B, a method-based simulation is generated by a method-based simulation submodule 534. A portion of the output 526 of the data inspection module 510, including the identification of the error data with embedded problems and the corresponding KPI formula characteristics 470, is shown as a segment 1002 with ten instances of non-error data 1004 and three instances of error data 1006. The data segment 1002 is transmitted to a plurality of remediation methods 1010, which include remediation methods M1, M2, M3, and M4, wherein each remediation method M1-M4 is associated with a different corresponding model 532, and the number four is non-limiting. Each remediation method M1-M4 includes generating one or more estimated values, which are included in the corresponding simulation snapshot as potential solutions or replacements for the error values ​​if the particular remediation method is to be used. Since there is no predetermined concept about which remediation method among the remediation methods M1-M4 provides the best or most correct potential replacement value for the specific current error data 1006, a plurality of models 532 are used, wherein each model 532 is used to execute a corresponding remediation method M1-M4. In some embodiments, the method-based simulation sub-module 534 is communicatively coupled to the KPI formula sub-module 516 for ready access to the KPI formula 468 resident therein.

[0131] In at least some embodiments, multiple simulated data snapshots 1020 are generated. For example, in the illustrated embodiment, the remediation method M1 employs the corresponding model 532 to calculate the estimated value 1024 in the simulated data snapshot 1022. In some embodiments, the non-erroneous data portion 1004 used to calculate the estimated value 1024 of the erroneous data 1006 depends on the specific remediation technique associated with the corresponding remediation method M1. For example, if it is determined that the missing value is replaced with the average of all values, then a substantially complete set of the corresponding non-erroneous data 1004 is used in the corresponding remediation method M1. Alternatively, if only three surrounding values ​​of the non-erroneous data 1004 are used to calculate the missing value, i.e., the erroneous data 1006, then the corresponding remediation method M1 only uses those surrounding values ​​of the non-erroneous data 1004. Similarly, simulated data snapshots 1032, 1042, and 1052 are generated by the corresponding remediation methods M2-M4, including the corresponding estimated values ​​1034, 1044, and 1054. Since the remediation methods M1-M4 are different, the corresponding estimated values ​​1024, 1034, 1044, and 1054 are expected to be different. Figure 4 and Figure 5B , simulation data snapshots 1022, 1032, 1042 and 1052 are shown as output 536 from the method-based simulation sub-module 534, where the output 536 is transmitted to the data simulation snapshot storage module 538, which in some embodiments resides within the data storage system 408.

[0132] In at least one embodiment, such as the exemplary embodiment, the three instances of error data 1006 are substantially identical. In at least one embodiment, each instance of error data 1006 is different. Therefore, since multiple models 532 and remediation methods M1-M4 are used for all error data 1006, it is convenient to generate multiple corresponding estimates 1024, 1034, 1044, and 1054 for each different error. Thus, method-based simulation in the form of remediation methods M1-M4 is used to generate one or more simulated snapshots 1022, 1032, 1042 and 1052 of non-erroneous original data 1004 and estimated values ​​1024, 1034, 1044 and 1054 for each erroneous original data value 1006, wherein each estimated value 1024, 1034, 1044 and 1054 indicates what the data value will look like when a particular remediation method M1-M4 is used, and each estimated value 1024, 1034, 1044 and 1054 is the product of a different remediation method M1-M4.

[0133] In at least some embodiments, collecting the raw time series data stream 420 by the sensor suite 416 includes using a heuristic-based feature that facilitates determining patterns in the data elements as they are collected. In some cases, one or more instances of the raw data 422 may appear to be incorrect because each data element exceeds a threshold, where the threshold is based on the probability of what the data element value should be based on the established data pattern. For example, but not limited to, a surface data excursion, i.e., an upward or downward data spike, may be generated by erroneous data grouping or based on an accurate depiction of what is happening in real time. Therefore, the snapshot generator module 530 is further configured to analyze errors using point-based simulation to determine whether the erroneous data on the surface is actually erroneous data, i.e., the snapshot generator module 530 includes a point-based simulation submodule 540.

[0134] refer to Fig.11 , provides a schematic diagram showing a process 1100 for point-based simulation. Figure 4 and Figure 5B, a point-based simulation is generated by the point-based simulation submodule 540. A portion of the output 526 of the data checking module 510 including identification of erroneous data with embedded problems and corresponding KPI formula features 470 is shown as a data segment 1102 having ten instances of non-erroneous data points 1104 and three instances of suspected potentially erroneous data points 1106. The three instances of the suspected potentially erroneous data points 1106 are referred to as 1106A, 1106B, and 1106C, respectively, and are collectively referred to as 1106. In one or more embodiments, the data segment 1102 including known correct original data (i.e., non-erroneous data points 1104) and the suspected potentially erroneous data points 1106 are combined into a configured matrix 1110 to initiate a determination of the probability of whether the value of the suspected potentially erroneous data point 1106 is correct or erroneous. As shown, the matrix 1110 is based on three suspected potentially erroneous data points 1106, i.e., 2 of the three suspected potentially erroneous data points 1106. 3 The matrix 1110 is configured with three columns 1112, 1114, and 1116, i.e., one for each suspected potential error data point 1106A, 1106B, and 1106C, respectively. The resulting eight rows (referred to as D1 to D8, respectively, and collectively referred to as 1120) include the available combinations of the three suspected potential error data points 1106.

[0135] Each of the three suspected potentially erroneous data points 1106 is individually inferred as a discrete “correct” or discrete “erroneous”, and the potentially erroneous data values ​​are then referred to as “inferred data points” to distinguish them from the known correct original data (i.e., non-erroneous data points 1104). Fig.11 As shown, the inferred erroneous data points are collectively referred to as 1130. Those inferred erroneous data points 1130 associated with the suspected potentially erroneous data point 1106A are shown individually and are referred to as 1122, 1132, 1162, and 1182 in column 1112. In addition, those inferred erroneous data points 1130 associated with the suspected potentially erroneous data point 1106B are shown individually and are referred to as 1124, 1144, 1164, and 1174 in column 1114. Further, those inferred erroneous data points 1130 associated with the suspected potentially erroneous data point 1106C are shown individually and are referred to as 1126, 1146, 1176, and 1186 in column 1116.

[0136] Similarly, if Fig.11As shown, the inferred correct data points are collectively referred to as 1140. Those inferred correct data points 1140 associated with the suspected potentially erroneous data point 1106A are shown individually and are referred to as 1142, 1152, 1172, and 1192 in column 1112. In addition, those inferred correct data points 1140 associated with the suspected potentially erroneous data point 1106B are shown individually and are referred to as 1134, 1154, 1184, and 1194 in column 1114. In addition, those inferred correct data points 1140 associated with the suspected potentially erroneous data point 1106C are shown individually and are referred to as 1136, 1146, 1166, and 1196 in column 1116. The simulation snapshot 542 of the execution matrix 1120 is performed.

[0137] Thus, the first row D1 indicates all three suspected potentially erroneous data points 1106 as inferred erroneous data points 1130. Similarly, the eighth row D8 indicates all three suspected potentially erroneous data points 1106 as inferred correct data points 1140. The second, third, and fourth rows D2, D3, and D4 indicate only one of the three suspected potentially erroneous data points 1106 as inferred erroneous data points 1130, and indicate two of the three suspected potentially erroneous data points 1106 as inferred correct data points 1140, respectively. The fifth, sixth, and seventh rows D5, D6, and D7 indicate two of the three suspected potentially erroneous data points 1106 as inferred erroneous data points 1130, and indicate only one of the three suspected potentially erroneous data points 1106 as inferred correct data points 1140, respectively.

[0138] In this way, the inferred erroneous data points 1130 and the inferred correct data points 1140 have the original data values ​​transmitted and the discrete inferred labels of correct or incorrect. The remainder of the analysis focuses specifically on the inferred erroneous data points 1130 and the inferred correct data points 1140. Specifically, the entire range of all possible combinations of the inferred data points 1130 and 1140 (as shown as D1 to D8) are collected in the aforementioned simulation snapshot 542 for further evaluation. The generation of all possible combinations of discrete "correct" labels (i.e., inferred correct data points 1140) and discrete "wrong" labels (i.e., inferred erroneous data points 1130) and the subsequent clustering of these combinations facilitate further determination of the "best" actions, whether those actions are to correct the erroneous data or accept the correct data. These operations take into account the accepted inaccuracies associated with the original data that may or may not be erroneous in the data segment 1102 by determining the probability that one or more suspected potential erroneous data values ​​are "correct" or "erroneous". In each of the combinations D1 to D8, it is assumed that some of the suspected potential error values ​​1106 are incorrectly identified as errors and some are correctly identified as errors. Therefore, for each combination D1 to D8, based on a predetermined remediation method (similar to but not limited to reference Fig.10 In the method discussed), erroneous values ​​are replaced with estimated values, so each combination D1 to D8 has a different set of correct and incorrect data points and requires different estimated values ​​based on the predetermined remediation technique.

[0139] As mentioned above, point-based simulations are better configured to handle outlier problems, and the outlier problem will be used to further describe Fig.11. As described above, patterns can be identified in the raw data 504, wherein the patterns include data segments 1102 and the probability of what the individual data element values ​​should be according to the established data patterns. Therefore, the inferred data points 1130 of discrete "errors" have a probability of being misclassified as errors with the percentage assigned to them. The probability of each of the three suspected potential error values ​​1106 is used to determine whether the value 1106 is wrong. When evaluating the eight combinations of changes D1 to D8, the probability of each of D1 to D8 being true is determined, and those rows D1 to D8 with the highest probability of being true are submitted for further analysis. The total probability of D1 to D8 is 100%. For example, but not limited to, given the heuristic analysis of each point 1122, 1124 and 1126 in D1 and its associated summed probability, it can be determined that the probability of all three points in D1 being wrong is relatively low, as in row D8 (all three values ​​are correct). These two rows D1 and D8 will no longer be considered further. Note that for those embodiments where row D8 without an erroneous value has the highest probability of being correct, no further analysis needs to be performed and value 1106 is not corrected by downstream operations as further described. Thus, value combinations with a higher probability of being true are processed further.

[0140] Typically, the total number of each possible combination of discrete “correct” and “wrong” inferred data points 1130 and 1140 grows exponentially with the number of inferred data points (i.e., 2 x , where x = the number of inferred data values), and generating all possible combinations and processing them can be time and resource intensive. Each combination of inferred data points is a potential simulation, and processing each combination as a potential simulation simply adds processing overhead. Therefore, the possible combinations D1 through D8 of the inferred data points 1130 and 1140 described are further considered; however, the possible combinations of inferred data points 1130 and 1140 are "pruned" so that only a subset of all possible combinations are further considered. As described above, initial pruning occurs when low probability combinations of potentially erroneous values ​​are removed from further processing.

[0141] The point-based simulation submodule 540 is operably coupled to the snapshot optimization submodule 544. In such an embodiment, the snapshot optimization feature is employed by using the KPI formula characteristics 470 determined as previously described, regardless of whether the KPI formula characteristics 470 are unobservable bins or observable bins. For example, but not limited to, the simulation of the inferred data points 1130 and 1140 can be filtered using the KPI formula characteristics 470 for maximum, minimum, average, and median analysis. Therefore, the snapshot optimization module 544 is communicatively coupled to the KPI formula submodule 516. Generally, only those combinations of inferred data points that successfully pass the pruning process will survive to generate simulations of corresponding suspicious point values ​​through the model, and corresponding simulation snapshots are generated with non-erroneous original data and estimated values ​​for the identified erroneous data, wherein a portion of the suspected erroneous point values ​​may not actually be erroneous and will not need to be replaced.

[0142] refer to Fig.12 , provides a text diagram illustrating an example algorithm 1200 for a snapshot optimizer configured to perform a snapshot optimization submodule 544 (e.g., Figure 5B Reference Figure 4 , Figure 5A , Figure 5B and Fig.11 , algorithm 1200 includes determining 1202 as previously determined by KPI characteristic determination submodule 514 and referring to Figures 6 to 8 The operations of the KPI formula characteristic 470 described. The data represented as the matrix 1120 in the exemplary embodiment (i.e., the data embedded in the remaining rows that were not removed due to low probability as described above) is further analyzed to produce the pruning effect as described herein through the data presentation operation 1204. As described above, the exemplary embodiment includes analyzing outliers. In one or more embodiments, consider executing the first sub-algorithm, namely the "maximum value" sub-algorithm 1206. In the case where the previously determined KPI formula characteristic 470 is a maximum value characteristic, a modify data operation 1208 is performed through one or more models 532. The modify data operation 1208 includes determining whether the suspected potential error data 1106 is an outlier within a rising peak of the data segment 1102 of the original data 504. If the data segment 1102 does not exhibit an upward trend, thereby eliminating the opportunity for any rising peak, the algorithm 1200 proceeds to the next set of operations. If the data segment 1102 does exhibit an upward trend, then the affected outliers are replaced with values ​​that provide a smoothing effect on the upward trend per modify data operation 1208, with the probability values ​​previously described (which provide some level of assurance that the suspected erroneous data is actually erroneous). These data points are selected for simulation by one or more models 532. Once the data replacement identification or "repair" is performed, the maximum sub-algorithm 1206 ends 1210.

[0143] Consider executing the second sub-algorithm, the "minimum" sub-algorithm 1212. In the event that the previously determined KPI formula characteristic 470 is a minimum characteristic, a modify data operation 1214 is performed by one or more models 532. The modify data operation 1214 includes determining whether the suspected potentially erroneous data 1106 is an outlier within a decreasing peak of a data segment 1102 of the original data 504. If the data segment 1102 does not exhibit a decreasing trend, thereby eliminating the chance of any decreasing peak, the algorithm 1200 proceeds to the next set of operations. If the data segment 1102 does exhibit a decreasing trend, the affected outliers are replaced with values ​​that provide a smoothing effect on the decreasing trend of each modify data operation 1214, with the probability values ​​previously described (which provide a certain level of assurance that the suspected erroneous data is actually an error). These data points are selected by one or more models 532 for simulation. Once the data remediation or "repair" is performed, the minimum sub-algorithm 1212 is terminated 1216.

[0144] Consider executing a third sub-algorithm, the "average" sub-algorithm 1218. In the event that the previously determined KPI formula characteristic 470 is an average characteristic, a modify data operation 1220 is performed through one or more models 532. The modify data operation 1220 includes determining whether the suspected potential error data 1106 is an outlier by considering all issues (i.e., all affected suspected potential error data 1106 and corresponding probabilities as described above), and grouping them into one or more clusters of potential error data values ​​based on the proximity of the relevant corresponding values ​​to each other. In some embodiments, there may be multiple clusters of potential error data values, indicating the average characteristic used as the basis for clustering. A cluster consideration operation 1222 is performed, in which a set of representative points (such as but not limited to the average value from each cluster) is considered as the representative point for simulation. Once the data selection for simulation is performed through one or more models 532, the average sub-algorithm 1218 ends 1224.

[0145] Consider executing a fourth sub-algorithm, the "median" sub-algorithm 1226. In the event that the previously determined KPI formula characteristic 470 is a median characteristic, a modify data operation 1228 is performed by one or more models 532. The modify data operation 1228 includes determining whether the suspected potential error data 1106 is an outlier by considering all issues (i.e., all affected suspected potential error data 1106 and corresponding probabilities discussed above). If the suspected potential error data 1106 is actually an outlier, and since the median-based KPI is not affected by the value perturbation, no further action is taken on the data, and the median sub-algorithm 1226 ends 1230. In some embodiments, sub-algorithms 1206, 1212, 1218, and 1226 can be executed simultaneously in parallel. The output of the snapshot optimization module 544 (shown as the optimized simulated data snapshot 546) is transmitted to the data simulation snapshot storage module 538, which in some embodiments resides within the data storage system 408. Thus, a plurality of simulation snapshots 536 and 546 are generated for further processing, wherein the simulation snapshots 536 and 546 are generated in a manner that significantly reduces the number of estimated values.

[0146] Continue to refer Figure 4 , Figure 5B , Fig.10 and Fig.11 In at least some embodiments, the simulation snapshots 536 and 546 created by the snapshot generator module (whether method-based or point-based) are transmitted to the KPI value inference module 550. As described above, each of the simulation snapshots 536 and 546 includes non-erroneous raw data (e.g., 1004 and 1104) and estimated values ​​for established erroneous data (e.g., 1006 and 1106). Each estimated value and associated raw data are presented to the corresponding KPI formula 468 to generate a predicted exchange value, i.e., an inferred snapshot value for each estimated value in the corresponding simulation snapshots 536 and 546. In this way, the raw data 504 is also transmitted to the KPI value inference module 550.

[0147] refer to Fig.13 , provides a graphical diagram illustrating at least a portion of a KPI value inference process 1300. Also refer to Figure 4 and Figure 5B The inferred snapshot values ​​for the simulated snapshots 536 and 546 are based on the corresponding KPI formula 468 and in the context of the non-erroneous original data on the time series data stream. Therefore, for each simulated snapshot 536 and 546 transmitted to the KPI value inference module 550, a predicted replacement value, i.e., the inferred snapshot value, is generated. Fig.13A horizontal axis (Y axis) 1302 and a vertical axis (X axis) 1304 are shown. The Y axis 1302 is shown to extend from 41.8 to 42.6, where the value is unitless. The X axis 1304 is shown to have no value and no unit. The nature of the value is not important; however, the process 1300 shows a portion of the value determined as a function of the KPI formula characteristic 470 presented together with the simulation snapshots 536 and 546. The original KPI value 1306 (i.e., the value generated by the corresponding KPI formula 468 processing the suspected erroneous data) is presented as a reference, where the corresponding value is 42.177. The simulated KPI maximum snapshot 1308 presents an inferred snapshot value of 42.548, the simulated KPI average snapshot 1310 presents an inferred snapshot value of 42.091, and the simulated KPI minimum snapshot 1312 presents an inferred snapshot value of 41.805. These inferred snapshot values ​​will be used in the discussion of the subsequent parts of the process 500.

[0148] refer to Figure 5C , provides Figure 5A and Figure 5B , which shows a process 500 for calculating confidence values ​​for corrected data within time series data. Figure 5B , the outputs of the KPI value inference module 550 include an inferred point-based snapshot value 552, an inferred method-based snapshot value 554, and the original data 504, which are transmitted to a confidence metric module 570 communicatively coupled to the KPI value inference module 550. Typically, for each corresponding inferred snapshot value generated from the simulation snapshot for the erroneous data in the KPI value inference module 550, the inferred snapshot value is scored separately within the confidence metric module 570. The corresponding scoring includes generating a scored inferred point-based snapshot value 562, i.e., an inferred point-based snapshot value 552 with a corresponding confidence value. In addition, the corresponding scoring generates a scored inferred method-based snapshot value 564, i.e., an inferred method-based snapshot value 554 with a corresponding confidence value. The generation of confidence values ​​is discussed further below. The best analysis score is selected, and the corresponding inferred snapshot value is now promoted to the selected KPI value to replace the erroneous data, where the selected KPI value is referred to as the inferred KPI value 566. Thus, inferred KPI value 566 is a value selected from one or more predicted replacement values ​​(ie, scored inferred snapshot values ​​562 and 564) to address potentially erroneous data instances.

[0149] In some embodiments, the confidence metric module 570 includes a plurality of additional submodules to facilitate the generation of confidence values ​​and the details and evidence supporting such values. In some of these embodiments, three confidence metric submodules are used, i.e., a quantity-based confidence metric submodule 572, an extension-based confidence metric submodule 574, and a quantity-based and extension-based confidence metric submodule 576.

[0150] The confidence metric submodule 572 based on quantity is configured to consider the size of the value obtained from the KPI value inference module 550, and generate associated confidence metric information including corresponding confidence scores. For example, but not limited to, whether the size of the KPI value is 50 or 1050, the confidence of the obtained KPI value can be different according to additional data and circumstances. The confidence metric submodule 574 based on diffusion considers the range in which the simulation value is located, and generates associated confidence metric information including corresponding confidence scores. Instead of the absolute size of the KPI value, the statistical characteristics of the KPI value, such as the mean value, minimum value, maximum value and standard deviation, are used based on the extended confidence metric, so it is basically not affected by the size. The confidence metric submodule 576 based on quantity and extension considers the size and range of the value to generate associated confidence metric information including corresponding confidence scores. In some embodiments, all three submodules 572, 574 and 576 are used in parallel, and the results of each submodule are considered and evaluated for selection. In some embodiments, only one or two of the sub-modules 572, 574, and 576 are selected based on the nature of the incoming inferred KPI value 566 and other data 568 (discussed further below).

[0151] refer to Fig.14 , provides a graphical / textual diagram illustrating the generation 1400 of a numerical confidence metric. Also refer to Figure 5B and Figure 5C , generating confidence values ​​for the inferred point-based snapshot value 552 and the inferred method-based snapshot value 554. The linear graphical representation 1410 has Fig.13 Specifically, other data 568 (such as Figure 5C ), such as but not limited to a simulated KPI minimum snapshot value 1412 having an inferred snapshot value of 41.805, a simulated KPI average snapshot value 1414 having an inferred snapshot value of 42.091, an original KPI value 1416 of 42.117, and a simulated KPI maximum snapshot value 1418 having an inferred snapshot value of 42.548. Fig.14Also shown in the figure is a first set of confidence metric evaluation algorithms, namely, maximum deviation confidence metric algorithms 1430. Confidence metric 1A algorithms determine the relationship between the maximum variances of the inferred snapshot values ​​1412, 1414, and 1418 as a function of the original KPI value 1416. Confidence metric 1B algorithms determine the relationship between the maximum variances of the inferred snapshot values ​​1412, 1414, and 1418 as a function of the simulated KPI average snapshot value 1414. Fig.14 A second set of confidence metric evaluation algorithms, namely, mean deviation confidence metric algorithms 1440, are further shown in FIG. Confidence metric 2A algorithms determine the variance relationship between the original KPI value 1416 and the simulated KPI average snapshot value 1414 as a function of the original KPI value 1416. Confidence metric 2B algorithms determine the variance relationship between the original KPI value 1416 and the simulated KPI average snapshot value 1414 as a function of the simulated KPI average snapshot value 1414. In addition, Fig.14 An extended-based metric algorithm 1450, i.e., a confidence metric 3 algorithm, is shown that evaluates the deviation 1452 between the original KPI value 1416 and the simulated KPI average snapshot value 1414 based on the spread 1454 between the simulated KPI maximum value 1418 and the simulated KPI minimum value 1412. The maximum deviation confidence metric algorithm 1430 for confidence metrics 1A and 1B and the average deviation confidence metric algorithm 1440 for confidence metrics 2A and 2B reside within the quantity-based confidence metric submodule 572 and the quantity-based and extended confidence metric submodule 576. Similarly, the confidence metric 3 algorithm of the extended-based metric algorithm 1450 resides within the extended-based confidence metric submodule 574 and the quantity-based and extended confidence metric submodule 576.

[0152] In addition, reference Fig.15 , a chart, namely a bar chart 1500, is provided, which shows that Fig.14 14. The simulated KPI average snapshot value 1414 provides the best confidence value for the erroneous data. In at least some embodiments, for this example, the simulated KPI average snapshot value 1414 is the inferred KPI value 566.

[0153] Typically, the confidence metric module 570 compares the inference snapshot values ​​552 and 554 generated by one or more of the above-mentioned simulations with the corresponding original error data. At least one of the results of the comparison is a confidence value in numerical form for each of the inference snapshot values ​​552 and 554 of the corresponding snapshot applied to the data, which indicates that the inference snapshot values ​​552 and 554 are confidence levels for the appropriate replacement of the error data. A relatively low confidence value indicates that the corresponding inference snapshot values ​​552 and 554 including the obtained inference KPI value 566 should not be used or used with caution. A relatively high confidence value indicates that the corresponding inference snapshot values ​​552 and 554 including the obtained inference KPI value 566 should be used. The threshold value for the associated confidence value can be established by the user and can also be used to train one or more models, both of which help to make the selection fully automated. In addition, subsequent actions can be automatic. For example, but not limited to, for confidence values ​​below a predetermined threshold, the inferred KPI value 566 will not be passed for further processing within a native application (e.g., the process control system 410) utilizing the raw data stream 420. Similarly, for confidence values ​​above a predetermined threshold, the inferred KPI value 566 will be passed for further processing within a native application utilizing the raw data 422. Thus, the systems and methods described herein automatically correct issues with erroneous data in the raw data stream 420 in a manner that prevents unintentional actions or initiates appropriate actions as indicated by the situation and correct data.

[0154] Reference again Figure 5C, the confidence metric module 570 includes an explanation submodule 578, which is configured to receive confidence-based data 580 from the confidence metric submodules 572, 574, and 576. The confidence-based data 580 includes, but is not limited to, the inferred KPI value 566 and its associated confidence value, corresponding information associated with the selection of the inferred KPI value 566, and additional information for generating an explanation of the selected inferred KPI value 566, including other data 568, which includes, but is not limited to, all inferred snapshot values ​​552 and 554, including corresponding confidence values. In addition, since the confidence value of the predicted (i.e., inferred) KPI value 566 cannot be 100%, the explanation submodule 578 provides an explanation basis for resolving one or more potential erroneous data instances by providing details and evidence for selecting a particular scored inferred snapshot value 562 and 564 as the inferred KPI value 566. The explanation submodule 578 provides such details, including but not limited to the types of problems detected in the data set, the number and nature of the simulations generated, the statistical characteristics of the scores obtained from the various simulations, and the comparison of the scores. Thus, the confidence metric module 570 generates various confidence metrics and information for the scored inference snapshot values ​​562 and 564 to facilitate the user's understanding of the properties of the distribution of the scored inference snapshot values ​​562 and 564 to further provide clarity in the selection of the corresponding inference KPI value 566, thereby generating confidence scores and explanations 582 as outputs of the process 500.

[0155] In addition, reference Fig.16 , a text diagram is provided showing a confidence metric explanation 1600. The data provided in the confidence metric explanation 1600 is essentially self-explanatory.

[0156] The systems, computer program products, and methods disclosed herein facilitate overcoming the disadvantages and limitations of unintentionally processing erroneous time series data, and potentially encountering unexpected consequences. For example, when generating corresponding data, and for a given business KPI, the automated system and method described herein determine whether the data quality problem has an impact on the corresponding business KPI. In addition, the system and method described herein identify the attributes (or characteristics) of the relevant business KPI, so that the relevant data problems can be identified, and optimization can be performed, regardless of whether the accurate KPI formula is clearly visible, that is, regardless of whether the formula is essentially an observable box or an unobservable box. In addition, the system and method described herein solves the identified data problems by selecting a prediction of a scored replacement value for the erroneous data. In addition, the system and method described herein optimizes the selection of possible replacement values ​​to efficiently use system resources. In addition, the scored prediction is accompanied by a quantified confidence value, with an explanation of the confidence value of the inferred confidence metric and the reason for the value. Thus, as described herein, data quality issues are filtered based on analysis of a given KPI, and various scenarios are considered to modify the data to mitigate the quality issues, thereby calculating their impact on the measurement of the given KPI and additionally measuring the confidence of the predicted replacement value.

[0157] In addition, the features of the systems, computer program products, and methods disclosed herein can be extended beyond implementation in commercial-only embodiments. Non-commercial implementations are also envisioned to overcome similar disadvantages and limitations of inadvertently processing erroneous time series data and potentially encountering unexpected consequences. Specifically, any computer-implemented process that relies on time series data to properly perform corresponding functions can be improved by implementing the features in the present disclosure. For example, but not limited to, any use of time series data collected from IoT devices including residential and vehicle users will avoid unintentional and unnecessary automated actions by replacing missing data values ​​with the highest confidence. Specifically, for residential users, erroneous data that erroneously indicates low voltage from the corresponding power company can be prevented from inadvertently and unnecessarily activating a low voltage protection circuit, which would otherwise interrupt satisfactory power delivery to the corresponding residence. In such an implementation, a corresponding KPI can be used to maintain the continuity of power to residential users. In addition, specifically, for vehicle users, erroneous data that erroneously indicates excessive propulsion mechanism temperature can be prevented from inadvertently and unnecessarily initiating an automatic emergency engine shutdown. In such an implementation, a corresponding KPI can be used to maintain the continuity of propulsion force to vehicle users.

[0158] Thus, embodiments disclosed herein provide improvements in computer technology by providing a mechanism for efficiently, effectively, and automatically identifying problems associated with erroneous time series data, determining whether data quality issues have an impact on a given business KPI by identifying characteristics of the business KPI so that relevant data problems can be identified, and optimization can be performed regardless of whether the exact KPI characteristics are publicly defined, i.e., whether the KPI formula is essentially an observable box or an unobservable box, and resolving identified data problems while presenting a confidence analysis of the potential solutions examined.

[0159] The description of various embodiments of the present disclosure has been provided for the purpose of illustration, but it is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements existing in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method comprising: identifying one or more potentially erroneous data instances in a time series data stream; determining one or more predicted replacement values ​​for the one or more potentially erroneous data instances; determining a confidence value for each of the one or more predicted replacement values; resolving the one or more potentially erroneous data instances using one of the one or more predicted replacement values; generating an explanatory basis for a resolution of the one or more potentially erroneous data instances; as well as Generate multiple simulation snapshots through point-based simulation, including: generating a plurality of inferred data points from the one or more potentially erroneous data instances, comprising alternately assigning one of a discrete correct label and a discrete erroneous label to each of the one or more potentially erroneous data instances; generating a set of all possible combinations of the plurality of inference data points; determining a probability that the plurality of inferred data points are actually erroneous; and A plurality of point-based simulation snapshots are generated over only a subset of the set of all possible combinations of the plurality of inferred data points, wherein each point-based simulation snapshot of the plurality of point-based simulation snapshots includes one or more estimated values.

2. The computer-implemented method of claim 1 , further comprising: One or more key performance indicators (KPIs) affected by the one or more potentially erroneous data instances are identified.

3. The computer-implemented method of claim 2, further comprising: One or more KPI formula characteristics are determined to be associated with the one or more potentially erroneous data instances, wherein each of the one or more KPIs includes one or more formulas, and each of the one or more formulas includes one or more characteristics.

4. The computer-implemented method of claim 3, further comprising: Analyze observable KPI formulas to analyze observable box formulas; as well as Analyze the unobservable KPI formula to analyze the unobservable box formula.

5. The computer-implemented method of claim 3, further comprising: One or more simulation snapshots are generated, wherein each of the one or more simulation snapshots includes one or more estimated values, wherein each of the one or more predicted replacement values ​​is based at least in part on the one or more estimated values ​​and the one or more KPI formula characteristics.

6. The computer-implemented method of claim 5, further comprising: Generate multiple simulation snapshots through method-based simulation, which include: The one or more estimated values ​​are generated for each potentially erroneous data instance, wherein each of the one or more estimated values ​​is generated by a corresponding remediation operation.

7. The computer-implemented method of claim 6, further comprising: Generating the subset of the set of all possible combinations of the plurality of inferred data points includes employing a snapshot optimization feature by using the KPI formula property.

8. The computer-implemented method of claim 1 , further comprising: The one or more potentially erroneous data instances are resolved by one or more of a quantity-based confidence metric and an extension-based confidence metric and the explanation basis for the resolution of the one or more potentially erroneous data instances is generated.

9. A computer program product comprising program instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 8.

10. A computer system comprising: One or more processing devices and at least one memory device operatively coupled to the one or more processing devices, the one or more processing devices being configured to perform the method according to any one of claims 1 to 8.

Images