A system, method, and device for unsupervised data quality assessment

Abstract

The invention relates to a system, device and method for unsupervised data quality assessment of datasets. The method of labeling a set of data points as anomalous includes the steps of receiving time series data from a plurality of time series data sources, dividing the time series data into partitions, and normalizing the values of the data points in the partition. For each partition, a representative value for each successive pair of data points in the partition is calculated and a two-dimensional distribution mapping of calculated representative values for each partition is generated. Thereafter, cluster analysis is performed on a plurality of distribution mappings and identifying clusters which are anomalous and the data points in each of the partitions of each distribution mapping which is associated with the anomalous cluster is labeled as anomalous.

Description

[0001] A System, method, and device for unsupervised data quality assessment

[0002] Field of the invention

[0003] The present invention relates to a system, method and device for unsupervised data quality assessment. More particularly, but not exclusively, the present invention particularly relates to an automated and unsupervised system capable of assessing quality of datasets in abundant data types and complex data structures.

[0004] Background to the invention

[0005] Time series data collected from industrial Internet of Things (loT) devices such as sensors and machines are essential for the proper control, operation, and maintenance of almost all plants and critical infrastructure in developing and developed countries. loT can be considered a suitable platform for industrial applications, enabling large systems that connect many intelligent sensors and collect data for analytical applications. The large volume of data generated by loT sensors can lead to high demands on cloud storage and data transmission bandwidths. A suitable approach to address these issues is through various data compression techniques, which result in variable sampling rates of recorded time-series data. The resulting sampling rates depend on numerous factors, for example, the amount of allocated computer memory storage, available communication bandwidth, electromagnetic interference (EMI) and noise levels, process time constants and the configuration of such system.

[0006] Many organizations are depending on powerful data mining technologies and Artificial Intelligence (Al) to optimize the usefulness of loT device generated data. However, data quality issues with incomplete or inaccurate data often prevent the application of powerful data mining algorithms for value extraction. The problems associated with data quality have a financial impact in many industries. Quality issues may, for example, include the outliers (spikes), loss of continuity (lost data), excessive noise, senility (frozen data), data shifts, data drifts, out of range measurements, inconsistent sensor readings (for multiple sensors), and excessive data compression. A wide range of data quality assessment algorithms, systems and methods have been proposed to detect such issues before applying any further mitigating actions. However, the most automated among these methods, being methods that require minimum supervision from subject matter experts, can only capture a very limited range of data quality symptoms and fail to detect even basic symptoms like senility (frozen data), or excessive data compression. Further, the state-of-the-art methods proposed so far rely considerably on contributions from technical experts and can be quite time consuming and expensive to implement. Publications from the state of the art are discussed in further detail below.

[0007] European patent number 3631637 entitled “System, device and method for frozen period detection in sensor datasets" in the name of Siemens Energy Global GmbH & Co. KG describes the automation of determining error condition (freezing) in sensors in a technical system. The invention discloses the detection of anomalies, forecasting recognition in sensors which is performed by converting a stream of sensor data into a stream of symbols. This conversion is performed by using run length encoding (RLE). Anomalies are detected based on a predetermined number of consecutive repetitions set before hand from knowledge of sensors. Any repetition greater than the threshold is classified as a “frozen period”. The detection of the frozen period is unsupervised, no threshold is set. The input data is a dataset of time series. RLE is used to detect repetition while K-means clustering is used on the repetitions to identify the frozen periods by inspecting each cluster mean. The cluster with the greatest mean would hold the repetitions that are detected as frozen periods. A problem with this approach is that it detects only frozen periods of sensors and no other anomalies.

[0008] United States patent number 11 ,113,627 in the name of Noble Artificial Intelligence Inc entitled “characterisation of data sets corresponding to dynamical statistical systems using machine learning" discloses a machine learning and encoding approach on an input of data for characterizing dynamical statistical systems that vary with time. A distribution function is encoded as a sum of contour functions in time series. Machine learning is applied on the encoded distribution function which is visualized. The approach claims it provides a non-linear improvement in machine learning, however, it lacks capability to perform for assessing and managing overly compressed data (excessive compression). Among the problems with the prior art described above is that many of the methods require human intervention and labeling in order to function correctly. Further, the unsupervised methods described above are aimed at eliminating only some forms of data inaccuracies.

[0009] Object of the invention

[0010] It is accordingly an object of the present invention to provide a system, device and method for unsupervised data quality assessment which alleviates some of the problems associated with the prior art.

[0011] Summary of the invention

[0012] Accordingly, one aspect of the present invention discloses a system for unsupervised data quality assessment, wherein the system is operably comprises: one or more receiver unit to receive one or more datasets in time series; a data partition unit for dividing the time series data into partitions containing a suitable number of data points, wherein each data point includes at least a value and a time; tangent analysis unit for considering varying sampling rate values for each partition wherein a first dimension of the mapping corresponds to the difference in normalized value and second dimension corresponds to the difference in time; distribution mapping unit for mapping between values derived from tangent analysis unit and time difference between two consecutive samples; image mapping and normalizing unit to generate range groups in both axes of the distribution mapping and to standardize resulting images; clustering unit to identify anomalies on a plurality of distribution mappings and identifying clusters which are anomalous; data quality assessment unit for accurate and efficient assessment of results based on the clustering of images.

[0013] In another aspect of the present invention, the dataset includes plurality of values that are repetitive and / or non-repetitive.

[0014] In another aspect of the present invention, the dataset includes a plurality of values from variety of sensors and / or other data sources, like computer calculated values, manual inputs, etc.

[0015] In another aspect of the present invention, the data partition unit for dividing the time series data, wherein each data point includes at least a value and a time for normalizing the values of the data points in the partition.

[0016] In another aspect of the present invention, the data partition unit for dividing the time series data, wherein each data point further includes at least a value and a time for calculating a representative value for each successive pair of data points in the partition wherein the representative value is based on the difference in normalized value and the difference in time between the successive pair of data points.

[0017] In another aspect of the present invention, the tangent analysis unit further provides a tangential value which is equal to the difference in normalized value divided by the difference in time between the successive pair of data points.

[0018] In another aspect of the present invention, the distribution mapping unit provides mapping value between all the tangential values paired with the associated time difference between two consecutive samples.

[0019] In another aspect of the present invention, the clustering unit are associated occurrence rates, wherein, clusters with larger populations occurrence corresponds to healthy data and fewer population occurrences corresponds to the anomalies.

[0020] In a method aspect of the present invention, the method is provided for the unsupervised data quality assessment comprising the steps of: receiving time series data from a plurality of time series data sources;

[0021] - dividing the time series data into partitions containing a suitable number of data points, wherein each data point includes at least a value and a time; o normalizing the values of the data points in the partition; o calculating a representative value for each successive pair of data points in the partition wherein the representative value is based on the difference in normalized value and the difference in time between the successive pair of data points; o generating a two-dimensional distribution mapping of calculated representative values for each partition wherein a first dimension of the mapping corresponds to the difference in normalized value and second dimension corresponds to the difference in time from which the representative value was calculated;

[0022] - performing cluster analysis on a plurality of distribution mappings and identifying clusters which are anomalous; and

[0023] - labeling the data points in each of the partitions of each distribution mapping which is associated with the anomalous cluster as anomalous.

[0024] The method may include the step of labeling the data points in each of the partitions of each distribution mapping which is not associated with the anomalous cluster as healthy.

[0025] In yet another aspect of the present invention, each data source may be a sensor which sense and sends data points.

[0026] In yet another aspect of the present invention, the step of generating a distribution mapping may be succeeded by the step of normalizing values of the distribution mapping. In yet another aspect of the present invention, the step of dividing the time series data into partitions is preceded by the step of determining a suitable number of datapoints and selecting a time period of a partition by varying the time period from a shorter to a longer time period, determining the number of data points in each varied time period and selecting a partition size of time period suitable to include a minimum number of data points based on archival assessments.

[0027] In another aspect of the present invention, the step of calculating a representative value for sequential datapoints is calculated as a tangential value which is equal to the difference in normalized value divided by the difference in time between the successive pair of data points.

[0028] In another aspect of the present invention, the two-dimensional distribution mapping may be a matrix of range groups wherein each range groups corresponds to a range of time and value differences. The dimensions of the matrix may be 13x13 to form 169 bins in the matrix, each bin corresponding to a range of time and value differences. The matrix may be visualized as a heat map.

[0029] In another aspect of the present invention, the clustering analysis may be performed using k-means clustering or t-distributed stochastic neighbor embedding (t-SNE). Clusters which have low occurrence rates or are isolated during the cluster analysis are identified as anomalous and clusters which have relatively larger occurrence rates are identified as healthy.

[0030] In a device aspect of the present invention, the device for unsupervised data quality assessment of datasets comprises a processing unit, memory, removable storage, and non-removable storage, and communications means which is configured to receive time series data from a plurality of time series data sources and to execute a method of labeling a set of data points as anomalous as described herein. The method is executed on the time series wherein the data may be received from the plurality of time series data sources and data quality of each set of time series data is assessed by the labelling of the data points as anomalous. In another aspect of the present invention the device for unsupervised data quality assessment of datasets includes an enclosure, a power supply, and a system for unsupervised data quality assessment of datasets as described herein; wherein the system is housed in the enclosure and powered by the power supply and the communication means is connected to at least one network such that the system may receive time series data from a plurality of time series sources such that the data quality of data points from the sources may be assessed and provided to external devices.

[0031] These and other aspects and features of non-limiting embodiments of the present invention will become apparent to those skilled in the art upon review of the following description of specific non-limiting embodiments of the invention in conjunction with the accompanying drawings.

[0032] Brief description of the drawings

[0033] An embodiment of the invention is described below, by way of a non-limiting example only, and with reference to the accompanying drawing in which:

[0034] Figure 1 represents a schematic diagram representing a method for the unsupervised data quality assessment according to one of the embodiments of the present invention.

[0035] Figure 2 represents graphs of value vs time of data points from a data source and a plot of normalized values and time difference for two sensors according to one of the embodiments of the present invention.

[0036] Figure 3 represents graphs of value vs time of data points from a data source and a plot of normalized values and time difference for two more sensors according to one of the embodiments of the present invention.

[0037] Figure 4 represents distribution mappings for the data of the four sensors of figures 2 and 3 according to one of the embodiments of the present invention. Figure 5 represents clustering analysis before and after log-based normalization of a plurality of sensors according to one of the embodiments of the present invention.

[0038] Figure 6 represents a schematic diagram of a system for unsupervised data quality assessment of datasets according to one of the embodiments of the present invention.

[0039] Figure 7 represents a schematic diagram of a field instrument including a sensor which is connected to a system for unsupervised data quality assessment of datasets according to one of the embodiments of the present invention.

[0040] Detailed description of the invention

[0041] The present invention may be understood more readily by reference to the following detailed description of the invention taken in connection with the accompanying drawing figures, which forms a part of this disclosure. It is to be understood that this invention is not limited to the specific device, method, conditions, or parameters described and / or shown herein and that the terminology used herein is for the example only and is not intended to be limiting the scope of the claimed invention.

[0042] Also, as used in the specification including the appended claims, the singular forms ‘a’, ‘an’, and ‘the’ include the plural, and references to a particular numerical value includes at least that particular value unless the content clearly directs otherwise.

[0043] Ranges may be expressed herein as from ‘about’ or ‘approximately’ another particular value when such a range is ex- pressed another embodiment. Also, it will be understood that unless otherwise indicated, dimensions and material characteristics stated herein are by way of example rather than limitation and are for better understanding of sample embodiment of suitable utility, and variations outside of the stated values may also be within the scope of the invention depending upon the application.

[0044] Embodiments will now be described in detail with reference to the accompanying drawings. To avoid unnecessarily obscuring the present disclosure, well-known features may not be described or substantially the same elements may not be redundantly described, for example. This is for ease of understanding.

[0045] The following description is provided to enable those skilled in the art to fully understand the present disclosure and are in no way intended to limit the scope of the present disclosure as set forth.

[0046] Embodiments described herein provide a system, method, and device for unsupervised and accurate data quality assessment.

[0047] With reference to the drawings, reference numerals are preceded by the figure number, for example, reference numerals 101 , 102, etc., will form part of figure 1 and reference numerals 201 , 202, etc., will form part of figure 2. As a result of this numbering, reference numerals related to the method will be preceded by a 1 in figure 1 , references related to data will be preceded by 2, 3, 4, and 5, and the system will be preceded by 6 and 7.

[0048] According to one embodiment of the present invention, the method 100 includes the step of receiving 101 time series data from a plurality of time series data sources. The data sources will typically be sensors associated with industrial equipment or infrastructure which measures values relating to the operation of the equipment or infrastructure and translates this to electronic (digital) values which may be interpreted by a computer or other electronic system. The values or in the form of data points which typically includes a value and a time stamp of the value. The values depend on what the sensor is measuring, for example, a pressure sensor will provide a timestamp and pressure value (expressed as Pascal or pounds per square inch or a voltage value having a relation to the pressure).

[0049] According to another embodiment, the method includes the step 102 of dividing the time series data into partitions (or time windows) containing a suitable number of data points. A partition is a definition of the time window for aggregating time series data. Typically, a partition is meaningful when it contains a few hundred samples (recorded sensor values). The larger the value of samples, the robustness of the application of the method is higher. In order for a suitable partition size to be selected, this step may be preceded by the step 103 of determining a suitable number of datapoints and selecting a time period of a partition by varying the time period from a shorter to a longer time period, determining the number of data points in each varied time period and selecting a partition size of time period suitable to include a minimum number of data points based on historical assessments. Typically, once a sample size of a few hundred data points has been reached by iteratively performing step 103 there should be sufficient data points for a partition. However, based on historical data and accuracy of the method, the sample may be predefined to be smaller or larger for more accurate results.

[0050] According to another embodiment, the method includes the step 104 of normalizing the values of the data points in the partition and calculating a representative value for each successive pair of data points in the partition. The representative value is based on the difference in normalized value and the difference in time between the successive pair of data points. The calculation of this representative value is referred to herein as tangential analysis as the step 105 of calculating a representative value for sequential datapoints is calculated as a tangential value which is equal to the difference in normalized value divided by the difference in time between the successive pair of data points according to the following formula: wherein k corresponds to the time step in question, AYk corresponds to the normalized value at time step k and AXk corresponds to the length of the time step or the time difference between consecutive samples. Tangent analysis is used for time-series data to address the challenge of varying sampling rates (the number of data points acquired per second). Data signals are continuously being read in sensors and converted to a digital output to be displayed. Tangent analysis is a technique that allows the determination of a standard time step for systems relying on varying sampling rates, which is prevalent in the industry.

[0051] According to another embodiment, the method includes the step 106 of generating a two-dimensional distribution mapping of calculated representative values for each partition wherein a first dimension of the mapping corresponds to the difference in normalized value and second dimension corresponds to the difference in time from which the representative value was calculated. The values of the distribution mapping may be normalized. The two-dimensional distribution mapping may be a matrix, typically with dimensions of 13x13 to form 169 bins therein. However, the dimensions and number of bins may be predefined 108 by a user. If the dimensions and number of bins are not predefined, then default dimensions of 13x13 bins will be applied 109. Each bin corresponds to a range of time and value differences and may be visualized as a heat map. Examples of such heat maps are shown in figure 4 for the data of sensors A, B, C, and D shown in figures 2 and 3. A heat map is a graphical representation of data where individual values are represented as colors or intensities. It is a way to visualize data in a two-dimensional space, where the x and y axis represent the coordinates, and the color of each point represents the value of the data at that coordinate. The colors are typically chosen to represent a range of values, with a color scale that ranges from low (typically blue or green) to high (typically red or yellow). A heat map can be created from a matrix or 2D array, where each element in the matrix represents a value, and the position of that element in the matrix represents the x and y coordinates. The color of each point in the heat map is determined by the value of the corresponding element in the matrix. The most common way to display a heat map is by using a color scale, where different colors represent different ranges of values. The color scale typically ranges from blue or green for low values to red or yellow for high values, but other color schemes can be used as well. Heat map is a graphical representation of data in a 2D space where colors are used to represent the values of the data at each coordinate. It is a powerful tool for visualizing and understanding complex data. By using this information in the form of images, it also allows cluster analysis 107 to be performed using graphical software and hardware tools which are readily available, cost-effective, and may speed of the execution of the analysis compared to traditional numerical approaches.

[0052] According to another embodiment, the method 100 includes the step 107 of performing cluster analysis on a plurality of distribution mappings and identifying clusters which are anomalous. This step 107 may be preceded by step 110 of aggregating the plurality of distribution mappings and each of the mappings may be compressed prior to performing cluster analysis 107. Clustering 107 is performed on sets of distribution which are effectively 3-dimensional data structures. The clustering analysis may be performed using k-means clustering or t-distributed stochastic neighbor embedding (t- SNE) with k-means being preferred. K-means is a clustering algorithm that is used to classify a set of n-dimensional data points into k clusters, where k is a user-specified parameter. The algorithm works by first initializing k centroids, which are the center points of each cluster. Then, the data points are iteratively assigned to the cluster whose centroid is closest to them, and the centroids are recomputed as the mean of all the points assigned to that cluster. The process is repeated until the assignment of points to clusters no longer changes or a maximum number of iterations is reached. In terms of visualizing k-means, it can be represented as a scatter plot where each data point is a dot (as can be seen in figure 5), and the dots are grouped into clusters. Each cluster is represented by a different color, and the centroid of each cluster is represented by a large dot in the center of the cluster. K-means can also be applied to images (such as the heat maps generated for the distribution mappings herein), in which case the data points are the pixels of the image, and each pixel is represented by a 3-dimensional vector (R,G,B values). The algorithm works in a similar way as for n-dimensional data points, but instead of clustering the pixels based on their numerical values, the algorithm clusters the pixels based on their colors. In the example described herein, colors are a standardized representation of the data in the form of a heat map. After the algorithm has finished running, each cluster represents a different color in the image, and the pixels in the image are replaced with the centroid of their cluster, resulting in a quantized image with a reduced color palette. It will assist in grouping the sensors which perform similarly into different clusters, to detect or emphasize anomalies (sensors values which do not conform to the norm). In general, clusters which have low occurrence rates or are isolated during the cluster analysis 107 are identified as anomalous and clusters which have relatively larger occurrence rates are identified as healthy.

[0053] According to another embodiment, the final step of the method is labeling the data points in each of the partitions of each distribution mapping which is associated with the anomalous cluster as anomalous. Similarly, the data points in each of the partitions of each distribution mapping which is not associated with the anomalous cluster is considered as healthy. By labeling data as anomalous or healthy, any subsequent data mining and analysis may incorporate such information there. For example, it may be preferable to merely exclude all anomalous data points from analysis.

[0054] According to another embodiment, anomalous data may be broadly classified as falling into one of the following categories: outliers 201 and 301 which are data points that lie an abnormal distance from other values in a random sample from a population; noise 202 and 302 - which are random errors on an underlying time series data signal; loss of continuity 203 and 303 - where there are missing values in time series data; or senility 204 and 304- which typically takes the form of a sequence of constant values in a given time series. Which can occur due to freezing of sensors. These anomalies can be seen in the sensor values in figures 2, 3 and for sensors A, B, C, and D.

[0055] According to another embodiment, the present invention includes a device 600 for unsupervised data quality assessment of datasets comprising: a processing unit 601 , memory (602, 603, 604, 611 ) removable storage 606, and non-removable storage 607, and communications means 610 which is configured to receive time series data from a plurality of time series data sources and to execute a method of labeling a set of data points as anomalous as described herein wherein the method is executed on the time series data received from the plurality of time series data sources and data quality of each set of time series data is assessed by whether or not the data points is labeled as anomalous or not.

[0056] According to another embodiment, a block diagram of an example computer device that performs data analysis from various sensors and is configured to implement the presently described method (in either the online or offline scenarios) and execute other programming is shown in figure 6. A general computing device in the form of a computer 605 may include a processing unit 601 , memory 602, removable storage 606, and non-removable storage 607. Memory 602 may include volatile memory 603 and non-volatile memory 604. Computer 605 may include, or have access to a computing environment that includes, a variety of computer-readable media, such as volatile memory 603 and non-volatile memory 604, removable storage 606 and nonremovable storage 607. Computer storage includes random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), Digital Versatile Disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions. Computer 605 may include or have access to a computing environment that includes input 608, output 609, and a communication connection 610. The computer may operate in a networked environment using a communication connection to connect to one or more remote computers. The remote computer may include a personal computer (PC, server, router, network PC, a peer device or other common network node, or the like. The communication connection may include a Local Area Network (LAN), a Wide Area Network (WAN) or other networks. Computer-readable instructions to execute methods and algorithms described above may be stored on a computer readable medium, such as illustrated at a program storage device 611 , are executable by the processing unit 601 of the computer 605. A hard drive, CD-ROM, and RAM are some examples of articles including a computer-readable medium. In one embodiment, a user interface is provided, such as a touchscreen device for providing both input 608 and output 609.

[0057] Another embodiment of the invention includes a device for unsupervised data quality assessment of datasets wherein the device may include an enclosure, a power supply, and a system for unsupervised data quality assessment of datasets as described herein; wherein the system 600 is housed in the enclosure and powered by the power supply and the communication means 610 is connected to at least one network such the system may receive time series data from a plurality of time series sources such that the data quality of data points from the sources may be assessed and provided to external devices. Such a device may be employed on a site where sensor data is collected in order to sanitize (label data as anomalous or healthy) data points before sending to a third party. Alternatively, such a device may be deployed where the data is analyzed in order to sanitize received data before analysis.

[0058] According to another embodiment, figure 7 illustrates an example monitoring system 700 deployed in a water distribution system using a plurality of field instruments represented by 701 and 702 connected to a data collection or processing unit 703 via network 704. The processing unit 703 may be configured to implement analysis of data collected from the field instruments with an online algorithm 707 (which executes according to the method described herein) or may collect the data for later processing with an offline algorithm 708 by the storage of archived data 710 in database 709. The processing unit 703 may be further connected to a management user interface, such as may be provided in computer system 706 via network connection 705. The management user interface may enable real-time or delayed user monitoring and control of the variety of components presented in tank farm monitoring system 700. The field instruments 701 , 702 deployed in the water distribution system 700 include a variety of sensors to measure process variables. A field instrument may comprise a gauge, sensor, or probe, although many other types of instruments and monitoring units may be used which provide time series data. In one embodiment, a field instrument simply measures some physical variable (hence it is a sensor); but various types of field instruments may be configured with increasing complexity to also perform computations, generate error codes, and receive commands to perform certain actions or analysis. Therefore, the field instrument may include a wide level of integrated processing or may provide raw data for analysis in a larger monitoring system.

[0059] It is envisaged that the invention will provide a method and associated systems and a device which enables various data anomalies such as outliers, noise, loss of continuity, and / or senility to be identified and labeled as such before any subsequent data analysis is performed thereon. It will provide cleaner data to be used for data mining technologies and Artificial Intelligence (Al) to optimize the usefulness of loT device generated data.

[0060] The invention is not limited to the precise details as described herein. For example, instead of having an online-system to provide alerts for time series data acquisition in near real time, an offline system that audits recorded historical data, or even an ad- hoc system / module that is used to pre-process time series data may be employed. Further, instead of using k-means or t-SNE for clustering analysis, any other clustering algorithm capable of clustering 3-dimensional or image data may be used.

Claims

CLAIMS1. A system for unsupervised data quality assessment, wherein the system is operably configured to comprising: one or more receiver unit to receive one or more datasets in time series; a data partition unit for dividing the time series data into partitions containing a suitable number of data points, wherein each data point includes at least a value and a time; tangent analysis unit for considering varying sampling rate values for each partition wherein a first dimension of the mapping corresponds to the difference in normalized value and second dimension corresponds to the difference in time; distribution mapping unit for mapping between values derived from tangent analysis unit and time difference between two consecutive samples; image mapping and normalizing unit to generate range groups in both axes of the distribution mapping and to standardize resulting images; clustering unit to identify anomalies on a plurality of distribution mappings and identifying clusters which are anomalous; data quality assessment unit for accurate and efficient assessment of results based on the clustering of images.

2. The system as claimed in claim 1 , wherein the dataset includes plurality of values that are repetitive.

3. The system as claimed in claim 1 , wherein the dataset includes plurality of values that are non-repetitive.

4. The system as claimed in claim 1 , wherein the dataset includes a plurality of values from variety of data sources.

5. The system of claim 4 wherein the data sources include sensors.

6. The system of claim 4 wherein the data sources include computer calculated values.

7. The system of claim 4 wherein the data sources include manual inputs.

8. The system as claimed in claim 1 wherein each data point includes at least a value and a time for normalizing the values of the data points in the partition by the data partition unit for dividing the time series data.

9. The system as claimed in claim 1 , wherein the data partition unit for dividing the time series data uses each data point which includes at least a value and a time for calculating a representative value for each successive pair of data points in the partition and wherein the representative value is based on the difference in normalized value and the difference in time between the successive pair of data points.

10. The system as claimed in claim 1 , wherein the tangent analysis unit further provides a tangential value which is equal to the difference in normalized value divided by the difference in time between the successive pair of data points.11 . The system as claimed in claim 1 , wherein the distribution mapping unit provides mapping value between all the tangential values paired with the associated time difference between two consecutive samples.

12. The system as claimed in claim 1 , wherein the clustering unit uses associated occurrence rates such that clusters with larger populations occurrence corresponds to healthy data and fewer population occurrences corresponds to the anomalies.

13. A method for the unsupervised data quality assessment comprising the steps of:- receiving time series data from a plurality of time series data sources;- dividing the time series data into partitions containing a suitable number of data points, wherein each data point includes at least a value and a time; o normalizing the values of the data points in the partition; o calculating a representative value for each successive pair of data points in the partition wherein the representative value is based on the difference in normalized value and the difference in time between the successive pair of data points;- generating a two-dimensional distribution mapping of calculated representative values for each partition wherein a first dimension of the mapping corresponds to the difference in normalized value and second dimension corresponds to the difference in time from which the representative value was calculated;- performing cluster analysis on a plurality of distribution mappings and identifying clusters which are anomalous; and- labeling the data points in each of the partitions of each distribution mapping which is associated with the anomalous cluster as anomalous.

14. The method of claim 13 wherein each data source is a sensor which senses and sends data points.

15. The method of claim 13 wherein the step of dividing the time series data into partitions is preceded by the step of determining a suitable number of datapoints and selecting a time period of a partition by varying the time period from a shorter to a longer time period, determining the number of data points in each varied time period and selecting a partition size of time period suitable to include a minimum number of data points based on previous assessments.

16. The method of claim 13 wherein the step of calculating a representative value for sequential datapoints is calculated as a tangential value which is equal to the difference in normalized value divided by the difference in time between the successive pair of data points.

17. The method of claim 13 wherein the two-dimensional distribution mapping is a matrix range groups wherein each range groups corresponds to a range of time and value differences.

18. The method of claim 13 wherein the analysis is performed using a method including k-means, Gaussian Mixture Models (GMMs), or Density-Based Spatial Clustering of Applications with Noise (DBSCAN).

19. The method of claim 13 wherein the clusters which have low occurrence rates or are isolated during the cluster analysis are identified as anomalous and clusters which have relatively larger occurrence rates are identified as healthy.

20. A device for unsupervised data quality assessment of datasets comprising: a processing unit, memory, removable storage, and non-removable storage, and communications means which is configured to receive time series data from a plurality of time series data sources and to execute the method of labeling a set of data points as anomalous of claim 9 wherein the method is executed on the time series data received from the plurality of time series data sources and data quality of each set of time series data is assessed by whether or not the data points is labeled as anomalous or not.

21. The device as claimed in claim 17 the wherein, the device is housed in the enclosure and powered by the power supply and the communication medium and is connected to at least one network such the system may receive time series data from a plurality of time series sources such that the data quality of data points from the sources may be assessed and provided to external devices.

Images