Distributed similarity query method and system for multivariate time series data

Abstract

The invention discloses a multivariate time sequence data distributed similarity query method and system.The method comprises the steps that firstly, prefix feature vectors are extracted from all dimensions of a multivariate time sequence through prefix features; then, through data index construction, spatial indexes are constructed for the feature vectors of all the dimensions respectively; then, through query processing, a filtering candidate set is queried on each sub-index based on the lower bound distance; and finally, through candidate set intersection taking and subsequent processing, accurate similarity query is completed. The dimension-based distributed index provided by the invention can effectively support unequal-length queries, has relatively high expandability and robustness, and can play an important role in the fields of big data, data mining, artificial intelligence, databases and the like.

Description

Technical Field

[0001] The present invention belongs to the fields of big data, database, data mining, etc., and specifically relates to a distributed similarity query method and system for multivariate time series data. Background Art

[0002] Time series are widely present in people's daily lives and industrial production, such as real-time transaction data of funds or stocks, daily sales data of the retail market, sensor monitoring data of process industries, astronomical observation data, aerospace radar, satellite monitoring data, real-time weather temperature and air quality index, etc.

[0003] Time series similarity query, also known as time series sample retrieval, has a wide range of application needs in life and production. For example, in the industrial Internet of Things, for abnormal alarms occurring on the assembly line, maintenance personnel want to quickly query the historical faults that are most similar to the current sensor sampling data from the massive sensor monitoring data as a reference, and solve the current problem based on the historical solution. Obviously, the most direct way to solve this problem is to linearly scan the database, but this method of comparing the query sequence with the database samples one by one has a high time overhead. Therefore, spatial indexing methods such as R-tree, R*-tree, grid file, etc. are usually used to index the database. However, the performance of existing spatial indexing methods will rapidly decrease as the dimension of the time series increases, that is, the "dimensionality disaster" problem occurs, so before constructing a spatial index for the time series, it is necessary to first use a data representation method to extract data features for dimensionality reduction.

[0004] At present, the commonly used data representation methods can be divided into non-data adaptive methods and data adaptive methods. For the former, the transformation parameters are not affected by the individual time series and remain unchanged; this type of representation is mostly based on spectral decomposition, such as discrete Fourier transform, discrete wavelet transform, and discrete cosine transform, which mainly extracts the main spectral coefficients as features by performing corresponding frequency domain transformations on the original time series; each of these methods has its own defects, such as discrete Fourier transform can only extract overall morphological features while ignoring local features, discrete wavelet transform can only process time series with an exponential length of 2, and the lower bound of discrete cosine transform features is too weak, resulting in very low query efficiency. Data adaptive representation means that the determination of transformation parameters needs to rely on the data itself; by adding data-sensitive selection processing, most non-data adaptive methods can be transformed into data adaptive methods. This type of method includes segmented clustering approximation, segmented linear approximation, symbolic clustering approximation, singular value decomposition, principal component analysis, etc. The first three methods need to segment the original time series first, and then process each sub-segment separately: segmented clustering approximation is to find the average value of each segment; segmented linear approximation is to fit each segment; symbolic clustering approximation is to discretize the average value of each segment into symbols based on segmented clustering approximation; according to the different approximation degrees of the original time series, their query performance will be different. Singular value decomposition and principal component analysis are achieved by performing a unified feature matrix decomposition on all time series; the typical defects of these two methods are that they have high computational complexity, and the decomposition process can only be completed in memory, and the scalability of data scale is very low. Summary of the invention

[0005] In order to solve the shortcomings of the prior art and reduce the space overhead of data storage to improve query efficiency, the present invention adopts the following technical solutions:

[0006] A distributed similarity query method for multivariate time series data, applied to industrial monitoring data, comprises the following steps:

[0007] (1) Feature extraction: obtain monitoring data of different dimensions at the same time from industrial monitoring data, obtain multivariate time series data of industrial monitoring data within a certain time range based on time flow, and extract features from the time series data of different dimensions to obtain sub-feature series data of each dimension;

[0008] (2) Data index construction: constructing an index based on the sub-feature sequence data of each dimension;

[0009] (3) Query processing: obtaining query data of different dimensions at the same time from the industrial query data, obtaining multivariate time series data within its time range based on the time flow of the industrial query data, extracting features from the query data of different dimensions respectively, obtaining query sub-feature sequence data of each dimension, calculating similarity based on the query sub-feature sequence data and the sub-feature sequence data on the index, and taking the industrial monitoring multivariate time series data corresponding to the index node with high similarity as the intermediate candidate set;

[0010] (4) Subsequent processing: Take the intersection of the intermediate candidate sets queried on the sub-feature data of each dimension, and compare the similarity of the final candidate set corresponding to the original industrial monitoring data and the industrial query data to obtain the query result.

[0011] Furthermore, in step (1), the prefix features of the sub-feature sequence data of each dimension are extracted; in step (3), the prefix features of the query sub-feature sequence data of each dimension are extracted, and on each dimensional index, the Euclidean distance between the prefix features of the query sub-feature sequence data of the corresponding dimension and the prefix features corresponding to each index node is calculated in turn to determine the similarity; the Euclidean distance based on the prefix feature is the lower bound distance of the multivariate dynamic time warping distance, which can ensure the completeness of the query result, that is, the result of the query based on the multivariate dynamic time warping distance on the original data must be included in the intermediate candidate set based on the Euclidean distance query of the prefix feature.

[0012] Furthermore, in step (1), data normalization is performed on each dimension of the multivariate time series data, and the starting value, final value, maximum value, and minimum value of the prefix sequence of each dimension are extracted as prefix features;

[0013] Furthermore, in step (2), a metric index or a spatial index is constructed on the prefix feature set of each dimension of the multivariate time series data.

[0014] Furthermore, in step (3), the nodes and their subtrees whose distance values ​​are greater than the query threshold are pruned, and the multivariate time series contained in all leaf nodes whose distance values ​​are less than or equal to the query threshold are returned as the intermediate candidate set.

[0015] Furthermore, in step (1), each dimension of the multivariate time series is z-normalized according to the following formula:

[0016]

[0017] Among them, t i Represents a time series data under the current dimension before normalization, t′ iIt represents a time series data in a single dimension after normalization, m represents the average value of all time series data in this dimension, and σ represents the standard deviation of all time series data in this dimension.

[0018] Furthermore, in step (4), since the final candidate set obtained is an address value, it is necessary to read the original data of the final candidate set in sequence through disk I / O, calculate the multivariate dynamic time warping distance between the multivariate time series data of each original industrial monitoring data and the multivariate time series data of the industrial query data, and return the data that is less than or equal to the query threshold as the result.

[0019] Furthermore, in step (4), the calculation of the multivariate dynamic time warping distance includes the following sub-steps:

[0020] (4.1) Construct a point-to-point distance matrix of two multivariate time series. Each cell value in the matrix is ​​the Euclidean distance between the sampling vectors at the corresponding position of the two series.

[0021] (4.2) Use the dynamic programming algorithm to find the optimal curved path on the point-to-point distance matrix so that the cumulative sum of the element values ​​on the path is minimized. The cumulative sum of the path is used as the final distance metric to determine the similarity.

[0022] A distributed similarity query system for multivariate time series data is applied to a distributed database. The system comprises a feature extraction module, a data index construction module, a query processing module and a subsequent processing module, which are respectively used for the feature extraction, the data index construction, the query processing and the subsequent processing. The sub-feature sequence data of each dimension is deployed in the distributed database.

[0023] The advantages and beneficial effects of the present invention are:

[0024] 1. The traditional multivariate time series data similarity query method uniformly constructs indexes in all dimensions. The large number of dimensions makes the lower bound distance not compact enough, resulting in a large size of the intermediate candidate set. The distributed index of the present invention significantly reduces the size of the candidate set through intersection operations, greatly improving the query efficiency.

[0025] 2. The traditional similarity query method of multivariate time series data only supports the Euclidean distance measurement on the original data, while the prefix vector Euclidean distance used in the present invention is the lower bound of the dynamic time warping distance, which can support complete query based on the dynamic time warping distance, not only has a higher measurement accuracy, but also can support sequence queries of unequal lengths;

[0026] 3. The dimension-based distributed index constructed by the present invention makes the sub-indexes independent of each other, so it has high scalability and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of a method in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0029] like Figure 1 As shown in FIG. 1 , a distributed similarity query method for multivariate time series data is implemented for industrial process sensor monitoring data (including four dimensions: pressure, temperature, weight, and displacement). Specifically, the following steps are included:

[0030] (1) Prefix feature extraction; the time series data sampled by the four-dimensional sensor is as follows:

[0031]

[0032]

[0033] Among them, n represents unit time, and 4 represents four dimensions;

[0034] All sensor sampling sequences are z-normalized in four dimensions:

[0035]

[0036] Among them, t i Represents a time series data under the current dimension before normalization, t′ i It represents a time series data in a single dimension after normalization, m represents the average value of all time series data in this dimension, and σ represents the standard deviation of all time series data in this dimension.

[0037] The starting value, final value, maximum value, and minimum value of the prefix sequence with a length of 100 in each dimension are extracted as features, as follows:

[0038]

[0039] Among them, the subscript max indicates that the current feature t' is the maximum value of the prefix sequence, the subscript min indicates that the current feature t' is the minimum value of the prefix sequence, and l indicates that the current feature t' is the final value of the prefix sequence.

[0040] (2) Data index construction: M-tree indexes are constructed on the four-dimensional prefix feature vectors of the data set. The specific four-dimensional prefix features are as follows:

[0041]

[0042] Among them, b represents the feature vector of industrial query data, the superscript represents the four dimensions of industrial query data: pressure, temperature, weight, and displacement, the subscript m represents the maximum number of steps that the sliding window slides in the industrial query data, the subscript max represents that the current feature b is the maximum value of the prefix sequence, the subscript min represents that the current feature b is the minimum value of the prefix sequence, and l represents that the current feature b is the final value of the prefix sequence.

[0043] (3) Query processing: Assuming a query sequence of length 200, first extract the prefix feature vectors of the four dimensions, then calculate the Euclidean distance from the query feature vector of the corresponding dimension to each index node on the four-dimensional index, prune the nodes and their subtrees with distance values ​​greater than the query threshold, and return the multivariate time series contained in all leaf nodes with distance values ​​less than or equal to the query threshold as the intermediate candidate set;

[0044] (4) Subsequent processing: taking the intersection of the intermediate candidate sets of the four dimensions to obtain the final candidate set, reading the original data of the final candidate set in sequence through disk I / O, calculating the multivariate dynamic time warping distance between each original data and the query sequence, and returning the data that is less than or equal to the query threshold as the result; wherein the calculation of the multivariate dynamic time warping distance includes the following sub-steps:

[0045] (4.1) Calculate the point-to-point distance matrix of two multivariate time series. Each cell value in the matrix is ​​the Euclidean distance between the sampling vectors at the corresponding positions of the two series.

[0046] (4.2) Use the dynamic programming algorithm to find the optimal curved path on the point-to-point distance matrix so that the cumulative sum of the element values ​​on the path is minimized. The cumulative sum of the path is used as the final distance metric to determine the similarity.

[0047] In addition to industrial process monitoring data, multivariate time series data also includes retail market sales data, environmental monitoring data, astronomical observation data, aerospace radar data, etc.

[0048] The present invention proposes a distributed similarity query method for multivariate time series data. By constructing an index based on prefix feature vectors in each dimension, it can effectively support similarity query based on dynamic time bending, which not only ensures the query accuracy but also improves the query efficiency. The present invention can be widely used in time series data mining tasks in industrial production and daily life.

[0049] A distributed similarity query system for multivariate time series data is applied to a distributed database. The system includes a feature extraction module, a data index construction module, a query processing module and a subsequent processing module, which are respectively used for feature extraction, data index construction, query processing and the subsequent processing. The sub-feature sequence data of each dimension is deployed in the distributed database.

[0050] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed similarity query method for multivariate time series data, applied to industrial monitoring data, characterized by The steps include: (1) Feature extraction: obtain monitoring data of different dimensions at the same time from industrial monitoring data, obtain multivariate time series data of industrial monitoring data within a certain time range based on time flow, and extract features from the time series data of different dimensions to obtain sub-feature series data of each dimension; (2) Data index construction: constructing an index based on the sub-feature sequence data of each dimension; (3) Query processing: obtaining query data of different dimensions at the same time from the industrial query data, obtaining multivariate time series data within its time range based on the time flow of the industrial query data, extracting features from the query data of different dimensions respectively, obtaining query sub-feature sequence data of each dimension, calculating similarity based on the query sub-feature sequence data and the sub-feature sequence data on the index, and taking the industrial monitoring multivariate time series data corresponding to the index node with high similarity as the intermediate candidate set; (4) Subsequent processing: Take the intersection of the intermediate candidate sets queried on the sub-feature data of each dimension, and compare the similarity of the final candidate set corresponding to the original industrial monitoring data and the industrial query data to obtain the query result.

2. The method for distributed similarity query of multivariate time series data according to claim 1, characterized in that: In the step (1), the prefix features of the sub-feature sequence data of each dimension are extracted; in the step (3), the prefix features of the query sub-feature sequence data of each dimension are extracted, and on each dimensional index, the Euclidean distance between the prefix features of the query sub-feature sequence data of the corresponding dimension and the prefix features corresponding to each index node is calculated in turn to determine the similarity.

3. The method for distributed similarity query of multivariate time series data according to claim 2, characterized in that: In the step (1), data normalization is performed on each dimension of the multivariate time series data, and the starting value, final value, maximum value, and minimum value of the prefix sequence of each dimension are extracted as prefix features.

4. The method for distributed similarity query of multivariate time series data according to claim 2, characterized in that: In the step (2), a metric index or a spatial index is constructed on the prefix feature set of each dimension of the multivariate time series data.

5. The method for distributed similarity query of multivariate time series data according to claim 2, characterized in that: In the step (3), the nodes and their subtrees whose distance values ​​are greater than the query threshold are pruned, and the multivariate time series contained in all leaf nodes whose distance values ​​are less than or equal to the query threshold are returned as the intermediate candidate set.

6. The method for distributed similarity query of multivariate time series data according to claim 1, characterized in that: In step (1), z-normalization is performed on each dimension of the multivariate time series according to the following formula: Among them, t i Represents a time series data under the current dimension before normalization, t′ i It represents a time series data in a single dimension after normalization, m represents the average value of all time series data in this dimension, and σ represents the standard deviation of all time series data in this dimension.

7. The method for distributed similarity query of multivariate time series data according to claim 1, characterized in that: In the step (4), the original data of the final candidate set is read in sequence through disk I / O, the multivariate dynamic time warping distance between the multivariate time series data of each original industrial monitoring data and the multivariate time series data of the industrial query data is calculated, and the data that is less than or equal to the query threshold is returned as the result.

8. A distributed similarity query method for multivariate time series data according to claim 7, characterized in that: In the step (4), the calculation of the multivariate dynamic time warping distance includes the following sub-steps: (4.1) Construct a point-to-point distance matrix of two multivariate time series. Each cell value in the matrix is ​​the Euclidean distance between the sampling vectors at the corresponding position of the two series. (4.2) Use the dynamic programming algorithm to find the optimal curved path on the point-to-point distance matrix so that the cumulative sum of the element values ​​on the path is minimized. The cumulative sum of the path is used as the final distance metric to determine the similarity.

9. The method and system for distributed similarity query of multivariate time series data according to claim 1, applied to a distributed database, is characterized in that: The system includes a feature extraction module, a data index construction module, a query processing module and a subsequent processing module, which are respectively used for the feature extraction, the data index construction, the query processing and the subsequent processing. The sub-feature sequence data of each dimension is deployed in the distributed database.

Images