A dynamic time warping-based time series data similarity measurement method and system

Abstract

The application discloses a time series data similarity measurement method and system based on dynamic time warping, relates to the computer technical field, and solves the problems that time series data from the same device or similar devices is not suitable for judging the similarity of non-homologous time series data and that the specific similarity degree of each time series data sample cannot be judged.The application preliminarily screens effective time series data and alternative time series data collected by using normalization processing, calculates the similarity by using dynamic time warping for the part of the effective time series data that cannot be directly judged in the screening, finds the part of the effective time series data with the highest similarity to the alternative time series data, compares the part of the effective time series data one by one, respectively finds the maximum similar subset, and returns the corresponding result.The application uses the common set to preliminarily screen and improves the efficiency when the data amount is large, calculates the similarity by using the dynamic time warping method for the part that cannot be directly judged in the screening, improves the accuracy, and returns the data including the similarity, the similar subset and the similar original data subset.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a method and system for measuring the similarity of time-series data based on dynamic time warping. Background Technology

[0002] With the development of computer technology, much offline recorded data is being transferred to online management. However, as the amount of data increases, management becomes more difficult. Sometimes, it's hard to identify fake data that has been slightly modified from real data. Manual verification is insufficient when dealing with large amounts of data. Therefore, it's necessary to automatically verify and manage data transferred online to determine its authenticity. For example, with supplier data, it has been found in practice that order data from one company has been processed and transferred to other companies. Therefore, before inserting the data into the list database, it's necessary to determine whether the inserted data is credible and perform similarity checks to uncover suspicious data. Time series is a common form of data representation. For time series processing, a common task is comparing the similarity of two series. In time series, the lengths of the two time series to be compared may not be equal. Dynamic time warping methods stretch or shorten (compress) unknown quantities until they match the length of the reference template. During this process, the time axis of the unknown quantities will be distorted or bent so that their features correspond to the standard pattern.

[0003] The closest existing method is to target matching data patterns and determine whether data patterns are reproduced by the similarity between the time series dataset and the dataset in the pattern database; in this way, it is determined whether the time series data has undergone concept shift. If a shift has occurred, a new model is added to the pattern database. For details, see the invention patent with application publication number CN114357037A.

[0004] Existing technologies and methods rely on time-series data originating from the same or similar devices, which exhibit fluctuations with a fixed mean and variance, making them unsuitable for determining similarity between data from different sources. Furthermore, existing technologies and methods rely on pattern databases to find approximate matching patterns, rather than determining the specific degree of similarity for each time-series data sample. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes to provide a time-series data similarity measurement method and system based on dynamic time warping. This aims to solve the problems that existing time-series data similarity measurement methods are not suitable for determining similarity between non-same-origin data and do not determine the specific degree of similarity for each time-series data sample.

[0006] A method for measuring the similarity of time-series data based on dynamic time warping includes the following steps:

[0007] Step 1: Obtain the time series data to be processed. The time series data is divided into valid time series data and candidate time series data. Then, normalize all the obtained time series data.

[0008] Step 2: Select a portion of the valid time series data and a portion of the candidate time series data after normalization in Step 1, combine them, calculate the common set, and then compare and filter them; if the judgment result can be obtained directly from the common set, then there is no need to perform Step 3.

[0009] Step 3: If the common set of the valid time series data and the candidate time series data selected in Step 2 cannot directly determine the result, then the dynamic time warping method is used to perform further similarity measurement to obtain the similarity score.

[0010] Step 4: Repeat steps 2 and 3 until a candidate time series data is selected and filtered between all valid time series data. Based on similarity, select a portion of valid time series data and compare them one by one with the candidate time series data to find the largest subset of similarity between the candidate time series data and each valid time series data. At the same time, restore the largest subset of similarity between the original time series data and return the similarity between the selected valid time series data and the candidate time series data, the largest subset of similarity, and the original data corresponding to the largest subset.

[0011] Step 5: Repeat steps 2 to 4 until all candidate time series data and all valid time series data have been filtered and calculated.

[0012] Preferably, the relationship between the valid time series data and the candidate time series data is that some candidate time series data may include false time series data, and the false time series data is fabricated based on the valid time series data to increase authenticity. Some false time series data only have simple amplitude translation and amplitude scaling compared to the valid time series data.

[0013] Preferably, the normalization process includes performing first-order difference processing on adjacent elements in each time series data to eliminate amplitude shift, and then performing quotient calculation to eliminate amplitude scaling.

[0014] Preferably, based on the time series data processed in step 1, a common set is calculated by combining one valid time series data and one candidate time series data for comparison and screening.

[0015] Preferably, there are three screening results. If the common set of the selected valid time series data and the candidate time series data is completely different in screening result 1, then the selected valid time series data and the candidate time series data are determined to be completely unrelated and the combination is ignored. If the common set of the selected valid time series data and the candidate time series data is completely the same in screening result 2, then the selected valid time series data and the candidate time series data are determined to be completely related, that is, the candidate time series data is determined to be abnormal time series data. If the common set of the selected valid time series data and the candidate time series data is partially the same in screening result 3, then it cannot be directly determined and further similarity measurement is required.

[0016] Preferably, when the abnormal time series data is discovered, the original time series data is directly restored and the corresponding original time series data and abnormal judgment result are returned.

[0017] Preferably, the dynamic time warping method in step 3 involves calculating the distance matrix between each point in the two sequences and finding a path from the top left corner to the bottom right corner of the matrix such that the sum of the elements on the path is minimized.

[0018] Preferably, in step 4, there are two methods for selecting some valid time series data based on similarity: the first is to select the top n valid time series data based on similarity ranking; the second is to determine a threshold based on similarity and select valid time series data that are greater than the threshold, where 1≤n≤10.

[0019] Preferably, in step 2, the data selection rule is to first select one candidate time series data and combine it with all valid time series data in turn. After the screening and judgment and possible dynamic time normalization are completed, the next candidate time series data is selected until all valid time series data are judged.

[0020] A time-series data similarity measurement system based on dynamic time warping includes a data acquisition and processing module and a judgment module. The data acquisition and processing module is used to collect valid time-series data and candidate time-series data, perform normalization processing, and transmit the data to the judgment module. The judgment module includes a filtering module and a dynamic time warping module. The normalized data is first sent to the filtering module, which uses a commonality set to perform filtering and judgment to determine whether the data is abnormal or safe data and then returns the result. If the filtering module encounters data that cannot be directly judged, it is sent to the dynamic time warping module for similarity measurement and then returns the result.

[0021] The beneficial effects of this invention include:

[0022] Using a commonality set for initial screening speeds up the calculation. When the actual data volume is large, those that do not meet the comparison criteria can be excluded first, and then the search can be carried out in the remaining data. The dynamic time warping method can not only find the time series with the highest similarity after processing, but also find the corresponding time series of the original data before processing. Therefore, the return value includes not only similarity, but also similar subsets and similar original data subsets. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a time-series data similarity measurement method based on dynamic time warping, as shown in Example 1.

[0024] Figure 2 This is a flowchart illustrating the process of processing data using the dynamic time warping algorithm in Example 1.

[0025] Figure 3 This is a schematic diagram of the system structure of a time-series data similarity measurement method based on dynamic time warping, as shown in Example 2. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0027] Example 1

[0028] The following is in conjunction with the appendix Figure 1 Specific embodiments of the present invention will be described in detail;

[0029] A method for measuring the similarity of time-series data based on dynamic time warping includes the following steps:

[0030] Step 1: Obtain the time series data to be processed. The time series data is divided into valid time series data and candidate time series data. Then, normalize all the obtained time series data.

[0031] Step 2: Select a portion of the valid time series data and a portion of the candidate time series data after normalization in Step 1, combine them, calculate the common set, and then compare and filter them; if the judgment result can be obtained directly from the common set, then there is no need to perform Step 3.

[0032] Step 3: If the common set of the valid time series data and the candidate time series data selected in Step 2 cannot directly determine the result, then the dynamic time warping method is used to perform further similarity measurement to obtain the similarity score.

[0033] Step 4: Repeat steps 2 and 3 until a candidate time series data is selected and filtered between all valid time series data. Based on similarity, select a portion of valid time series data and compare them one by one with the candidate time series data to find the largest subset of similarity between the candidate time series data and each valid time series data. At the same time, restore the largest subset of similarity between the original time series data and return the similarity between the selected valid time series data and the candidate time series data, the largest subset of similarity, and the original data corresponding to the largest subset.

[0034] Step 5: Repeat steps 2 to 4 until all candidate time series data and all valid time series data have been filtered and calculated.

[0035] Corresponding to step 1 above:

[0036] Data is retrieved from a database already loaded into the computer. The data is mainly divided into two parts: one part consists of valid time-series data already in the list database, and the other part consists of candidate time-series data to be pre-implanted into the list database for verification. The candidate time-series data comes from suppliers. In practice, it has been found that order data from one company is processed and then transferred to other companies. Therefore, before implanting the candidate time-series data into the list database, it is necessary to determine whether the implanted time-series data is reliable and perform similarity checks to uncover questionable time-series data. Some questionable time-series data only exhibit simple amplitude shifting and compression compared to valid time-series data. Data processing mainly involves normalizing the original time-series data to eliminate the effects of amplitude shifting and compression, obtaining a dataset corresponding to the time-series data. The normalization operation principle is as follows:

[0037] First, assume there are two sets of candidate time series data, y1 and y2, which are derived from two sets of valid original time series data, x1 and x2, respectively. x1 and x2 are the same valid original time series data. Then, y1 and y2 are only shifted or scaled by a fixed unit distance on the monetary axis:

[0038] y1=x1

[0039] y2=βx2+μ

[0040] Perform first-order differencing on the original data to eliminate amplitude shifts on the amount axis:

[0041] z 11 =y 12 -y 11 =x 12 -x 11

[0042] z21 =y 22 -y 21 =(βx 22 +μ)-(βx 21 +μ)=β(x 22 -x 21 )

[0043] To eliminate the amplitude scaling effect on the amount axis, the above data results are then divided by two adjacent amount elements.

[0044] w 11 =z 12 / z 11 =(x 13 -x 12 ) / (x 12 -x 11 )

[0045]

[0046] And so on, up to w. 1n w 2n .

[0047] Corresponding to step 2 above:

[0048] After normalization, valid dataset y1 and candidate dataset y2 are selected from the dataset using commonality sets for preliminary screening. The screening process is as follows:

[0049] 1. If the only differences between the candidate dataset y1 and the valid dataset y2 are amplitude translation and amplitude scaling, then the transformed datasets will be completely identical, i.e.:

[0050] w 11 =w 21 ,w 12 =w 22 ,w 13 =w 23 ,…,w 1n =w 2n ;

[0051] At this point, the candidate time series data before normalization can be directly determined to be abnormal data. The candidate dataset and the valid dataset are restored to the candidate time series data and the valid time series data before the transformation. Then, the transformed dataset and the time series data before the transformation are output together.

[0052] 2. If the normalized datasets of candidate time series data y1 and y2 are not completely identical, calculate whether there are any identical elements in the transformed datasets of the two time series to determine whether the data meet the most basic similarity definition. If not, discard that part of the data in advance.

[0053] W1 = {w 11 , w 12 , w 13 , …, w 1n}

[0054] W2 = {w 21 , w 22 , w 23 , …, w 2n}

[0055] W'1 = list(W1); W'2 = list(W2); W' = list(W1 + W2)

[0056] L1 = len(W'1); L2 = len(W'2); L = len(W')

[0057] Among them, W'1, W'2, and W' are the data sets after removing duplicates, and L1, L2, and L are the number of set elements. If L1 + L2 = L, it means there are no identical elements between the two time series, so the pre - comparison time series that meet this condition are screened out;

[0058] 3. If L1 + L2 < L, it indicates that there are identical elements between the two time series, then it is necessary to enter step 3 to use the dynamic time warping algorithm to measure the similarity of the pre - comparison time series that meet the conditions.

[0059] Corresponding to step 3 above:

[0060] After step 2, the common set can directly determine that the screened - out data is the early - discarded data. Step 3 mainly aims at the data that cannot be discarded early in step 2, that is, the combination of the effective data set and the alternative data set that cannot be directly judged, and uses the dynamic time warping method to calculate the similarity. The specific schematic diagram is shown in the appendix Figure 2 , and the specific steps of calculating the similarity using the dynamic time warping method are as follows:

[0061] The two data sets are used as time series Q and C, and their lengths are n and m respectively:

[0062] Q = [q1, q2, …, q n

[0063] C = [c1, c2, …, c m

[0064] If n = m, the distance between the two sequences can be directly calculated to obtain the similarity between the two sequences; if n ≠ m, linear scaling is required, that is, the shorter sequence is linearly enlarged to the same length as the longer sequence, or the longer sequence is linearly shortened to the same length as the shorter sequence, and then compared.

[0065] ​​To align the two sequences, construct an n×m matrix grid, where the element at position (i,j) is q. i and c j The distance d(q) between two points i ,c j The similarity between each point in sequence Q and each point in sequence C is calculated using Euclidean distance, where the smaller the distance, the higher the similarity. Generally, Euclidean distance is used, i.e., d(q). i ,c j )=(q i -c j ) 2 This method can be reduced to finding a path through several grid points in the grid, and the grid points that the path passes through are the points where the two sequences are aligned for computation.

[0066] We define this path as a normalized path and denote it as W, where the k-th element of w is defined as w_k. k =(i,j) k ,

[0067] W = w1, w2, ..., w k ,…,w K max(m,n)≤K <m+n-1

[0068] This path needs to satisfy the following constraints:

[0069] (1) Boundary conditions - The selected path must start from the bottom left corner and end at the top right corner:

[0070] w1 = (1,1)

[0071] w K =(m,n)

[0072] (2) Continuity – can only be aligned with adjacent points:

[0073] If w k-1 If (a′, b′) is a given path, then the next point w on the path is a given path. k = (a, b) needs to satisfy

[0074] (aa′)≤1

[0075] (bb′)≤1

[0076] (3) Monotonicity – Restriction: Points on W must be monotonically increasing with time.

[0077] If w k-1 = (a′, b′), then for the next point w on the path k = (a, b) needs to satisfy:

[0078] 0≤(aa′)

[0079] 0≤(bb′)

[0080] Combining the constraints of continuity and monotonicity, each grid point has only three paths. There are an exponential number of paths that satisfy these constraints, and the path that minimizes the regularization cost is the one that achieves this. It can be obtained through dynamic programming algorithm.

[0081] Corresponding to step 4 above:

[0082] Determine whether the candidate dataset has been combined with all valid datasets to complete the filtering and similarity calculation. If not, repeat steps 2 to 3 until the candidate dataset has been combined with all valid datasets to complete the filtering and similarity calculation. Then, select the top 10 valid datasets in the similarity ranking and compare them one by one with the candidate datasets. Find the largest similar subset of each dataset, restore the original data of the largest similar subset, and return the similarity between the selected valid time series data and the candidate time series data, the largest similar subset, and the original data corresponding to the largest subset.

[0083] Corresponding to step 5 above:

[0084] Determine whether all candidate datasets have been combined with all valid datasets. If not, continue to select the next candidate dataset and repeat steps 2 to 4 until all candidate datasets have been filtered and similarity calculated with all valid datasets.

[0085] This invention uses a commonality set for initial screening to speed up calculations. When the actual amount of data is large, a large number of data that do not meet the comparison criteria can be excluded first, and then the search can be carried out in the remaining data. The dynamic time warping method can not only find the time series with the highest similarity after processing, but also find the time series of the original data before processing. Therefore, the return value includes not only similarity, but also similar subsets and similar original data subsets.

[0086] Example 2

[0087] See attached document Figure 3 , attached Figure 3 The diagram shows the structure of a time-series data similarity measurement system based on dynamic time warping. It should be understood that this system is similar to the one mentioned above. Figure 2 Corresponding to the method implementation examples, it is capable of executing the attached... Figure 2 The various steps involved in the method implementation, and the specific functions of the system, can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here. (See attached...) Figure 3The system shown includes a data acquisition and processing module and a judgment module. The data acquisition and processing module is used to acquire valid time-series data and candidate time-series data, perform first-order difference quotient processing, and transmit the processed data to the judgment module. The judgment module includes a filtering module and a dynamic time warping module. First, it selects one valid dataset and one candidate dataset from the data transmitted from the data acquisition and processing module to the judgment module as a combination and submits it to the filtering module. The filtering module calculates the common set between the two datasets. If the obtained common sets are completely identical, the candidate dataset is determined to be abnormal data, and the two datasets contained in the combination are restored to their original data and the corresponding result is returned. If the obtained common sets are completely different, the candidate dataset is determined to be not abnormal data relative to the valid dataset, and the combination is ignored. If the obtained common sets are partially identical, the situation of the candidate data in the combination cannot be directly determined. The combination is then submitted to the dynamic time warping module to calculate similarity and store it. After a candidate dataset and all valid datasets have been filtered and judged, the valid datasets with the highest similarity to the candidate dataset are found. The maximum similarity subset between the candidate dataset and each of them is calculated. The original data is restored for the maximum similarity subset and the corresponding result is returned.

[0088] This invention uses a commonality set for initial screening to speed up calculations. When the actual amount of data is large, a large number of data that do not meet the comparison criteria can be excluded first, and then the search can be carried out in the remaining data. The dynamic time warping method can not only find the time series with the highest similarity after processing, but also find the time series of the original data before processing. Therefore, the return value includes not only similarity, but also similar subsets and similar original data subsets.

[0089] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. A method for measuring the similarity of time-series data based on dynamic time warping, characterized in that, Includes the following steps: Step 1: Obtain the time series data to be processed. The time series data is divided into valid time series data and candidate time series data. Then, normalize all the obtained time series data. Step 2: Select a portion of the valid time series data and a portion of the candidate time series data after normalization in Step 1, combine them, calculate the common set, and then compare and filter them; if the judgment result can be obtained directly from the common set, then there is no need to perform Step 3. Step 3: If the common set of the valid time series data and the candidate time series data selected in Step 2 cannot directly determine the result, then the dynamic time warping method is used to perform further similarity measurement to obtain the similarity score. Step 4: Repeat steps 2 and 3 until a candidate time series data and all valid time series data are selected for screening calculation. Based on similarity, select a portion of valid time series data and compare them one by one with the candidate time series data. Find the largest subset of similarity between the candidate time series data and each of the selected valid time series data. At the same time, restore the largest subset of similarity between the original time series data and return the similarity between the selected valid time series data and the candidate time series data, the largest subset of similarity, and the original data corresponding to the largest subset. Step 5: Repeat steps 2 to 4 until all candidate time series data and all valid time series data have been filtered and calculated.

2. The method for measuring the similarity of time-series data based on dynamic time warping according to claim 1, characterized in that, In step 1, the relationship between the valid time series data and the candidate time series data is that some candidate time series data may include false time series data, and the false time series data is fabricated based on the valid time series data to increase authenticity. Some false time series data only have simple amplitude translation and amplitude scaling compared to the valid time series data.

3. The method for measuring the similarity of time-series data based on dynamic time warping according to claim 2, characterized in that, The normalization process includes performing first-order difference processing on adjacent elements in each time series data to eliminate amplitude shift, and then performing quotient calculation to eliminate amplitude scaling.

4. The method for measuring the similarity of time-series data based on dynamic time warping according to claim 1, characterized in that, In step 2, based on the time series data processed in step 1, a common set is calculated by combining one valid time series data and one candidate time series data for comparison and screening.

5. The method for measuring the similarity of time-series data based on dynamic time warping according to claim 4, characterized in that, In step 2, there are three possible filtering results. If the selected valid time series data in filtering result 1 is completely different from the candidate time series data, then the selected valid time series data and the candidate time series data are determined to be completely unrelated and the combination is ignored. If the selected valid time series data in filtering result 2 is completely identical to the candidate time series data, then the selected valid time series data and the candidate time series data are determined to be completely related, i.e., the candidate time series data is determined to be abnormal time series data. If the selected valid time series data in filtering result 3 is partially identical to the candidate time series data, then it cannot be directly determined and further similarity measurement is required.

6. The method for measuring the similarity of time-series data based on dynamic time warping according to claim 5, characterized in that, When abnormal time-series data is discovered, the original time-series data is directly restored and the corresponding original time-series data and abnormal judgment result are returned.

7. The method for measuring the similarity of time-series data based on dynamic time warping according to claim 1, characterized in that, The dynamic time warping method in step 3 involves calculating the distance matrix between each point in the two sequences and finding a path from the top left corner to the bottom right corner of the matrix such that the sum of the elements on the path is minimized.

8. The method for measuring the similarity of time-series data based on dynamic time warping according to claim 1, characterized in that, In step 4, there are two methods for selecting some valid time series data based on similarity: the first is to select the top n valid time series data based on similarity ranking; the second is to determine a threshold based on similarity and select valid time series data that are greater than the threshold.

9. The method for measuring the similarity of time-series data based on dynamic time warping according to claim 1, characterized in that, In step 2, the data selection rule is to first select a candidate time series data and combine it with all valid time series data in turn. After the screening and judgment and possible dynamic time normalization are completed, the next candidate time series data is selected until all valid time series data are judged.

10. A time-series data similarity measurement system based on dynamic time warping, characterized in that, It includes a data acquisition and processing module and a judgment module. The data acquisition and processing module is used to collect valid time-series data and candidate time-series data, perform normalization processing, and transmit the data to the judgment module. The judgment module includes a filtering module and a dynamic time warping module. The normalized data is first sent to the filtering module, which uses a commonality set to perform filtering and judgment to determine whether the data is abnormal or safe data and then returns the result. If the filtering module encounters data that cannot be directly judged, it is sent to the dynamic time warping module for similarity measurement and then returns the result.

Images