Streaming time sequence sub-sequence anomaly detection method fusing global and local features

Abstract

The invention belongs to the technical field of time sequence anomaly detection, and particularly relates to a streaming time sequence sub-sequence anomaly detection method fusing global and local features, which comprises the following steps: clustering all sub-sequences in an initial batch, constructing an inverted index, and performing approximate neighbor search on each sub-sequence; the used calculation formula is used for measuring deviation between the subsequences and neighbors of the subsequences and between the subsequences and cluster centers of clusters containing the neighbors of the subsequences and dispersity of neighbor distribution, and global and local feature fusion is achieved; taking each current cluster as a historical cluster, calculating the maximum distance between each sub-sequence in each historical cluster and the cluster center of the cluster, and taking the maximum distance as a density parameter of the historical cluster; and clustering all the subsequences in the latest batch, calculating the Euclidean distance between the cluster center of each new cluster and the cluster center of each historical cluster, carrying out cluster combination based on the comparison of the Euclidean distance and the density parameter of each historical cluster, and repeatedly executing neighbor search and abnormal score calculation. According to the invention, accurate detection of large-scale streaming data can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field related to time series anomaly detection, and more specifically, relates to a streaming time series subsequence anomaly detection method that integrates global and local features. Background Art

[0002] With the rapid development of big data and artificial intelligence technology, time series are increasingly used in finance, medical care, industrial monitoring and other fields. Time series data is recorded in chronological order and can reflect the dynamic changes of the system. However, there are often some abnormal points or abnormal patterns in time series, which may represent errors, rare times or key point changes. The main purpose of time series subsequence anomaly detection is to identify continuous abnormal fragments in time series, rather than single abnormal values, because a single value may be within the normal value range in the overall time series, but continuous similar values ​​may form abnormal patterns. Therefore, accurately detecting abnormal subsequences in time series is very important for industrial production and application. In practical applications, time series often arrive in a streaming manner, and their length is unknown. Real-time anomaly detection is required to reduce response time. When large-scale streaming time series continue to arrive, how to accurately detect anomalies in a short time is what streaming time series subsequence anomaly detection algorithms need to consider and study.

[0003] Subsequence anomaly detection algorithms are usually divided into two categories, neighbor-based methods and pattern-based methods. The neighbor-based method measures the degree of anomaly by the relationship between the subsequence and the nearest neighbor (Euclidean distance). The most widely used one is Matrix Profile, which defines anomaly as the subsequence with the largest distance to the nearest neighbor in the time series, also known as discord. Yel et al. proposed the STAMP algorithm, which uses the fast Fourier transform (FFT) technology to accelerate the calculation of the Euclidean distance between all subsequence pairs in the time series. Lu et al. proposed DAMP, which uses the pruning strategy to reduce the search space and effectively improve the efficiency of anomaly detection. Breunig et al. proposed the LOF algorithm, which focuses on the degree of anomaly of the data point relative to the surrounding neighborhood and defines the degree of anomaly according to the ratio of the local density of the nearest neighbor to its own local density. Kriegel et al. proposed the ABOD algorithm, which considers both angle and distance to measure the degree of anomaly. However, this type of method faces the "twin-freak" problem, that is, similar or identical anomalies in the time series are repeated many times. In this case, the repeated anomalies are likely to become neighbors of each other, resulting in the failure of the above neighbor-based methods.

[0004] Pattern-based methods capture global patterns and discover anomalies based on the distribution of subsequences. Among them, Boniol et al. proposed the NormA and SAND algorithms, which extract the normal pattern of time series through clustering and calculate its anomaly score by the sum of the weighted distances of each subsequence to all cluster centroids. In addition, SAND also proposed a streaming subsequence anomaly detection framework to update the clustering-based normal pattern. Boniol et al. constructed a graph structure for time series, in which each vertex is a pattern formed by similar subsequences, and each edge connects two temporally adjacent vertices. Liu et al. proposed the Isolation Forest algorithm, which uses random space partitioning trees to capture global patterns, in which anomalies located in sparse areas are often isolated at shallow nodes. Pattern-based methods discover anomalies by considering the distribution of subsequences. Therefore, in time series, even if anomalies may appear repeatedly, they are easily distinguished from normal patterns due to their sparse distribution. This type of method can solve the "twin-freak" problem encountered in nearest neighbor-based methods. However, the extraction of global patterns is often performed in a coarse-grained manner, so some minor anomalies that are similar to normal subsequences may be mixed with normal patterns and difficult to detect.

[0005] Existing methods only rely on local or global perspectives. Neighborhood-based methods do not consider the global distribution of subsequences, while pattern-based methods lack attention to local neighborhoods. Therefore, they all fail to solve the time series anomaly detection problem in different fields at the same time, which limits their generalization performance. In addition, most existing methods are designed for static scenarios, where the entire time series is known in advance. However, in practical applications, time series usually arrive in a streaming manner, and historical data needs to be considered for real-time detection. Although a few existing methods can solve the anomaly detection problem in streaming scenarios, their performance is limited due to their single perspective when designing anomaly scores. Summary of the invention

[0006] In view of the above defects or improvement needs of the prior art, the present invention provides a streaming time series subsequence anomaly detection method that integrates global and local features, aiming to overcome the defects of the existing time series subsequence anomaly detection algorithm, which only considers a single local or global feature and has poor anomaly detection performance and cannot adapt to large-scale streaming data.

[0007] To achieve the above objective, according to one aspect of the present invention, a streaming time series subsequence anomaly detection method integrating global and local features is provided, comprising:

[0008] S1. Divide the initial batch of time series data into multiple subsequences; cluster the multiple subsequences to obtain nlist clusters, and construct an inverted list for each cluster to form an inverted index;

[0009] S2. Perform an approximate nearest neighbor search on each subsequence through an inverted index to obtain k nearest neighbors of the subsequence; calculate the abnormal score of each subsequence according to the subsequence abnormal score calculation formula, wherein the calculation formula is used to measure the deviation of the subsequence from its nearest neighbors and the cluster centers containing its nearest neighbors, as well as the dispersion of the nearest neighbor distribution, so as to achieve the fusion of global and local features;

[0010] S3. Treat all current clusters as historical clusters, divide the latest batch of time series data into multiple subsequences, and cluster all subsequences of the batch to obtain nlist new clusters; calculate the Euclidean distance between the cluster center of each new cluster and the cluster center of each historical cluster, merge the new cluster into a historical cluster whose Euclidean distance is less than the density parameter of the historical cluster, and the unmerged new clusters exist as independent clusters. Update the cluster center and density parameters of each merged cluster, delete the expired subsequences in the inverted list, and update the inverted index, where the density parameter of each cluster corresponding to the initial batch is the maximum distance between each subsequence in the cluster of the corresponding cluster and the cluster center of the cluster; repeat S2 based on the updated cluster to obtain the anomaly score of each subsequence in the latest batch of time series data, and realize streaming time series subsequence anomaly detection.

[0011] Further, in S2, the subsequence anomaly score calculation formula is:

[0012]

[0013] Where, AS GF (T q ) represents the subsequence T q Anomaly score of Represents the subsequence T q The jth nearest neighbor of Represents the subsequence T q The set of k nearest neighbors of ; express The cluster center of the cluster; inner product Var[·] represents the calculation of variance.

[0014] Furthermore, in S2, the k nearest neighbors of each subsequence are determined as follows:

[0015] PQ codes are allocated to all current subsequences through product quantization, and the PQ codes of each subsequence are stored in the inverted list, and the inverted index is the IVFPQ index; through the IVFPQ index, an approximate neighbor search is performed on each current subsequence to obtain k neighbors of the subsequence; wherein, the PQ code of each subsequence is allocated as follows:

[0016] Divide all subsequences of the current batch into segments of equal length, and obtain M segments for each subsequence. Cluster the segments of all subsequences in the same segmentation space to obtain K segmentation centers of the corresponding segmentation space.

[0017] Each segment in each subsequence is quantized by the segment center closest to it in the segment space to which the segment belongs, that is, the segment is approximated to the segment center closest to it, and encoded as the integer number of the segment center, thereby obtaining M integer codes of the subsequence to constitute the PQ code of the subsequence. The PQ codebook is the Cartesian product of the set composed of K segment centers in each segment space.

[0018] Further, the quantization results of each segment of each subsequence are used to calculate the anomaly score of the subsequence, that is, the PQ codebook is used to calculate the anomaly score of the subsequence, and the calculation formula of the anomaly score of the subsequence is:

[0019]

[0020] Where, AS GF (T q ) represents the subsequence T q Anomaly score of Represents the subsequence T q The jth nearest neighbor of ; express The quantization result of the i-th segment of ; Var[·] represents the calculation of variance; express The cluster center of the cluster;

[0021] Then, in S2, before calculating the subsequence anomaly score, the method further includes: updating the j-th segment center of the i-th segment space quantized to each subsequence of the latest batch in the current PQ codebook to obtain a new PQ codebook.

[0022] Furthermore, the update method of the jth segment center of the i-th segment space is:

[0023]

[0024] In the formula, represents the jth segment center of the i-th segment space after update, represents the jth segment center of the i-th segment space before updating, Indicates all quantifications including the latest batch and historical batches The number of segments, Indicates that the latest batch subsequence T o The i-th segment Quantized to express The quantization loss, Indicates all quantized in the latest batch The sum of the quantization losses corresponding to the subsequences.

[0025] Furthermore, when S2 is repeatedly executed, the search method for the k nearest neighbors of each subsequence is changed to:

[0026] Construct an inverted list for each new cluster, and quantize each segment in each subsequence of the latest batch by the segment center closest to it in the segment space to which the segment belongs. The segment center comes from the PQ codebook obtained from the historical batch subsequence before the update, and obtain the PQ code of each subsequence of the latest batch; store the PQ code in the inverted list of each new cluster, obtain the IVFPQ index corresponding to all new clusters, and use it as the subsidiary index; use the IVFPQ index corresponding to all historical clusters as the main index;

[0027] Calculate the subsequence T of each neighbor to be found in the latest batch q The distance from the center of each new cluster; according to the attached index, get all subsequences T in the nearest nprobe new clusters o ; In each segment space, calculate the subsequence T q The Euclidean distance between each segment and all the segment centers in the segment space in the PQ codebook corresponding to the historical batch is used to construct a distance table; based on the distance table, the subsequence T is calculated by looking up the table q Respectively with all the subsequences T o The Euclidean distance between them is expressed as: In the formula, Represents the subsequence T o The quantization result of the i-th segment, Represents the subsequence T q The i-th segment of ; the subsequence T with the first k minimum distances o As the preliminary approximate neighbor search results, they are all recorded as the first candidate neighbors, and the Euclidean distance of the kth approximate neighbor is taken as the BSF;

[0028] Calculate the subsequence T q The Euclidean distance between the centers of each historical cluster; get all subsequences in the nearest nprobe historical clusters according to the main index;

[0029] According to the formula Traverse all subsequences in nprobe history clusters as T in the formula o , based on stored calculation results and Directly calculate LB(T o ), Indicates T o The cluster center of the current cluster; select LB(T o ) is less than the T corresponding to BSF o are all used as the second candidate neighbors; and the subsequence T is calculated q The Euclidean distance to each second candidate neighbor;

[0030] The distance subsequence T between all the first candidate nearest neighbors and the second candidate nearest neighbors q The most recent k subsequences are taken as subsequence T q The nearest neighbor of q Approximate nearest neighbor search of the k nearest neighbors of .

[0031] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0032] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the steps of the method described above.

[0033] According to another aspect of the present invention, a computer program product is provided, comprising a computer program or instructions, wherein the computer program or instructions implement the steps of the method described above when executed by a processor.

[0034] In general, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects:

[0035] 1. The present invention proposes a streaming time series subsequence anomaly detection method that integrates global and local features, obtains local features of subsequences through nearest neighbor search, extracts global features based on clustering, and designs anomaly scores in combination with local features and global features, thereby achieving effective capture of complex data patterns, overcoming the defects of existing methods, and improving the accuracy of anomaly detection. In addition, the index update method proposed in the present invention can capture the distribution changes of streaming data in a timely and effective manner, and has a lower index construction time compared to building an index for all data from scratch. Therefore, the streaming time series subsequence anomaly detection method based on global and local features provided by the present invention can accurately measure the degree of abnormality of subsequences and perform real-time detection on streaming time series. It is practical and can be used as a key module in applications such as intelligent monitoring, industrial production anomaly detection, financial risk assessment, and health monitoring.

[0036] 2. The present invention compresses and stores data through product quantization (PQ) and updates the codebook online, thereby improving the efficiency of neighbor search and reducing the storage overhead of streaming data.

[0037] 3. The present invention speeds up the neighbor search process and the calculation of anomaly scores through a dual index structure, thereby improving the response efficiency of the anomaly detection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of a method for detecting anomalies of streaming time series subsequences by integrating global and local features provided by an embodiment of the present invention;

[0039] Figure 2 is a flowchart of an algorithm for initializing an index structure provided by an embodiment of the present invention;

[0040] Figure 3 is a flow chart of an algorithm for updating an index structure provided by an embodiment of the present invention;

[0041] Figure 4 It is a framework diagram of a dual-index structure optimization solution provided by an embodiment of the present invention;

[0042] Figure 5 It is a framework diagram of another streaming time series subsequence anomaly detection method that integrates global and local features provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0044] The following are the basic concepts and definitions involved in the present invention:

[0045] A time series is a series of data points collected in chronological order, usually used to describe changes in data over time. Its formal definition is:

[0046] Time Series: A time series T is an ordered sequence of n data points, expressed as T = [t1,…,t n ],in i∈[1,n], the length of the time series T is expressed as T|=n.

[0047] In real applications, time series are usually collected in a streaming manner. The total length of the time series is unknown and may be infinite. Therefore, a batch of newly arrived data points is usually analyzed, where the batch is defined as:

[0048] Batch: A batch of data points in a time series T is a finite sequence, denoted as B i =[t j ,…,t j+b-1 ], where b is B i The length of two consecutive B i and B i+1 , B i The last data point in B is immediately followed by i+1 The first data point of .

[0049] Subsequence anomaly detection is a critical task that aims to identify consecutive anomalous data points in a time series rather than a single anomalous point. This is because a single data point may fall within the normal range of the entire time series, but a group of similar data points may form an abnormal pattern. Therefore, this paper focuses on subsequence anomaly detection, and a subsequence is defined as:

[0050] Subsequence: In the time series T = [t1,…,t n ], a subsequence T i,l is defined as i The sequence of l consecutive points at the beginning is denoted as T i,l =[t i,…,t i+l-1 ], where there is no ambiguity, T i,l Simplified to T i For a batch B = [t i ,…,t i+b-1 ], the present invention records all possible subsequence sets as where j∈[i,i+bl].

[0051] Embodiment 1

[0052] A streaming time series subsequence anomaly detection method that integrates global and local features, such as Figure 1 As shown, including:

[0053] S1. Divide the initial batch of time series data into multiple subsequences; cluster the multiple subsequences to obtain nlist clusters, and construct an inverted list for each cluster to form an inverted index;

[0054] S2. Perform an approximate nearest neighbor search on each subsequence through an inverted index to obtain k nearest neighbors of the subsequence; calculate the abnormal score of each subsequence according to the subsequence abnormal score calculation formula, wherein the calculation formula is used to measure the deviation of the subsequence from its nearest neighbors and the cluster centers containing its nearest neighbors, as well as the dispersion of the nearest neighbor distribution, so as to achieve the fusion of global and local features;

[0055] S3. Treat all current clusters as historical clusters, divide the latest batch of time series data into multiple subsequences, and cluster all subsequences of the batch to obtain nlist new clusters; calculate the Euclidean distance between the cluster center of each new cluster and the cluster center of each historical cluster, merge the new cluster into a historical cluster whose Euclidean distance is less than the density parameter of the historical cluster, and the unmerged new clusters exist as independent clusters. Update the cluster center and density parameters of each merged cluster, delete the expired subsequences in the inverted list, and update the inverted index, where the density parameter of each cluster corresponding to the initial batch is the maximum distance between each subsequence in the cluster of the corresponding cluster and the cluster center of the cluster; repeat S2 based on the updated cluster to obtain the anomaly score of each subsequence in the latest batch of time series data, and realize streaming time series subsequence anomaly detection.

[0056] In this embodiment, local features are obtained through nearest neighbors. The k Nearest Neighbor Search (kNNS) method provides an efficient search function by building an index structure. In order to better balance the search accuracy and efficiency, some methods choose to solve the approximate k nearest neighbors, namely, k Approximate Nearest Neighbor Search (kANNS), which is defined as:

[0057] k-Approximate Nearest Neighbor Search (kANNS): Given a set of subsequences of length l and the query subsequence T q , and the distance function d(·,·), let is an exact set of k nearest neighbors, satisfying have The goal of k-nearest neighbor search is to As much as possible Make an approximation.

[0058] This embodiment considers both global and local feature anomaly scores during anomaly detection, specifically:

[0059] (1) After obtaining the k nearest neighbors of the subsequence, extract local features

[0060] When designing anomaly scores, simply focusing on local information will make the method vulnerable to repeated similar anomalies, while relying solely on global information may miss minor anomalies similar to normal subsequences. Therefore, this embodiment designs a new anomaly score calculation function that uses the Euclidean distance between a subsequence and its neighbors as local information. Given a subsequence T q , its k nearest neighbors are represented as Then the Euclidean distance between a subsequence and its nearest neighbor It reflects the degree of deviation of a subsequence from its neighborhood, where i∈{1,…,k}.

[0061] (2) Extracting global patterns based on clustering

[0062] This embodiment uses K-means to cluster subsequences and obtain global patterns. Divide it into nlist non-overlapping clusters, each cluster has a cluster center point, that is, the cluster center, the cluster center and the cluster are represented as and C1,…,C nlist Each pattern (i.e., each feature pattern) is represented by a cluster center, and each cluster contains similar subsequences belonging to the corresponding pattern.

[0063] Given a subsequence T q and its neighbor set First select the one containing T q All clusters of neighbors where i∈[1,nlist], then, calculate T q With cluster center The Euclidean distance between To evaluate T q degree of abnormality.

[0064] (3) Combining global and local features

[0065] After obtaining the global, local and local features, if they are simply combined in a multiplication manner, information loss may result. The present invention further improves performance through a sophisticated and effective combination. Simply calculating the absolute deviation of the subsequence from the global pattern and the nearest neighbor is not enough to capture anomalies, and the distribution direction of the subsequence's nearest neighbors in space also plays a vital role. For subsequences in a time series, clusters usually extend along a specific direction to form a non-spherical shape. The subsequence can be regarded as a trajectory in a high-dimensional space, and adjacent subsequences are continuous in time. In a high-dimensional space, normal subsequences are usually distributed along a specific direction, and their nearest neighbors are also distributed in a specific direction, while abnormal subsequences are usually distributed outside the trajectory, and their nearest neighbors are also distributed in different directions. Compared with normal subsequences, the distribution direction of the abnormal subsequence's nearest neighbors is more dispersed, and it forms a larger angle with the nearest neighbors. Since it is difficult to calculate the angle covering all the nearest neighbors in a high-dimensional space, the present invention uses the fluctuation of the angle formed by the subsequence, the cluster center and the nearest neighbor to measure this dispersion. When the distribution of the nearest neighbors of the subsequence is more dispersed, the angle formed is larger, resulting in its large volatility. In addition, the influence of distance factor is equally important.

[0066] Taking into account the influence of angle and distance, it can be preferred that this embodiment proposes an abnormal score calculation function, which is defined as:

[0067]

[0068] Among them, AS GF (T q ) represents the subsequence T q The anomaly score of Represents the subsequence T q The jth nearest neighbor of Represents the subsequence T q The set of k nearest neighbors of ; express The cluster center of the cluster; Var[·] represents the calculation of variance.

[0069] The preferred method is to calculate the inner product because it takes into account both distance and angle information, and also takes into account both global and local features. The distance term measures the deviation of the subsequence from the global pattern and neighbors, while the angle term measures the dispersion of the subsequence's neighbor distribution.

[0070] The method of this embodiment also involves index construction, refer to Figure 2 ,When the initial batch of data comes in, it involves the initialization of the index structure: processing the initial batch of data, building the index structure, and calculating the anomaly score of the initial subsequence.

[0071] Specifically, when the initial batch data When it arrives, this embodiment uses the K-means algorithm to The subsequences in are divided into nlist clusters, that is, And build an inverted list for each cluster, and the inverted index is the set of inverted lists of all clusters.

[0072] The k nearest neighbors of each subsequence in S2 are determined as follows:

[0073] PQ codes are allocated to all current subsequences through product quantization, and the PQ codes of each subsequence are stored in the inverted list, and the inverted index is the IVFPQ index; through the IVFPQ index, an approximate neighbor search is performed on each current subsequence to obtain k neighbors of the subsequence; wherein, the PQ code of each subsequence is allocated as follows:

[0074] Divide all subsequences of the current batch into segments of equal length, and obtain M segments for each subsequence. Cluster the segments of all subsequences in the same segmentation space to obtain K segmentation centers of the corresponding segmentation space.

[0075] Each segment in each subsequence is quantized by the segment center closest to it in the segment space to which the segment belongs, that is, the segment is approximated to the segment center closest to it, and encoded as the integer number of the segment center, thereby obtaining M integer codes of the subsequence to constitute the PQ code of the subsequence. The PQ codebook is the Cartesian product of the set composed of K segment centers in each segment space.

[0076] The inverted index stores the IDs and PQ codes assigned to all subsequences in each cluster, and obtains the IVFPQ index. This PQ code-based index not only significantly reduces the search space of approximate nearest neighbor search, but also captures the global pattern in the initial batch time series.

[0077] In addition, regarding the use of product quantization to compress the initial batch of subsequences. PQ is used to compress the subsequences, reducing memory usage. PQ divides each subsequence into smaller segments and quantizes each segment. Given a subsequence Where l is the length of the subsequence, and PQ divides it evenly into M segments, expressed as Each segment And i∈[1,M], so the entire subspace is divided into M segmented spaces. In each segmented space, the K-means algorithm is used to cluster the segments of all subsequences located in the segmented space to obtain K segment centers, which are recorded as Subsequence T o Each segment in is quantized by the segment center closest to it, using the function ui (·) to indicate, for example Indicates that T o The i-th paragraph in Quantized to the jth segment center closest to it, then Approximated as and is encoded as an integer j. In this way, PQ converts the subsequence T o Quantized to At this time, the subsequence T o The PQ code (i.e. u(T o )) consists of M integers, and the codebook is represented as W = W 1 ×…×W M , where × represents the Cartesian product. It is worth noting that this embodiment does not use residuals for quantization, while residuals are usually used by IVFPQ to reduce quantization loss and improve the accuracy of approximate neighbor search at the cost of reduced efficiency. However, this method does not require high-quality approximate neighbor search results to accurately detect anomalies.

[0078] One way is to store the original subsequence data in the inverted index and use the original subsequence data to calculate the anomaly score. Another preferred implementation is to use the quantization result of the subsequence to approximate the anomaly score. That is, the above PQ codebook is used to calculate the subsequence anomaly score. The calculation formula of the subsequence anomaly score is:

[0079]

[0080] Where, AS GF (T q ) represents the subsequence T q Anomaly score of Represents the subsequence T q The jth nearest neighbor of ; express The quantization result of the i-th segment of ; Var[·] represents the calculation of variance; express The cluster center of the cluster;

[0081] Then, in S2, before calculating the subsequence anomaly score, the method further includes: updating the j-th segment center of the i-th segment space quantized to each subsequence of the latest batch in the current PQ codebook to obtain a new PQ codebook.

[0082] As a preferred implementation, the updating method of the jth segment center of the i-th segment space is:

[0083]

[0084] In the formula, represents the jth segment center of the i-th segment space after update, represents the jth segment center of the i-th segment space before updating, Indicates all quantifications including the latest batch and historical batches The number of segments, Indicates that the latest batch subsequence T o The i-th segment Quantized to express The quantization loss, Indicates all quantized in the latest batch The sum of the quantization losses corresponding to the subsequences of ;

[0085] As a preferred implementation, when S2 is repeatedly executed, the search method for the k nearest neighbors of each subsequence is changed to:

[0086] Construct an inverted list for each new cluster, and quantize each segment in each subsequence of the latest batch by the segment center closest to it in the segment space to which the segment belongs. The segment center comes from the PQ codebook corresponding to the historical batch before the update, and obtain the PQ code of each subsequence of the latest batch; store the PQ code in the inverted list of each new cluster, obtain the IVFPQ index corresponding to all new clusters, and use it as the subsidiary index; use the IVFPQ index corresponding to all historical clusters as the main index;

[0087] Calculate the subsequence T of each neighbor to be found in the latest batch q The distance from the center of each new cluster; according to the attached index, get all subsequences T in the nearest nprobe new clusters o ; In each segment space, calculate the subsequence T q The Euclidean distance between each segment and all the segment centers in the segment space in the PQ codebook corresponding to the historical batch is used to construct a distance table; based on the distance table, the subsequence T is calculated by looking up the table q Respectively with all the subsequences T o The Euclidean distance between them is expressed as: In the formula, Represents the subsequence T o The quantization result of the i-th segment, Represents the subsequence T q The i-th segment of ; the subsequence T with the first k minimum distances o As the preliminary approximate neighbor search results, they are all recorded as the first candidate neighbors, and the Euclidean distance of the kth approximate neighbor is taken as the BSF;

[0088] Calculate the subsequence T qThe Euclidean distance between the centers of each historical cluster; get all subsequences in the nearest nprobe historical clusters according to the main index;

[0089] According to the formula Traverse all subsequences in nprobe history clusters as T in the formula o , based on stored calculation results and Directly calculate LB(T o ), Indicates T o The cluster center of the current cluster; select LB(T o ) is less than the T corresponding to BSF o are all used as the second candidate neighbors; and the subsequence T is calculated q The Euclidean distance to each second candidate neighbor;

[0090] The distance subsequence T between all the first candidate nearest neighbors and the second candidate nearest neighbors q The most recent k subsequences are taken as subsequence T q The nearest neighbor of q Approximate nearest neighbor search of the k nearest neighbors of .

[0091] like Figure 2 Initialization of the index structure shown: processing the initial batch of data, building the index structure, and calculating the anomaly score of the initial subsequence.

[0092] For the initial batch data After arrival, anomaly scores based on global and local features are calculated for each subsequence in chronological order. Specifically, for each subsequence First, an approximate nearest neighbor search is performed to obtain global and local features, and then the anomaly score is calculated.

[0093] Given a subsequence T q , first calculate its distance to the center (cluster center) of each inverted list, and select the nprobe inverted lists with the smallest distance for further evaluation. Then, in each segment space (for example, the i-th segment), calculate The distances from all segment centers are stored in the distance table. q and subsequence T o When the distance between q With T o The quantized distance between them is used to approximate the Euclidean distance between them, that is:

[0094]

[0095] in The subsequence T with the first k minimum distances can be obtained by looking up the distance table. o will be returned as the result of an approximate nearest neighbor search.

[0096] Preferably, the product quantization result of the nearest neighbors is also used when calculating the anomaly score, rather than the original subsequences, which are not stored. Therefore, the anomaly score calculation function is converted to:

[0097]

[0098] However, directly computing the inner product The computational cost will increase linearly with the length of the subsequence. In order to avoid this overhead, the inner product is converted into a distance calculation, which is specifically expressed as:

[0099]

[0100] The first two items can be obtained by approximate nearest neighbor search, and the third item can be pre-calculated and stored in the inverted list, that is, it has been calculated in advance during the clustering process, so the calculation of the anomaly score only requires a constant time and is independent of the subsequence length. Although the purpose of the present invention is to solve the subsequence anomaly detection problem in streaming, it can also solve the subsequence anomaly detection problem in static scenarios, in which case only the complete time series needs to be input as the initial batch.

[0101] like Figure 3 An index update method shown is to receive the continuously incoming streaming data and process it in batches to update the index structure.

[0102] (1) Cluster the new batch data and update the index structure

[0103] In streaming scenarios, time series change over time, so the index structure should also change to adapt to the continuous data changes. However, when a new batch of data arrives, building the IVF index and PQ codebook from scratch will result in a huge time overhead. Therefore, a preferred approach is to design a real-time update mechanism to update the index and codebook.

[0104] For each subsequent batch of newly arrived data The purpose of updating the IVF index is to adjust the center of the inverted list and redistribute the historical subsequences and new subsequences so that the IVF index can provide a more accurate global pattern for streaming subsequence anomaly detection and a more reasonable candidate set for approximate neighbor search.

[0105] If a fixed nlist is used, the best way to update the IVF index is to assign new subsequences to the current inverted list by Euclidean distance and execute the K-means algorithm to iteratively optimize the clustering results. However, clustering historical subsequences and new subsequences in this way will cause high computational overhead, not to mention the need to decode the historical subsequences from PQ codes back to subsequences. Another method is to assign new subsequences to the inverted list and then only update the center point of the inverted list. However, if the new subsequences are far away from the historical subsequences in the subsequence space, they may be frequently assigned to several inverted lists. In the worst case, the subsequences in the new batch may be completely assigned to a certain inverted list, resulting in the failure of two functions of the IVF index, namely capturing global patterns and reducing the search space for approximate neighbor searches. The present invention refers to this phenomenon as cluster explosion. In addition, in the streaming scenario, nlist is a parameter that cannot be correctly set in advance, and the number of clusters should be flexibly changed to better adapt to the distribution of streaming data.

[0106] Therefore, an online update strategy is preferably proposed, which can firstly avoid processing historical subsequences, that is, re-decoding historical subsequences, and secondly avoid cluster explosion. Finally, the number of clusters generated by this strategy is flexible and variable, which adapts to the distribution of streaming data. Specifically, the present invention first updates the new batch data. Perform K-means clustering and Divide into nlist clusters, then merge the new clusters with the existing clusters based on the similarity between the clusters, and finally delete the expired subsequences to avoid the impact of expired data.

[0107] When new batch data When arriving, assume that the existing clustering result is First of all Execute K-means and get nlist clusters, recorded as Then cluster merging is performed. only close enough When the two clusters are merged, that is, only when the Euclidean distance between the cluster centers of the two clusters is small enough (that is, less than the density parameter), the two clusters are merged, otherwise will be treated as separate clusters. However, the ability of existing clusters to absorb new clusters is different. This method tends to be more cautious in merging new clusters for dense clusters and more relaxed for sparse clusters to avoid cluster explosion and obtain more balanced clustering results. Therefore, for each cluster Using the density parameter ρ j To evaluate its density, the parameter is initialized to cluster C j The maximum distance between the neutron sequence and the cluster center, that is, Intuitively, a larger density parameter indicates that the cluster is located in a sparse area. Therefore, when the Euclidean distance between two cluster centers is less than the density parameter, that is, When Merge into cluster C j After merging, the cluster center and density parameters are updated as and Unmerged clusters will be added as separate clusters to Then the new subsequence will be added to the corresponding inverted list. Since only similar clusters are merged, the change of cluster center will be small, so this method does not need to reallocate the historical subsequence (that is, although the cluster center is updated, the present invention does not change the position of the subsequence, that is, the subsequence originally in this cluster is still in this cluster, and the position of the historical subsequence is not considered to be changed, because the change of cluster center is small, so the subsequence is no longer reallocated).

[0108] In addition, this method will also delete some expired data for three reasons. First, since the total length of the streaming time series is unknown and may be infinite, storing all historical subsequences will lead to uncontrollable storage overhead. Second, the infinite growth of the amount of data in the index will also cause the efficiency of approximate neighbor search to continue to decline. Finally, as the time series continues to increase, the normal pattern in the history may no longer be normal. Retaining all historical subsequences in the index will not improve the detection accuracy, but will reduce it. Therefore, only the latest subsequence with a window length of H is retained, and for expired data, it is deleted from the IVFPQ index. When a cluster becomes empty, its corresponding inverted list is deleted from the IVFPQ index.

[0109] (2) Update the product quantization codebook

[0110] PQ is effective in subsequence compression, but it is a static method that cannot adapt to the evolution of time series. In essence, the purpose of updating the PQ codebook is similar to updating the IVF index. The difference is that the number of segment centers in each segment space of PQ is fixed (i.e., K), that is, the number of center points in each segment space should remain unchanged during the codebook update. The present invention uses online product quantization (online PQ) to update the codebook. Online PQ uses the current codebook to product quantize the new subsequence, and then updates the segment center through the quantization error.

[0111] Considering that time series change gradually, assuming that the spatial distribution of subsequences will not change dramatically in a short time (that is, for time series, the observations at adjacent time points are usually similar, so the subsequences generated in a short time are also similar, and the distribution of subsequences will remain relatively stable in a short time), therefore, the current codebook W can be used to quantify the latest subsequence That is, for the subsequence First, perform segmented quantization, and the quantization result in the i-th segment space can be obtained as Then update the codebook. If only one subsequence T is considered o (Assuming that the batch has only one subsequence), in order to minimize the quantization loss in the i-th segment space, Should be updated to If historical data is considered, the quantization loss is quantized to The number of segments is averaged, that is, However, updating the codebook subsequence by subsequence may cause frequent fluctuations in the codebook and produce inconsistent quantization quality within a batch, thus deteriorating performance. Therefore, the codebook is updated in batches to adapt to streaming scenarios. All subsequences in a batch (the number is bl, where b is the batch length and l is the subsequence length) are quantized and the codebook is updated by the overall quantization loss, which is specifically expressed as:

[0112]

[0113] in It includes all the latest batches and historical batches. The number of segments.

[0114] (3) Calculate the anomaly score of the current batch subsequence

[0115] After the inverted index and product quantization codebook are updated, an approximate neighbor search is performed first, and then the anomaly score is calculated for each subsequence in chronological order. The specific process is the same as the above calculation of the anomaly score of the initial batch of subsequences. The calculation of the inner product can be converted into the calculation of the distance, making full use of the intermediate results in the approximate neighbor search process to accelerate the calculation of the anomaly score.

[0116] like Figure 4 The dual-index based optimization shown: speeds up the calculation of anomaly scores based on a dual-index structure.

[0117] (1) Build an attached index for new batch data

[0118] In the streaming subsequence anomaly detection problem, when new batches of data arrive, the most important task is to score each subsequence with low time delay, rather than focusing on maintaining a high-quality index structure. The update of the IVF index and PQ codebook can be completed at any time between the arrival of two consecutive batches (assuming It has been fully arrived, so the next step is to Clustering is performed, and then the clusters are merged and updated, the product quantization codebook is updated, and then the expired subsequences are deleted, and finally the nearest neighbor search is performed and the anomaly score is calculated. In the process of performing the above series of operations, new data It is also constantly arriving, and only needs to be Complete the above operations before reaching the destination. After arrival, Perform the above operations. Therefore, you only need to Complete before arrival A series of operations are sufficient). Therefore, a radical optimization strategy is to exchange the two stages of update and anomaly score calculation. However, this may cause the loss of key information from the new batch data. For example, if a new normal pattern appears in the new batch data, then the subsequence from the normal pattern may be mistakenly judged as abnormal due to the deviation from the original pattern. In order to avoid this problem, the present invention obtains the global and local features of the new batch data, performs clustering and approximate neighbor search, and then postpones the operation of merging clusters and deleting expired data until the anomaly score calculation is completed. The process can be referred to Figure 4 For ease of understanding, the index structure constructed by historical data is called the primary index, and the index structure constructed by new batch data is called the subsidiary index.

[0119] This embodiment constructs an attached IVFPQ index to separately obtain information in the new batch data. Unlike initialization from scratch, the attached IVFPQ index reuses the original codebook to avoid additional overhead. When you arrive, first Perform K-means clustering and obtain nlist clusters, represented as And build a reverse list for each cluster, storing the ID and PQ code of the subsequence in the list. Secondly, the codebook W = W obtained by using the historical subsequence 1 ×…×W M Quantize the new batch subsequences and After segmentation processing, we get Each segment is quantized by the segment center closest to it, and the quantization result is Indicates that Quantized into the i-th segment space W i The jth segment center in , where i∈[1,M], Approximated as and encoded as integer j, T o It is represented by M integers.

[0120] (2) performing an approximate neighbor search in the subsidiary index structure to obtain preliminary approximate neighbors;

[0121] For subsequence The present invention first calculates the distance from the center of each posting list in the attached index, and selects the nearest nprobe posting lists for further evaluation. Then, in each segment space (e.g., the i-th segment), calculate The distances from all segment centers are stored in the distance table. q and subsequence T o When the distance between q With T o The quantized distance between them is used to approximate the Euclidean distance between them, that is:

[0122]

[0123] in It can be obtained by looking up the distance table, and the subsequence with the first k minimum distances will be returned as the result of the preliminary approximate nearest neighbor search, and the distance of the kth approximate nearest neighbor is used as the BSF (Best-So-Far).

[0124] (3) Pruning is performed based on the preliminary approximate nearest neighbor results to speed up the calculation of anomaly scores;

[0125] In order to reduce the distance calculation overhead in the final approximate neighbor search process in the main index, a pruning strategy with a lower bound provided by the dual index structure and the triangle inequality is preferably proposed, that is, firstly perform the approximate neighbor search in the subsidiary index to obtain the BSF, and then use the BSF for pruning to speed up the process of approximate neighbor search and anomaly score calculation in the main index structure. The lower bound based on the dual index structure and the triangle inequality is specifically expressed as:

[0126]

[0127] in and It has been calculated that all subsequences of the history are traversed as T o Once LB(T o ) exceeds BSF, then T o It is impossible to become the final approximate neighbor and can be pruned.

[0128] Since streaming time series changes gradually, the neighbors of a subsequence are likely to be close to it in time. Therefore, the present invention can obtain high-quality BSF with a smaller search space, improve the efficiency of pruning the main index, and accelerate the process of approximate neighbor search. After completing the approximate neighbor search, the present invention searches for each subsequence in the latest batch in chronological order. The specific process of calculating the anomaly score is the same as that of calculating the anomaly score of the initial batch of subsequences mentioned above. The overall framework can be seen in Figure 5 .

[0129] Embodiment 2

[0130] The present application also relates to an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0131] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The memory may be used to store computer programs and / or modules, and the processor may perform various functions of the electronic device by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory.

[0132] The relevant technical solutions are the same as above and will not be described again here.

[0133] Embodiment 3

[0134] The present application also relates to a computer-readable storage medium having a computer program stored thereon, and the steps of the above method are implemented when the computer program is executed by a processor.

[0135] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0136] The relevant technical solutions are the same as above and will not be described again here.

[0137] Embodiment 4

[0138] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the method of the above embodiment of the present application.

[0139] The relevant technical solutions are the same as above and will not be described again here.

[0140] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A streaming time series subsequence anomaly detection method integrating global and local features, characterized in that: include: S1. Divide the initial batch of time series data into multiple subsequences; cluster the multiple subsequences to obtain nlist clusters, and construct an inverted list for each cluster to form an inverted index; S2. Perform an approximate nearest neighbor search on each subsequence through an inverted index to obtain k nearest neighbors of the subsequence; calculate the abnormal score of each subsequence according to the subsequence abnormal score calculation formula, wherein the calculation formula is used to measure the deviation of the subsequence from its nearest neighbors and the cluster centers containing its nearest neighbors, as well as the dispersion of the nearest neighbor distribution, so as to achieve the fusion of global and local features; S3. Treat all current clusters as historical clusters, divide the latest batch of time series data into multiple subsequences, and cluster all subsequences of the batch to obtain nlist new clusters; calculate the Euclidean distance between the cluster center of each new cluster and the cluster center of each historical cluster, merge the new cluster into a historical cluster whose Euclidean distance is less than the density parameter of the historical cluster, and the unmerged new clusters exist as independent clusters. Update the cluster center and density parameters of each merged cluster, delete the expired subsequences in the inverted list, and update the inverted index, where the density parameter of each cluster corresponding to the initial batch is the maximum distance between each subsequence in the cluster of the corresponding cluster and the cluster center of the cluster; repeat S2 based on the updated cluster to obtain the anomaly score of each subsequence in the latest batch of time series data, and realize streaming time series subsequence anomaly detection.

2. The method for detecting anomalies of streaming time series subsequences according to claim 1, characterized in that: In S2, the subsequence anomaly score calculation formula is: Where, AS GF (T q ) represents the subsequence T q The anomaly score of Represents the subsequence T q The jth nearest neighbor of Represents the subsequence T q The set of k nearest neighbors of ; express The cluster center of the cluster; inner product Var[·] represents the calculation of variance.

3. The method for detecting abnormalities in a streaming time series subsequence according to claim 1, characterized in that: In S2, the k nearest neighbors of each subsequence are determined as follows: PQ codes are allocated to all current subsequences through product quantization, the PQ codes of each subsequence are stored in the inverted list, and the inverted index is the inverted product quantization IVFPQ index; through the IVFPQ index, an approximate neighbor search is performed on each current subsequence to obtain k neighbors of the subsequence; wherein the PQ code of each subsequence is allocated as follows: Divide all subsequences of the current batch into segments of equal length, and obtain M segments for each subsequence. Cluster the segments of all subsequences in the same segmentation space to obtain K segmentation centers of the corresponding segmentation space. Each segment in each subsequence is quantized by the segment center closest to it in the segment space to which the segment belongs, that is, the segment is approximated to the segment center closest to it, and encoded as the integer number of the segment center, thereby obtaining M integer codes of the subsequence to constitute the PQ code of the subsequence. The PQ codebook is the Cartesian product of the set composed of K segment centers in each segment space.

4. The method for detecting abnormalities in a streaming time series subsequence according to claim 3, characterized in that: The quantization results of each segment of each subsequence are used to calculate the anomaly score of the subsequence, that is, the PQ codebook is used to calculate the anomaly score of the subsequence. The calculation formula of the anomaly score of the subsequence is: Where, AS GF (T q ) represents the subsequence T q The anomaly score of Represents the subsequence T q The jth nearest neighbor of ; express The quantization result of the i-th segment of ; Var[·] represents the calculation of variance; express The cluster center of the cluster; Then, in S2, before calculating the subsequence anomaly score, the method further includes: updating the j-th segment center of the i-th segment space quantized to each subsequence of the latest batch in the current PQ codebook to obtain a new PQ codebook.

5. The method for detecting abnormalities in a streaming time series subsequence according to claim 4, characterized in that: The update method of the jth segment center of the i-th segment space is: In the formula, represents the jth segment center of the i-th segment space after update, represents the jth segment center of the i-th segment space before updating, Indicates all quantified batches including the latest batch and historical batches The number of segments, Indicates that the latest batch subsequence T o The i-th segment Quantized to express The quantization loss, Indicates all quantized in the latest batch The sum of the quantization losses corresponding to the subsequences.

6. The method for detecting abnormalities in a streaming time series subsequence according to claim 4, characterized in that: When S2 is repeatedly executed, the search method for the k nearest neighbors of each subsequence is changed to: Construct an inverted list for each new cluster, and quantize each segment in each subsequence of the latest batch by the segment center closest to it in the segment space to which the segment belongs. The segment center comes from the PQ codebook obtained from the historical batch subsequence before the update, and obtain the PQ code of each subsequence of the latest batch; store the PQ code in the inverted list corresponding to each new cluster, obtain the IVFPQ index corresponding to all new clusters, and use it as the subsidiary index; use the IVFPQ index corresponding to all historical clusters as the main index; Calculate the subsequence T of each neighbor to be found in the latest batch q The distance from the center of each new cluster; according to the attached index, get all subsequences T in the nearest nprobe new clusters o ; In each segment space, calculate the subsequence T q The Euclidean distance between each segment and all the segment centers in the segment space in the PQ codebook corresponding to the historical batch is used to construct a distance table; based on the distance table, the subsequence T is calculated by looking up the table q Respectively with all the subsequences T o The Euclidean distance between them is expressed as: In the formula, Represents the subsequence T o The quantization result of the i-th segment, Represents the subsequence T q The i-th segment of ; the subsequence T with the first k minimum distances o As the preliminary approximate neighbor search results, they are all recorded as the first candidate neighbors, and the Euclidean distance of the kth approximate neighbor is taken as the BSF; Calculate the subsequence T q The Euclidean distance between the centers of each historical cluster; get all subsequences in the nearest nprobe historical clusters according to the main index; According to the formula Traverse all subsequences in nprobe history clusters as T in the formula o , based on stored calculation results and Directly calculate LB(T o ), Indicates T o The cluster center of the current cluster; select LB(T o ) is less than the T corresponding to BSF o are all used as the second candidate neighbors; and the subsequence T is calculated q The Euclidean distance to each second candidate neighbor; The distance subsequence T between all the first candidate neighbors and the second candidate neighbors q The most recent k subsequences are taken as subsequence T q The nearest neighbor of q Search for k nearest neighbors.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the steps of the method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images