A method for anomaly detection in streaming time-series subsequences that integrates global and local features

Abstract

This invention belongs to the field of time series anomaly detection technology, specifically relating to a streaming time series subsequence anomaly detection method that integrates global and local features. The method includes: clustering all subsequences in an initial batch, constructing an inverted index, and performing an approximate nearest neighbor search for each subsequence; calculating anomaly scores for each subsequence, using a formula to measure the deviation of the subsequence from its nearest neighbors and the cluster centers of each cluster containing its nearest neighbors, as well as the dispersion of the nearest neighbor distribution, thus achieving global and local feature fusion; treating the current clusters as historical clusters, calculating the maximum distance between each subsequence in each historical cluster and the cluster center of that cluster, using this distance as the density parameter of that historical cluster; clustering all subsequences in the latest batch, calculating the Euclidean distance between the new cluster center and the cluster centers of each historical cluster, performing cluster merging based on the comparison of this Euclidean distance with the density parameters of each historical cluster, and repeating the nearest neighbor search and anomaly score calculation. This invention enables accurate detection of large-scale streaming data.

Description

Technical Field

[0001] This invention belongs to the technical field of time series anomaly detection, and more specifically, relates to a method for detecting anomalies in streaming time series subsequences by integrating global and local features. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, time series data is increasingly widely used in finance, healthcare, industrial monitoring, and other fields. Time series data is recorded chronologically and can reflect the dynamic changes of a system. However, time series often contain outliers or abnormal patterns, which may represent errors, rare occurrences, or critical changes. The main purpose of time series subsequence anomaly detection is to identify consecutive abnormal segments within a time series, rather than single outliers. While a single value may fall within the normal range of the overall time series, consecutive similar values ​​may form anomaly patterns. Therefore, accurately detecting anomalous subsequences within a time series is crucial for industrial production and applications. In practical applications, time series often arrive in streaming form, with unknown lengths, requiring real-time anomaly detection to reduce response time. How to accurately detect anomalies in a short time when large-scale streaming time series arrive continuously is a key consideration and research area for streaming time series subsequence anomaly detection algorithms.

[0003] Subsequence anomaly detection algorithms are generally divided into two categories: nearest-neighbor-based methods and pattern-based methods. Nearest-neighbor-based methods measure the degree of anomaly by the relationship (Euclidean distance) between a subsequence and its nearest neighbor. The most widely used is the Matrix Profile, which defines anomalies as subsequences in a time series with the largest distance to their nearest neighbor, also known as discord. Yel et al. proposed the STAMP algorithm, which uses Fast Fourier Transform (FFT) to accelerate the calculation of Euclidean distance between all pairs of subsequences in a time series. Lu et al. proposed DAMP, which uses a pruning strategy to reduce the search space and effectively improve anomaly detection efficiency. Breunig et al. proposed the LOF algorithm, which focuses on the degree of anomaly of a data point relative to its surrounding neighborhood, defining the degree of anomaly based on the ratio of the local density of its nearest neighbor to its own local density. Kriegel et al. proposed the ABOD algorithm, which considers both angle and distance to measure the degree of anomaly. However, these methods face the "twin-freak" problem, where similar or identical anomalies in the time series recur multiple times. In this case, repeatedly occurring anomalies are likely to be neighbors of each other, causing the aforementioned nearest-neighbor-based methods to fail.

[0004] Pattern-based methods detect anomalies by capturing global patterns and considering the distribution of subsequences. Boniol et al. proposed the NormA and SAND algorithms, which extract normal patterns from time series through clustering and calculate anomaly scores by summing the weighted distances of each subsequence to all cluster centroids. Furthermore, SAND proposes a streaming subsequence anomaly detection framework to update cluster-based normal patterns. Boniol et al. constructed graph structures for time series, where each vertex is a pattern formed by similar subsequences, and each edge connects two temporally adjacent vertices. Liu et al. proposed the Isolation Forest algorithm, which uses a random spatial partitioning tree to capture global patterns, where anomalies located in sparse regions are often isolated at shallow nodes. Pattern-based methods detect anomalies by considering the distribution of subsequences; therefore, even if anomalies may occur repeatedly in a time series, their sparse distribution makes them easily distinguishable from normal patterns. These methods can solve the "twin-freak" problem encountered in nearest-neighbor based methods. However, global pattern extraction is often performed in a coarse-grained manner, so some minor anomalies that are similar to normal subsequences may be mixed with normal patterns and difficult to detect.

[0005] Existing methods rely on only local or global perspectives. Neighborhood-based methods fail to consider the global distribution of subsequences, while pattern-based methods lack attention to local neighborhoods. Therefore, none of these methods can simultaneously solve time series anomaly detection problems across different domains, limiting their generalization performance. Furthermore, most existing methods are designed for static scenarios where the entire time series is known in advance. However, in real-world applications, time series typically arrive in streaming form, requiring consideration of historical data for real-time detection. While a few existing methods can solve anomaly detection in streaming scenarios, their performance is limited by the single perspective they employ when designing anomaly scores. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a streaming time-series subsequence anomaly detection method that integrates global and local features. Its purpose is to overcome the defects of existing time-series subsequence anomaly detection algorithms that only consider a single local or global feature, resulting in poor anomaly detection performance and inability to adapt to large-scale streaming data.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for detecting anomalies in streaming time-series subsequences by fusing global and local features is provided, comprising:

[0008] S1. Divide the initial batch time series data into multiple subsequences; cluster the multiple subsequences to obtain nlist clusters, and construct an inverted list for each cluster to form an inverted index;

[0009] S2. Perform an approximate nearest neighbor search on each subsequence using an inverted index to obtain the k nearest neighbors of the subsequence; calculate the anomaly score of each subsequence according to the subsequence anomaly score calculation formula, wherein the calculation formula is used to measure the deviation of the subsequence from its nearest neighbors and the cluster centers of each cluster containing its nearest neighbors, as well as the dispersion of the nearest neighbor distribution, to achieve the fusion of global and local features.

[0010] S3. Treat each current cluster as a historical cluster, divide the latest batch of time series data into multiple subsequences, and cluster all subsequences in the batch to obtain nlist new clusters; calculate the Euclidean distance between the cluster center of each new cluster and the cluster centers of each historical cluster, merge the new cluster into a certain historical cluster whose Euclidean distance is less than the density parameter of a certain historical cluster, and treat the unmerged new clusters as independent clusters, update the cluster centers and density parameters of each merged cluster, delete expired subsequences from the inverted list, and update the inverted index. The density parameter of each cluster corresponding to the initial batch is the maximum distance between each subsequence in the corresponding cluster and the cluster center of that cluster; repeat S2 based on the updated clusters to obtain the anomaly score of each subsequence in the latest batch of time series data, and realize the anomaly detection of streaming time series subsequences.

[0011] Furthermore, in step S2, the formula for calculating the subsequence anomaly score is:

[0012]

[0013] In the formula, AS GF (T q ) represents the subsequence T q Abnormal scores; Represents the subsequence T q The j-th nearest neighbor, Represents the subsequence T q The set of k nearest neighbors; express The cluster center of the cluster; inner product Var[·] indicates the calculation of variance.

[0014] Furthermore, in S2, the method for determining the k nearest neighbors of each subsequence is as follows:

[0015] PQ codes are assigned to all current subsequences through product quantization. The inverted list stores the PQ codes of each subsequence, and the inverted index is the IVFPQ index. Using the IVFPQ index, an approximate nearest neighbor search is performed on each current subsequence to obtain its k nearest neighbors. The PQ code allocation method for each subsequence is as follows:

[0016] Divide all subsequences of the current batch into segments of equal length. Each subsequence corresponds to M segments. Cluster the segments of all subsequences that are located in the same segment space to obtain K segment centers of the corresponding segment space.

[0017] Each segment in each subsequence is quantized by the nearest segment center in the segment space to which the segment belongs, that is, the segment is approximated as the nearest segment center and encoded as the integer number of the segment center, thus obtaining M integer codes for the subsequence, which constitute the PQ code of the subsequence. The PQ codebook is the Cartesian product of the sets of K segment centers in each segment space.

[0018] Furthermore, the quantization results of each segment of each subsequence are used to calculate the anomaly score of that subsequence; that is, the PQ codebook is used to calculate the anomaly score of the subsequence. The formula for calculating the anomaly score of the subsequence is as follows:

[0019]

[0020] In the formula, AS GF (T q ) represents the subsequence T q Abnormal scores; Represents the subsequence T q The j-th nearest neighbor; express The quantization result of the i-th segment; Var[·] represents the calculation of variance; express The cluster center of the cluster;

[0021] In step S2, before calculating the subsequence anomaly score, the method further includes updating the j-th segment center of the i-th segment space quantized to the latest batch of subsequences in the current PQ codebook to obtain a new PQ codebook.

[0022] Furthermore, the update method for the center of the j-th segment in the i-th segment space is as follows:

[0023]

[0024] In the formula, This represents the center of the j-th segment in the updated i-th segment space. This represents the center of the j-th segment of the i-th segment space before the update. This indicates that all quantizations, including the latest batch and historical batches, are included. The number of segments, This indicates that the latest batch of subsequence T will be... o The i-th segment Quantified as express Quantitative loss, This indicates all quantized values ​​in the latest batch. The sum of the quantization losses corresponding to the subsequences.

[0025] Furthermore, when S2 is executed repeatedly, the search method for the k nearest neighbors of each subsequence is changed to:

[0026] For each new cluster, an inverted index is constructed, and each segment in the latest batch of subsequences is quantized by the nearest segment center in the segment space to which the segment belongs. This segment center comes from the PQ codebook obtained from the subsequences of the previous batch before the update, thus obtaining the PQ code for each subsequence of the latest batch. The PQ code is stored in the inverted index of each new cluster, thus obtaining the IVFPQ index corresponding to all new clusters, which is used as a secondary index. The IVFPQ index corresponding to all historical clusters is used as the primary index.

[0027] Calculate the subsequence T of each nearest neighbor to be found in the latest batch. q The distance to the cluster center of each new cluster; based on the auxiliary index, obtain all subsequences T in the nprobe new clusters with the closest distance. o In each segmented space, compute the subsequence T. q Construct a distance table by calculating the Euclidean distances between the centers of all segments in the PQ codebook corresponding to each segment and the historical batch; and then calculate the subsequence T by looking up the table based on this distance table. q Each of the subsequences T o The Euclidean distance between them is expressed as: In the formula, Represents the subsequence T o The quantization result of the i-th segment, Represents the subsequence T q The i-th segment; the subsequence T with the first k minimum distances. o As preliminary search results for approximate nearest neighbors, all are recorded as first candidate nearest neighbors, and the Euclidean distance of the kth approximate nearest neighbor is taken as the BSF.

[0028] Calculate the subsequence T q Calculate the Euclidean distance between the cluster centers of each historical cluster and the cluster center of each historical cluster; retrieve all subsequences in the nprobe historical clusters with the nearest distance based on the primary index;

[0029] According to the formula Traverse all subsequences in nprobe historical clusters as T in this formula o Based on the stored calculation results and Directly calculate LB(T) o ), T represents o The cluster center of the current cluster; select LB(T) o The T corresponding to BSF is less than o All are considered as second candidate nearest neighbors; and the subsequence T is calculated. q Euclidean distance to each second candidate nearest neighbor;

[0030] Distance subsequences T from all first and second candidate nearest neighbors q The most recent k subsequences are taken as subsequence T q The nearest neighbors, completing the subsequence T q Approximate nearest neighbor search of k nearest neighbors.

[0031] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0032] According to another aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the storage medium is located to perform the steps of the method described above.

[0033] According to another aspect of the present invention, a computer program product is provided, comprising a computer program or instructions, characterized in that, when the computer program or instructions are executed by a processor, they implement the steps of the method described above.

[0034] In summary, compared with the prior art, the solutions conceived by this invention have the following main advantages:

[0035] 1. This invention proposes a streaming time series subsequence anomaly detection method that integrates global and local features. It acquires local features of the subsequence through nearest neighbor search, extracts global features based on clustering, and designs anomaly scores by combining local and global features. This effectively captures complex data patterns, overcomes the shortcomings of existing methods, and improves the accuracy of anomaly detection. Furthermore, the index update method proposed in this invention can capture the distribution changes of streaming data in a timely and effective manner, resulting in a lower indexing time compared to building an index from scratch for all data. Therefore, the streaming time series subsequence anomaly detection method based on global and local features provided by this invention can accurately measure the degree of anomaly in subsequences and perform real-time detection of streaming time series. It is practical and can serve as a key module in applications such as intelligent monitoring, industrial production anomaly detection, financial risk assessment, and health monitoring.

[0036] 2. This invention uses Product Quantization (PQ) to compress and store data, and updates the codebook online, thereby improving the efficiency of nearest neighbor search and reducing the storage overhead of streaming data.

[0037] 3. This invention accelerates the nearest neighbor search process and speeds up the calculation of anomaly scores through a dual-index structure, thereby improving the response efficiency of the anomaly detection process. Attached Figure Description

[0038] Figure 1 This is a flowchart of a streaming time-series subsequence anomaly detection method that integrates global and local features, provided by an embodiment of the present invention.

[0039] Figure 2 This is a flowchart of the algorithm for initializing the index structure provided in an embodiment of the present invention;

[0040] Figure 3 This is a flowchart of the algorithm for updating the index structure provided in an embodiment of the present invention;

[0041] Figure 4 This is a framework diagram of the dual-index structure optimization scheme provided in the embodiments of the present invention;

[0042] Figure 5 This is a framework diagram of another streaming time-series anomaly detection method that integrates global and local features, provided by an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0044] The following are the basic concepts and their definitions involved in this invention:

[0045] A time series is a series of data points collected in chronological order, typically used to describe how data changes over time. Its formal definition is:

[0046] Time Series: A time series T is an ordered sequence of n data points, represented as T = [t1, ..., t2]. n ],in The length of time series T is represented as T|=n, where i∈[1,n].

[0047] In practical applications, time series data are typically collected in a streaming manner. The total length of the time series is unknown and may be infinitely large. Therefore, analysis is usually performed on newly arriving batches of data points, where a batch is defined as follows:

[0048] Batch: A batch of data points in a time series T is a finite sequence, denoted as B. i =[t j ,…,t j+b-1 ], where b is B i The length of B, for two consecutive B i and B i+1 B i The last data point in B is immediately followed by B. i+1 The first data point.

[0049] Subsequence anomaly detection is a critical task, aiming to identify consecutive outlier data points in a time series, rather than individual outliers. This is because a single data point may fall within the normal range of the entire time series, but a group of similar data points may form an anomalous pattern. Therefore, this invention focuses on subsequence anomaly detection, where a subsequence is defined as:

[0050] Subsequence: In a time series T = [t1, ..., t] n In the sequence T, a subsequence i,l Defined as from t i The sequence consisting of the first l consecutive points is denoted as T. i,l =[t i,…,t i+l-1 Without causing ambiguity, T i,l Simplified to T i For a batch B = [t] i ,…,t i+b-1 In this invention, the set of all possible subsequences is denoted as... where j∈[i,i+bl].

[0051] Example 1

[0052] A method for anomaly detection in streaming time-series subsequences that integrates global and local features, such as... Figure 1 As shown, it includes:

[0053] S1. Divide the initial batch time series data into multiple subsequences; cluster the multiple subsequences to obtain nlist clusters, and construct an inverted list for each cluster to form an inverted index;

[0054] S2. Perform an approximate nearest neighbor search on each subsequence using an inverted index to obtain the k nearest neighbors of the subsequence; calculate the anomaly score of each subsequence according to the subsequence anomaly score calculation formula, wherein the calculation formula is used to measure the deviation of the subsequence from its nearest neighbors and the cluster centers of each cluster containing its nearest neighbors, as well as the dispersion of the nearest neighbor distribution, to achieve the fusion of global and local features.

[0055] S3. Treat each current cluster as a historical cluster, divide the latest batch of time series data into multiple subsequences, and cluster all subsequences in the batch to obtain nlist new clusters; calculate the Euclidean distance between the cluster center of each new cluster and the cluster centers of each historical cluster, merge the new cluster into a certain historical cluster whose Euclidean distance is less than the density parameter of a certain historical cluster, and treat the unmerged new clusters as independent clusters, update the cluster centers and density parameters of each merged cluster, delete expired subsequences from the inverted list, and update the inverted index. The density parameter of each cluster corresponding to the initial batch is the maximum distance between each subsequence in the corresponding cluster and the cluster center of that cluster; repeat S2 based on the updated clusters to obtain the anomaly score of each subsequence in the latest batch of time series data, and realize the anomaly detection of streaming time series subsequences.

[0056] This embodiment obtains local features through nearest neighbors. The k-nearest neighbor search (kNNS) method provides efficient search functionality by constructing an index structure. To better balance search accuracy and efficiency, some methods choose to solve for approximate k-nearest neighbors, i.e., k-approximate nearest neighbor search (kANNS), which is defined as:

[0057] k-Approximate Nearest Neighbor Search (kANNS): Given a set of subsequences of length l. and the subsequence T to be queried q And the distance function d(·,·), let For an exact set of k nearest neighbors, satisfying have The goal of k-approximate nearest neighbor search is... As much as possible Approximation is performed.

[0058] This embodiment considers both global and local feature anomaly scores during anomaly detection, specifically:

[0059] (1) After obtaining the k nearest neighbors of the subsequence, extract local features.

[0060] When designing anomaly scores, focusing solely on local information makes the method susceptible to repeated similar anomalies, while relying solely on global information may miss subtle anomalies similar to normal subsequences. Therefore, this embodiment designs a novel anomaly score calculation function that utilizes the Euclidean distance between a subsequence and its nearest neighbors as local information. Given a subsequence T q Its k nearest neighbors are represented as Then the Euclidean distance between the subsequence and its nearest neighbor This reflects the degree of deviation of a subsequence from its neighborhood, where i∈{1,…,k}.

[0061] (2) Extract global patterns based on clustering.

[0062] This embodiment uses K-means clustering to obtain a global pattern from the subsequences. The subsequence set consists of all the subsequences. Divide it into nlist disjoint clusters, each cluster having a cluster center, denoted as [cluster center and cluster name]. and C1,…,C nlist Each pattern (i.e., each feature pattern) is represented by a cluster center, and each cluster contains similar subsequences belonging to the corresponding pattern.

[0063] Given a subsequence T q and its nearest neighbor set First select the option containing T. q All neighboring clusters Where i∈[1,nlist], then calculate T. q With cluster center Euclidean distance between Used to evaluate T q The degree of abnormality.

[0064] (3) Combining global and local features

[0065] After acquiring global and local features, simply combining them by multiplication may lead to information loss. This invention further improves performance through a clever and effective combination. Simply calculating the absolute deviation of a subsequence from the global pattern and its nearest neighbors is insufficient to capture anomalies; the spatial distribution direction of the subsequence's nearest neighbors also plays a crucial role. For subsequences in a time series, clusters typically extend along specific directions, forming non-spherical shapes. Subsequences can be viewed as trajectories in a high-dimensional space, with adjacent subsequences being temporally continuous. In high-dimensional space, normal subsequences are usually distributed along specific directions, and their nearest neighbors are also distributed along specific directions, while anomalous subsequences are usually distributed outside these trajectories, and their nearest neighbors are distributed in different directions. Compared to normal subsequences, the distribution direction of the nearest neighbors of anomalous subsequences is more dispersed, forming larger angles with their neighbors. Since it is difficult to calculate angles covering all nearest neighbors in high-dimensional space, this invention uses the fluctuation of the angles formed by the subsequences, cluster centers, and nearest neighbors to measure this dispersion. When the distribution of the nearest neighbors of a subsequence is more dispersed, the angles formed are larger, resulting in greater volatility. In addition, distance is also an important factor.

[0066] Taking into account the influence of angle and distance, this embodiment proposes an anomaly score calculation function, which is defined as the preferred option:

[0067]

[0068] Among them, AS GF (T q ) represents the subsequence T q Abnormal scores; Represents the subsequence T q The j-th nearest neighbor, Represents the subsequence T q The set of k nearest neighbors; express The cluster center of the cluster; Var[·] indicates the calculation of variance.

[0069] This preferred method chooses to calculate the inner product because it considers both distance and angle information, as well as both global and local features. Specifically, the inner product... The distance term measures the deviation of the subsequence from the global pattern and its nearest neighbors, while the angle term measures the dispersion of the subsequence's nearest neighbor distribution.

[0070] This embodiment also involves index construction, see reference... Figure 2 When the initial batch of data arrives, the initialization of the index structure is involved: processing the initial batch of data, building the index structure, and calculating the anomaly scores of the initial subsequences.

[0071] Specifically, when the initial batch data Upon arrival, this embodiment uses the K-means algorithm to... The subsequences in the data are divided into nlist clusters, i.e. An inverted list is constructed for each cluster, with the inverted index being the set of inverted lists for all clusters.

[0072] The method for determining the k nearest neighbors of each subsequence in S2 is as follows:

[0073] PQ codes are assigned to all current subsequences through product quantization. The inverted list stores the PQ codes of each subsequence, and the inverted index is the IVFPQ index. Using the IVFPQ index, an approximate nearest neighbor search is performed on each current subsequence to obtain its k nearest neighbors. The PQ code allocation method for each subsequence is as follows:

[0074] Divide all subsequences of the current batch into segments of equal length. Each subsequence corresponds to M segments. Cluster the segments of all subsequences that are located in the same segment space to obtain K segment centers of the corresponding segment space.

[0075] Each segment in each subsequence is quantized by the nearest segment center in the segment space to which the segment belongs, that is, the segment is approximated as the nearest segment center and encoded as the integer number of the segment center, thus obtaining M integer codes for the subsequence, which constitute the PQ code of the subsequence. The PQ codebook is the Cartesian product of the sets of K segment centers in each segment space.

[0076] The inverted index stores the IDs and PQ codes assigned to all subsequences in each cluster, resulting in the IVFPQ index. This PQ code-based index not only significantly reduces the search space of the approximate nearest neighbor search, but also captures the global patterns in the initial batch of time series.

[0077] Additionally, regarding the use of product quantization (PQ) to compress the initial batch of subsequences, PQ reduces memory usage. PQ divides each subsequence into smaller segments and quantizes each segment. Given a subsequence... Where l is the length of the subsequence, PQ divides it evenly into M segments, denoted as: Each segment And i∈[1,M], thus dividing the entire subspace into M segment spaces. Within each segment space, the K-means algorithm is used to cluster all segments of the subsequence located in that segment space, obtaining K segment centers, denoted as . Subsequence T o Each segment in the algorithm is quantized by the nearest segment center, using the function u.i (·) is used to represent, for example Indicates T o The i-th segment Quantize to the j-th segment center that is closest to it, then Approximated as And encoded as an integer j. In this way, PQ will convert the subsequence T o Quantified as At this time, the subsequence T o PQ code (i.e., u(T) o The codebook consists of M integers and is represented as W = W 1 ×…×W M , where × represents the Cartesian product. It is worth noting that this embodiment does not use residuals for quantization, while residuals are typically used in IVFPQ to reduce quantization loss and improve the accuracy of approximate nearest neighbor search, at the cost of reduced efficiency. However, this method does not require high-quality approximate nearest neighbor search results to accurately detect anomalies.

[0078] One approach is to store the original subsequence data in the inverted index and use the original subsequence data to calculate the anomaly score. Another preferred implementation is to use the quantization result of the subsequence to approximate the anomaly score, that is, the aforementioned PQ codebook is used for calculating the subsequence anomaly score. The formula for calculating the subsequence anomaly score is:

[0079]

[0080] In the formula, AS GF (T q ) represents the subsequence T q Abnormal scores; Represents the subsequence T q The j-th nearest neighbor; express The quantization result of the i-th segment; Var[·] represents the calculation of variance; express The cluster center of the cluster;

[0081] In step S2, before calculating the subsequence anomaly score, the method further includes updating the j-th segment center of the i-th segment space quantized to the latest batch of subsequences in the current PQ codebook to obtain a new PQ codebook.

[0082] As a preferred implementation, the update method for the center of the j-th segment of the i-th segment space is as follows:

[0083]

[0084] In the formula, This represents the center of the j-th segment in the updated i-th segment space. This represents the center of the j-th segment in the i-th segment space before the update. This indicates that all quantizations, including the latest batch and historical batches, are included. The number of segments, This indicates that the latest batch of subsequence T will be... o The i-th segment Quantified as express Quantitative loss, This indicates all quantized values ​​in the latest batch. The sum of the quantization losses corresponding to the subsequences;

[0085] As a preferred implementation, when S2 is repeated, the search method for the k nearest neighbors of each subsequence is changed to:

[0086] For each new cluster, an inverted index is constructed, and each segment in the latest batch of subsequences is quantized by the nearest segment center in the segment space to which the segment belongs. This segment center comes from the PQ codebook corresponding to the historical batch before the update, thus obtaining the PQ code for each subsequence in the latest batch. The PQ code is stored in the inverted index of each new cluster, thus obtaining the IVFPQ index corresponding to all new clusters, which is used as a secondary index. The IVFPQ index corresponding to all historical clusters is used as the primary index.

[0087] Calculate the subsequence T of each nearest neighbor to be found in the latest batch. q The distance to the cluster center of each new cluster; based on the auxiliary index, obtain all subsequences T in the nprobe new clusters with the closest distance. o In each segmented space, compute the subsequence T. q Construct a distance table by calculating the Euclidean distances between the centers of all segments in the PQ codebook corresponding to each segment and the historical batch; and then calculate the subsequence T by looking up the table based on this distance table. q Each of the subsequences T o The Euclidean distance between them is expressed as: In the formula, Represents the subsequence T o The quantization result of the i-th segment, Represents the subsequence T q The i-th segment; the subsequence T with the first k minimum distances. o As preliminary search results for approximate nearest neighbors, all are recorded as first candidate nearest neighbors, and the Euclidean distance of the kth approximate nearest neighbor is taken as the BSF.

[0088] Calculate the subsequence T qCalculate the Euclidean distance between the cluster centers of each historical cluster and the cluster center of each historical cluster; retrieve all subsequences in the nprobe historical clusters with the nearest distance based on the primary index;

[0089] According to the formula Traverse all subsequences in nprobe historical clusters as T in this formula o Based on the stored calculation results and Directly calculate LB(T) o ), T represents o The cluster center of the current cluster; select LB(T) o The T corresponding to BSF is less than o All are considered as second candidate nearest neighbors; and the subsequence T is calculated. q Euclidean distance to each second candidate nearest neighbor;

[0090] Distance subsequences T from all first and second candidate nearest neighbors q The most recent k subsequences are taken as subsequence T q The nearest neighbors, completing the subsequence T q Approximate nearest neighbor search of k nearest neighbors.

[0091] like Figure 2 The initialization of the index structure shown: process the initial batch of data, build the index structure, and calculate the anomaly scores of the initial subsequences.

[0092] For the initial batch data Upon arrival, anomaly scores based on global and local features are calculated for each subsequence in chronological order. Specifically, for each subsequence... First, an approximate nearest neighbor search is performed to obtain global and local features, and then anomaly scores are calculated.

[0093] Given a subsequence T q First, calculate the distance between the index and the center (cluster center) of each inverted list, and select the nprobe inverted lists with the smallest distance for further evaluation. Then, in each segmented space (e.g., the i-th segment), calculate... The distances to the centers of all segments are stored in a distance table. When calculating T... q and subsequence T o When calculating the distance between them, T is used. q With T o The quantized distance between them is used to approximate the Euclidean distance between them, that is:

[0094]

[0095] in The subsequence T with the k minimum distances can be obtained by looking up a distance table. o It will be returned as the result of an approximate nearest neighbor search.

[0096] Preferably, the product quantization result of nearest neighbors is used instead of the original subsequences, which are not stored, when calculating the outlier score. Therefore, the outlier score calculation function is transformed into:

[0097]

[0098] However, directly calculating the inner product This causes the computational cost to increase linearly with the length of the subsequence. To avoid this overhead, the inner product is converted into a distance calculation, specifically expressed as:

[0099]

[0100] The first two terms can be obtained through approximate nearest neighbor search, and the third term can be pre-calculated and stored in the inverted list, meaning it is pre-calculated during the clustering process. Therefore, the calculation of the anomaly score only requires constant time and is independent of the subsequence length. Although the purpose of this invention is to solve the subsequence anomaly detection problem in streaming media, it can also solve the subsequence anomaly detection problem in static scenarios, in which case only the complete time series needs to be used as the initial batch input.

[0101] like Figure 3 One index update method shown is to receive continuously arriving streaming data, process it in batches, and update the index structure.

[0102] (1) Cluster the new batch of data and update the index structure.

[0103] In streaming scenarios, time series data constantly changes over time, so the index structure should also change accordingly to adapt to the continuous data changes. However, building the IVF index and PQ codebook from scratch when new batches of data arrive would result in a huge time overhead. Therefore, a preferred approach is to design a real-time update mechanism to update the index and codebook.

[0104] For each subsequent batch of data The purpose of updating the IVF index is to adjust the center of the inverted list and redistribute historical and new subsequences so that the IVF index can provide a more accurate global pattern for streaming subsequence anomaly detection and a more reasonable candidate set for approximate nearest neighbor search.

[0105] If a fixed nlist is used, the optimal way to update the IVF index is to assign new subsequences to the current inverted index using Euclidean distance and perform a K-means algorithm to iteratively optimize the clustering results. However, performing clustering on both historical and new subsequences in this way incurs high computational overhead, not to mention the need to decode historical subsequences back from PQ codes. Another approach is to assign new subsequences to the inverted index and then only update the centroids of the inverted index. However, if new subsequences are far from historical subsequences in the subsequence space, they may be frequently assigned to a few inverted indexes. In the worst case, subsequences in a new batch may be completely assigned to a single inverted index, causing two functions of the IVF index to fail: capturing global patterns and reducing the search space for approximate nearest neighbor searches. This phenomenon is termed cluster explosion in this invention. Furthermore, in streaming scenarios, nlist is a parameter that cannot be correctly pre-set; the number of clusters should be flexibly varied to better adapt to the distribution of streaming data.

[0106] Therefore, an online update strategy is preferably proposed. This strategy firstly avoids processing historical subsequences (i.e., re-decoding historical subsequences), secondly avoids cluster explosion, and finally, the number of clusters generated by this strategy is flexible and variable, adapting to the distribution of streaming data. Specifically, this invention first updates the new batch of data... Perform K-means clustering, and The data is divided into nlist clusters. Then, new clusters are merged with existing clusters based on the similarity between them. Finally, expired subsequences are deleted to avoid the impact of expired data.

[0107] When new batch data Upon arrival, assuming the existing clustering results are... First of all Performing K-means yields nlist clusters, denoted as Then cluster merging is performed, for clusters only close enough Two clusters are merged only when the Euclidean distance between their cluster centers is sufficiently small (i.e., less than the density parameter). These will be treated as separate clusters. However, existing clusters have varying abilities to absorb new clusters; this method tends to be more cautious in merging dense clusters and more lenient in sparse clusters to avoid cluster explosion and achieve more balanced clustering results. Therefore, for each cluster... Using density parameter ρ j To assess its density, this parameter is initialized to cluster C. j The maximum distance between the neutron sequence and the cluster center, i.e. Intuitively, a larger density parameter indicates that the cluster is located in a sparse region. Therefore, when the Euclidean distance between two cluster centers is smaller than the density parameter, i.e. At that time, the cluster Merge into cluster C j In the middle. After merging, the updates to the cluster centers and density parameters are respectively... and Unmerged clusters will be added as independent clusters. Then the new subsequence will be added to the corresponding inverted list. Since only similar clusters are merged, the change in cluster centers will be small, so this method does not need to reallocate historical subsequences (i.e., although the cluster centers are updated, this invention does not change the position of the subsequences, that is, the subsequences that were originally in this cluster are still in this cluster. We do not consider changing the position of historical subsequences because the change in cluster centers is very small, so we do not reallocate the subsequences).

[0108] Furthermore, this method also deletes some expired data for three reasons. First, since the total length of the streaming time series is unknown and potentially infinite, storing all historical subsequences would lead to uncontrollable storage overhead. Second, the infinite growth of data in the index would cause the efficiency of approximate nearest neighbor search to continuously decline. Finally, as the time series increases, normal patterns in the past may no longer be normal; retaining all historical subsequences in the index will not improve detection accuracy but will instead reduce it. Therefore, only the latest subsequence with a window length of H is retained, and expired data is deleted from the IVFPQ index. When a cluster becomes empty, its corresponding inverted list is deleted from the IVFPQ index.

[0109] (2) Update the product quantization codebook

[0110] PQ (Programmable Quantization) is highly effective in subsequence compression, but it is a static method and cannot adapt to the evolution of time series. Essentially, updating the PQ codebook is similar to updating the IVF index. The difference lies in that the number of segment centers in each segment space of PQ is fixed (i.e., K), meaning the number of center points in each segment space should remain constant during codebook updates. This invention employs online product quantization (online PQ) to update the codebook. Online PQ uses the current codebook to perform product quantization on the new subsequence, and then updates the segment centers based on the quantization error.

[0111] Considering that time series change gradually, and assuming that the spatial distribution of subsequences will not change drastically in a short period of time (i.e., for time series, observations at adjacent time points are usually quite similar, so subsequences generated in a short period of time are also quite similar, and the distribution of subsequences will remain relatively stable in a short period of time), the current codebook W can be used to quantify the latest arriving subsequence. That is, for subsequences First, segmented quantization is performed to obtain the quantization result in the i-th segment space. Then update the codebook. If only one subsequence T is considered... o (Assuming the batch has only one subsequence), in order to minimize the quantization loss in the i-th segment space... It should be updated to If historical data is considered, the quantization loss is obtained by quantizing to the i-th segment space. The number of segments is averaged, that is However, updating the codebook subsequence by subsequence can lead to frequent fluctuations in the codebook and inconsistent quantization quality within a batch, thus degrading performance. Therefore, a batch approach is adopted to update the codebook to suit streaming scenarios. All subsequences in a batch (number b, where b is the batch length and l is the subsequence length) are quantized, and the codebook is updated using the overall quantization loss, specifically as follows:

[0112]

[0113] in It includes all quantified data from the latest batch and historical batches. The number of segments.

[0114] (3) Calculate the outlier score of the current batch subsequence.

[0115] After the inverted index and product quantization codebook are updated, an approximate nearest neighbor search is performed first, and then anomaly scores are calculated for each subsequence in chronological order. The specific process is the same as the calculation of anomaly scores for the initial batch of subsequences, but the calculation of the inner product can be converted into a distance calculation, making full use of the intermediate results in the approximate nearest neighbor search process to accelerate the calculation of anomaly scores.

[0116] like Figure 4 The optimization shown is based on a dual-index structure to accelerate the calculation of anomaly scores.

[0117] (1) Build a secondary index for the new batch of data.

[0118] In the problem of anomaly detection in streaming subsequences, the most important task when new batches of data arrive is to score each subsequence with low time latency, rather than focusing on maintaining a high-quality index structure. Updates to the IVF index and PQ codebook can be completed at any time between the arrival of two consecutive batches (assuming...). We've arrived completely, so the next step is to... The process involves clustering, merging and updating clusters, updating the product quantization codebook, deleting expired subsequences, and finally performing nearest neighbor search and calculating anomaly scores. During this series of operations, new data... It is also constantly arriving, and only needs to be in The above steps can be completed before the destination is fully reached. Upon arrival, then... Perform the above steps. Therefore, it is only necessary to... Complete before fully arriving (This involves a series of operations). Therefore, an aggressive optimization strategy involves two stages: exchanging updates and calculating anomaly scores. However, this may lead to the loss of crucial information from new batches of data. For example, if a new normal pattern appears in the new batch of data, subsequences from that normal pattern may be incorrectly identified as anomalies due to deviations from the original pattern. To avoid this problem, this invention acquires the global and local features of the new batch of data and performs clustering and approximate nearest neighbor search. Then, the operations of merging clusters and deleting expired data are postponed until after the anomaly score calculation is completed. The process can be referred to... Figure 4 The diagram illustrates one possible implementation. For ease of understanding, the index structure built from historical data is called the primary index, and the index structure built from new batch data is called the secondary index.

[0119] This embodiment constructs a supplementary IVFPQ index to retrieve information from new batch data separately. Unlike initialization from scratch, the supplementary IVFPQ index reuses the original codebook to avoid additional overhead. When new batch data... Upon arrival, the first thing to do is... Performing K-means clustering yields nlist clusters, denoted as For each cluster, an inverted index is constructed to store the ID and PQ code of the subsequence to which the list belongs. Next, the codebook W = W... 1 ×…×W M Quantize the new batch of subsequences, and then quantize the subsequences. Perform segmentation processing to obtain Each segment is quantized by its nearest segment center, and the quantization result is: Indicates will Quantized into the i-th segment space W i The j-th segment center in the matrix, where i∈[1,M], Approximated as And encoded as integers j, T o It is represented by M integers.

[0120] (2) Perform an approximate nearest neighbor search in the auxiliary index structure to obtain preliminary approximate nearest neighbors;

[0121] For subsequence This invention first calculates the distance to the center of each inverted list in the subordinate index, and selects the nearest nprobe inverted lists for further evaluation. Then, in each segment space (e.g., the i-th segment), it calculates... The distances to the centers of all segments are stored in a distance table. When calculating T... q and subsequence T o When considering the distance between them, this invention calculates T. q With T o The quantized distance between them is used to approximate the Euclidean distance between them, that is:

[0122]

[0123] in This can be obtained by looking up a distance table. The subsequence with the top k minimum distances will be returned as the result of the initial approximate nearest neighbor search, and the distance of the kth approximate nearest neighbor will be taken as the BSF (Best-So-Far).

[0124] (3) Prune the branches based on the preliminary approximate nearest neighbor results to speed up the calculation of outlier scores;

[0125] To reduce the distance calculation overhead during the final approximate nearest neighbor search in the main index, a preferred pruning strategy based on the lower bound provided by the dual index structure and the triangle inequality is proposed. This involves first performing an approximate nearest neighbor search in the secondary index to obtain the BSF (Browser-Side Frame), and then using the BSF for pruning. This accelerates the approximate nearest neighbor search and anomaly score calculation process in the main index structure. The lower bound based on the dual index structure and the triangle inequality is specifically expressed as follows:

[0126]

[0127] in and It has been calculated that traversing all subsequences of the history as T o Once LB(T) o If T exceeds BSF, then T o It cannot become the final near neighbor; it can be pruned.

[0128] Since streaming time series change gradually, the nearest neighbors of a subsequence are likely to be close to it in time. Therefore, this invention can obtain a high-quality BSF with a smaller search space, improve the pruning efficiency of the main index, and accelerate the approximate nearest neighbor search process. After completing the approximate nearest neighbor search, this invention processes each subsequence in the latest batch in chronological order. The calculation of anomaly scores follows the same process as calculating the anomaly scores of the initial batch subsequences. The overall framework can be found in [link to relevant documentation]. Figure 5 .

[0129] Example 2

[0130] This application also relates to an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0131] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.

[0132] The relevant technical solutions are the same as above, and will not be repeated here.

[0133] Example 3

[0134] This application also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0135] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0136] The relevant technical solutions are the same as above, and will not be repeated here.

[0137] Example 4

[0138] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method described in the above embodiments of this application.

[0139] The relevant technical solutions are the same as above, and will not be repeated here.

[0140] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting anomalies in streaming time-series subsequences by fusing global and local features, characterized in that, include: S1. Divide the initial batch of time-series data into multiple subsequences; Clustering the multiple subsequences yields... Each cluster is used to construct an inverted list, forming an inverted index; S2. Perform an approximate nearest neighbor search on each subsequence using the inverted index to obtain the subsequence's... k The anomaly score of each subsequence is calculated according to the subsequence anomaly score calculation formula. The calculation formula is used to measure the deviation of the subsequence from its nearest neighbors and the cluster centers of each cluster containing its nearest neighbors, as well as the dispersion of the nearest neighbor distribution, so as to achieve the fusion of global and local features. S3. Treat all current clusters as historical clusters, divide the latest batch of time-series data into multiple sub-sequences, and cluster all sub-sequences of this batch to obtain... S2 is performed on each new cluster. The Euclidean distance between the cluster center of each new cluster and the cluster centers of each historical cluster is calculated. The new cluster is merged into a historical cluster whose Euclidean distance is less than the density parameter of the historical cluster. The unmerged new clusters exist as independent clusters. The cluster centers and density parameters of each merged cluster are updated. Expired subsequences in the inverted list are deleted. The inverted index is updated. The density parameter of each cluster in the initial batch is the maximum distance between each subsequence in the corresponding cluster and the cluster center of that cluster. Based on the updated clusters, S2 is repeated to obtain the anomaly score of each subsequence in the latest batch of time series data, thereby realizing the anomaly detection of streaming time series subsequences.

2. The streaming time-series subsequence anomaly detection method as described in claim 1, characterized in that, In S2, the formula for calculating the subsequence anomaly score is as follows: In the formula, Representing a subsequence Abnormal scores; Representing a subsequence The A neighbor, , Representing a subsequence of A set of nearest neighbors; express The cluster center of the cluster; inner product ; This indicates the calculation of variance.

3. The streaming time-series subsequence anomaly detection method as described in claim 1, characterized in that, In S2, for each subsequence The method for determining the nearest neighbors is as follows: PQ codes are assigned to all current subsequences through product quantization. The inverted list stores the PQ codes of each subsequence, and the inverted index is the inverted product quantization IVFPQ index. Using the IVFPQ index, an approximate nearest neighbor search is performed on each current subsequence to obtain the subsequence's... There are 1 nearest neighbors; and the PQ code of each subsequence is allocated as follows: Divide all subsequences of the current batch into equal-length segments. Each subsequence corresponds to... Each subsequence is divided into segments, and segments that belong to the same segment space are clustered to obtain the corresponding segment space. Each segment center; Each segment in each subsequence is quantized by the nearest segment center in its segment space, that is, the segment is approximated as the nearest segment center and encoded as the integer number of that segment center, thus obtaining the subsequence. A set of integer codes forms the PQ code for this subsequence. The PQ codebook is composed of the segments in the space. The Cartesian product of the sets formed by the centers of the segments.

4. The streaming time-series subsequence anomaly detection method as described in claim 3, characterized in that, The quantization results of each segment of each subsequence are used to calculate the anomaly score of that subsequence. That is, the PQ codebook is used to calculate the anomaly score of the subsequence. The formula for calculating the anomaly score of the subsequence is: In the formula, Representing a subsequence Abnormal scores; Representing a subsequence The A close neighbor; , express The The quantization results for each segment; Represents the calculation of variance; express The cluster center of the cluster; Indicates Euclidean distance; In step S2, before calculating the subsequence anomaly score, the method further includes: quantizing the latest batch of each subsequence in the current PQ codebook to the first... The first segment of the space Each segment center is updated to obtain a new PQ codebook.

5. The streaming time-series subsequence anomaly detection method as described in claim 4, characterized in that, No. The first segment of the space The update method for each segment center is as follows: In the formula, Indicates the updated number The first segment of the space Each segment center, Indicates the number before the update The first segment of the space Each segment center, This indicates that all quantizations, including the latest batch and historical batches, are included. The number of segments, This indicates the latest batch of subsequences. The Each segment Quantified as ; express Quantitative loss, This indicates all quantized values ​​in the latest batch. The sum of the quantization losses corresponding to the subsequences.

6. The streaming time-series subsequence anomaly detection method as described in claim 4, characterized in that, When S2 is executed repeatedly, for each subsequence The search method for nearest neighbors has been changed to: For each new cluster, construct an inverted index and quantize each segment in the latest batch of subsequences by the nearest segment center in the segment space to which the segment belongs. This segment center comes from the PQ codebook obtained from the subsequences of the previous batch before the update, thus obtaining the PQ code for each subsequence of the latest batch. Store the PQ code in the inverted index corresponding to each new cluster to obtain the IVFPQ index corresponding to all new clusters, and use it as a secondary index. Use the IVFPQ index corresponding to all historical clusters as the primary index. Calculate the subsequence of each nearest neighbor to be found in the latest batch. Distance to each new cluster center; obtain the nearest distance based on the secondary index. All subsequences in the new cluster ;Calculate the subsequence in each segment space Construct a distance table by calculating the Euclidean distances between the centers of all segments in the PQ codebook corresponding to each segment and the historical batch; and then calculate the subsequences based on this distance table. Each with all the subsequences The Euclidean distance between them is expressed as: In the formula, , Representing a subsequence The The quantization results of each segment, Representing a subsequence The Each segment; will have the preceding The subsequence with minimum distance As preliminary search results for approximate nearest neighbors, all are recorded as the first candidate nearest neighbor, and the [number]th [number] ... The Euclidean distance of the nearest neighbor is taken as the BSF; Calculate subsequence Euclidean distance between each historical cluster center; obtain the nearest cluster based on the primary index. All subsequences in a historical cluster; According to the formula traversal All subsequences in a historical cluster are used as the formula. Based on the stored calculation results and Direct calculation , express The cluster center of the current cluster; select Less than the corresponding BSF All were selected as second candidate nearest neighbors; and subsequences were calculated. Euclidean distance to each second candidate nearest neighbor; Distance subsequences among all first-candidate nearest neighbors and second-candidate nearest neighbors recent former Each subsequence is used as a subsequence The nearest neighbors complete the subsequence. of Searching for the nearest neighbors.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein the computer program, when executed by a processor, controls the device on which the storage medium is located to perform the steps of the method as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 6.

Images