A data anomaly detection method and system based on time series vector retrieval
Through the method based on time series vector retrieval, abnormal detection of time series data is performed, which solves the false alarm problems caused by weekend effects and holiday effects, and achieves fast and accurate abnormal detection, which is highly adaptable and is suitable for characteristic fluctuations of time series data.
Patent Information
- Application Number
- CN202111551183.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-17
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2041-12-17
AI Technical Summary
When existing anomaly detection methods process timing data, especially weekend effects and holiday effects, they are prone to false positives, and it is difficult to effectively adapt to the characteristic fluctuations of these data.
Using the data exception detection method based on time series vector search, by obtaining the index data in the preset time period for processing and vectorization, a vector index file is constructed, and a vector search engine is used to search the detection index data to determine whether the data is abnormal. data.
It realizes fast and accurate abnormal detection of time series data, can effectively adapt to weekend effects or holiday effects, reduce false alarms, and improve the accuracy and practicality of detection.
Smart Images
Figure CN114218164B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of anomaly detection, and in particular to a data anomaly detection method and system based on time series vector retrieval. Background Art
[0002] Anomaly detection is an important technology. It can find outliers that are different from the "main data distribution", such as finding fraud cases from credit card transactions, finding intrusions from normal network data flows, and finding patients with rare diseases in normal cases. It has a wide range of commercial application value. For Internet companies, there are a large number of systems that continuously generate data, especially time series data. Anomaly detection of time series data is a very valuable task. The commonly used anomaly detection methods are based on statistics, neural networks, and clustering.
[0003] The above traditional anomaly detection methods have good performance for time series data with regular fluctuations. However, for time series data with weekend and holiday effects, false positives are often generated on weekends or holidays. The weekend and holiday effects refer to the fact that the performance of some time series data on weekends or holidays is different from that on weekdays. Summary of the invention
[0004] The embodiments of the present invention provide a data anomaly detection method and system based on time series vector retrieval. Based on the vector retrieval file of time series data, anomaly detection can be quickly performed on time series data, and the system can well adapt to the weekend effect or holiday effect of time series data.
[0005] To achieve the above-mentioned purpose, on the one hand, an embodiment of the present invention provides a data anomaly detection method based on time series vector retrieval, comprising:
[0006] Obtaining indicator data of a first preset number of time points based on time series within a preset time period, processing the indicator data of the first preset number of time points based on time series, constructing a vector index file in a vector engine through the processed indicator data, and forming a vector search engine based on time series;
[0007] Obtain the index data of the time point to be detected, perform quantization processing and quantization on the index data of the time point to be detected, and obtain the vector data of the index to be detected;
[0008] Put the vector data of the indicator to be detected into a time-series-based vector retrieval engine, and retrieve the similarity between the vector data of the indicator to be detected and the vector index file through the time-series-based vector retrieval engine;
[0009] It is determined whether the indicator data corresponding to the indicator vector data to be detected is abnormal data based on the similarity retrieval between the indicator vector data to be detected and the vector index file.
[0010] On the other hand, an embodiment of the present invention provides a data anomaly detection method system based on time series vector retrieval, including:
[0011] A vector search engine construction unit, used to obtain index data of a first preset number of time points based on time series within a preset time period, process the index data of the first preset number of time points based on time series, and construct a vector index file in the vector engine through the processed index data to form a vector search engine based on time series;
[0012] A data processing unit to be detected is used to obtain the index data at the time point to be detected, perform quantization processing and quantization on the index data at the time point to be detected, and obtain the index vector data to be detected;
[0013] The data retrieval unit to be detected is used to put the indicator vector data to be detected into the time-series-based vector retrieval engine, and retrieve the similarity between the indicator vector data to be detected and the vector index file through the time-series-based vector retrieval engine;
[0014] The determination unit is used to determine whether the indicator data corresponding to the indicator vector data to be detected is abnormal data based on the similarity between the indicator vector data to be detected and the vector index file.
[0015] The above technical solution has the following beneficial effects: based on the vector retrieval file of time series data, anomaly detection can be quickly performed on the time series data, and the weekend effect or holiday effect of the time series data can be well adapted. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1 is a flow chart of a data anomaly detection method based on time series vector retrieval according to an embodiment of the present invention;
[0018] Figure 2 is a structural diagram of a data anomaly detection system based on time series vector retrieval according to an embodiment of the present invention;
[0019] Figure 3 is a structural diagram of another data anomaly detection system based on time series vector retrieval according to an embodiment of the present invention;
[0020] Figure 4 The embodiment of the present invention Figure 3The process of implementing the data anomaly detection system. DETAILED DESCRIPTION
[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0022] like Figure 1 As shown, in combination with an embodiment of the present invention, a data anomaly detection method based on time series vector retrieval is provided, comprising:
[0023] S101: Obtaining index data of a first preset number of time points based on time series within a preset time period, processing the index data of the first preset number of time points based on time series, constructing a vector index file in a vector engine through the processed index data, and forming a vector search engine based on time series;
[0024] S102: Acquire the index data of the time point to be detected, perform quantization processing and quantization on the index data of the time point to be detected, and obtain the index vector data to be detected;
[0025] S103: putting the to-be-detected indicator vector data into a time-series-based vector search engine, and searching the similarity between the to-be-detected indicator vector data and the vector index file through the time-series-based vector search engine;
[0026] S104: Determine whether the indicator data corresponding to the indicator vector data to be detected is abnormal data based on the similarity search between the indicator vector data to be detected and the vector index file.
[0027] Preferably, step 101 specifically includes:
[0028] S1011: for the indicator data at each time point, obtain the indicator data of a second preset number of time points from the time point forward, form a numerical column of the indicator data of the second preset number of time points in chronological order, and use the formed numerical column as the initial vector of the indicator at the time point; wherein the dimension of the initial vector of the indicator at the time point is equal to the second preset number, and the second preset number is less than the first preset number;
[0029] S1012: setting the quantization value of the indicator data of the first dimension in the initial vector of the indicator at the time point to 1, and taking the ratio of the indicator data of other dimensions to the indicator data of the first dimension as the corresponding quantization value of the dimension; forming the quantization vector of the indicator at the time point through the quantization values of each dimension of the indicator;
[0030] S1013: taking the quantitative vector of the indicator at each time point as a row and arranging them based on time series to form an indicator vector data set based on time series;
[0031] S1014: Create a vector index file from the time series-based indicator vector data set through the vector engine faiss, and use the vector index file as a search vector of faiss to obtain a time series-based vector search engine.
[0032] Preferably, in step 1011, for the indicator data at each time point, obtaining the indicator data at a second preset number of time points from the time point forward specifically includes:
[0033] If the indicator data acquired from the time point forward does not meet the second preset number, the indicator data at the time point is discarded;
[0034] The data anomaly detection method based on time series vector retrieval also includes:
[0035] S105: using the retrieval method provided by faiss to traverse and search the vectors in the created vector index file;
[0036] S106: When there are no isolated vectors in the created vector index file and there is no abnormality in the business corresponding to the vectors in the vector index file, the created vector index file is determined to be successful; wherein, the isolated vector refers to a vector in the created vector index file whose distance from the existing vectors in the retrieval method provided by faiss is greater than a preset test threshold.
[0037] Preferably, step 102 specifically includes:
[0038] S1021: Acquire indicator data at a time point to be detected, and acquire a third preset number of time-series-based indicator data from the time point forward to form an initial vector of the indicator at the time point to be detected;
[0039] S1022: The quantization value of the indicator data of the first dimension in the initial vector of the indicator at the time point to be detected is set to 1, and the ratio of the indicator data of other dimensions to the indicator data of the first dimension is used as the quantization value corresponding to the dimension; the quantization vector of the indicator at the time point to be detected is formed by the quantization values of each dimension of the indicator, as the indicator vector data to be detected.
[0040] Preferably, step 103 specifically includes:
[0041] Retrieving the distance between the quantized vector of the indicator at the time point to be inspected and each vector in the vector index file in the time-series-based vector retrieval engine through the vector retrieval engine, and taking the distance between the quantized vector of the indicator at the time point to be inspected and each vector in the vector index file in the time-series-based vector retrieval engine as the similarity between the vector data of the indicator to be inspected and the vector index file;
[0042] Step 104 specifically includes:
[0043] S1041: when the distance between the quantized vector of the indicator at the time point to be detected and the nearest vector in the vector index file is not less than a preset distance threshold, determining that the indicator data corresponding to the quantized vector of the indicator at the time point to be detected is abnormal data; and issuing an alarm for the abnormal data;
[0044] S1042: When the distance between the quantized vector of the indicator at the time point to be detected and the nearest vector in the vector index file is less than a preset distance threshold, determine that the indicator data corresponding to the quantized vector of the indicator at the time point to be detected is normal data; and add the quantized vector of the indicator at the time point to be detected to the vector index file;
[0045] After the alarm is issued for abnormal data, the method further includes:
[0046] S107: confirm whether the business corresponding to the abnormal data has an abnormality. When the corresponding business has not an abnormality, the abnormal data is determined to be a false alarm, and the quantized vector of the false alarm time point indicator to be detected is added to the vector index file to update the vector index file.
[0047] like Figure 2 As shown, in combination with an embodiment of the present invention, a data anomaly detection system based on time series vector retrieval is provided, comprising:
[0048] The vector search engine construction unit 21 is used to obtain the index data of the first preset number of time points based on the time series within a preset time period, process the index data of the first preset number of time points based on the time series, and construct a vector index file in the vector engine through the processed index data to form a vector search engine based on the time series;
[0049] The to-be-detected data processing unit 22 is used to obtain the index data of the to-be-detected time point, perform quantization processing and quantization on the index data of the to-be-detected time point, and obtain the to-be-detected index vector data;
[0050] The to-be-detected data retrieval unit 23 is used to put the to-be-detected index vector data into a time-series-based vector retrieval engine, and retrieve the similarity between the to-be-detected index vector data and the vector index file through the time-series-based vector retrieval engine;
[0051] The determination unit 24 is used to determine whether the indicator data corresponding to the indicator vector data to be detected is abnormal data based on the similarity between the indicator vector data to be detected and the vector index file.
[0052] Preferably, the vector search engine building unit 21 includes:
[0053] The initial vector generating subunit 211 is used to obtain the indicator data of a second preset number of time points from the time point forward for the indicator data at each time point, form a numerical column of the indicator data of the second preset number of time points in chronological order, and use the formed numerical column as the initial vector of the indicator at the time point; wherein the dimension of the initial vector of the indicator at the time point is equal to the second preset number, and the second preset number is less than the first preset number;
[0054] The initial vector quantization subunit 212 is used to set the quantization value of the indicator data of the first dimension in the initial vector of the indicator at the time point to 1, and use the ratio of the indicator data of other dimensions to the indicator data of the first dimension as the corresponding quantization value of the dimension; and form the quantization vector of the indicator at the time point through the quantization values of each dimension of the indicator;
[0055] The data set generation subunit 213 is used to treat the quantitative vector of the indicator at each time point as a row and arrange them based on the time series to form an indicator vector data set based on the time series;
[0056] The vector search engine subunit 214 is used to create a vector index file from the time series-based indicator vector data set through the vector engine faiss, and use the vector index file as a search vector of faiss to obtain a time series-based vector search engine.
[0057] Preferably, the initial vector generating subunit 211 is specifically used for:
[0058] If the indicator data acquired from the time point forward does not meet the second preset number, the indicator data at the time point is discarded;
[0059] The data anomaly detection system based on time series vector retrieval further includes a verification unit 25, wherein the verification unit 25 is used to:
[0060] Use the retrieval method provided by faiss to traverse and search the vectors in the created vector index file;
[0061] When there are no isolated vectors in the created vector index file and there is no abnormality in the business corresponding to the vectors in the vector index file, the created vector index file is determined to be successful; wherein, the isolated vector refers to the vector in the created vector index file whose distance from the existing vector in the retrieval method provided by faiss is greater than the preset test threshold.
[0062] Preferably, the to-be-detected data processing unit 22 is specifically used for:
[0063] Acquire the indicator data of the time point to be detected, and acquire a third preset number of time-series-based indicator data from the time point forward to form an initial vector of the indicator of the time point to be detected;
[0064] The quantization value of the indicator data of the first dimension in the initial vector of the indicator at the time point to be detected is set to 1, and the ratio of the indicator data of other dimensions to the indicator data of the first dimension is used as the quantization value corresponding to the dimension; the quantization vector of the indicator at the time point to be detected is formed by the quantization values of each dimension of the indicator, as the indicator vector data to be detected.
[0065] Preferably, the to-be-detected data retrieval unit 23 is specifically used for:
[0066] Retrieving the distance between the quantized vector of the indicator at the time point to be inspected and each vector in the vector index file in the time-series-based vector retrieval engine through the vector retrieval engine, and taking the distance between the quantized vector of the indicator at the time point to be inspected and each vector in the vector index file in the time-series-based vector retrieval engine as the similarity between the vector data of the indicator to be inspected and the vector index file;
[0067] The determination unit 24 is specifically configured to:
[0068] When the distance between the quantized vector of the indicator at the time point to be detected and the nearest vector in the vector index file is not less than a preset distance threshold, the indicator data corresponding to the quantized vector of the indicator at the time point to be detected is determined to be abnormal data; an alarm is issued for the abnormal data;
[0069] When the distance between the quantized vector of the indicator at the time point to be detected and the nearest vector in the vector index file is less than a preset distance threshold, it is determined that the indicator data corresponding to the quantized vector of the indicator at the time point to be detected is normal data; and the quantized vector of the indicator at the time point to be detected is added to the vector index file;
[0070] The data anomaly detection system based on time series vector retrieval further includes:
[0071] An alarm unit 26, configured to issue an alarm for abnormal data after determining that the data corresponding to the quantized vector of the indicator at the time point to be detected is abnormal data;
[0072] The vector search engine updating unit 27, after determining that the data corresponding to the quantized vector of the indicator at the time point to be detected is normal data, adds the quantized vector of the indicator at the time point to be detected to the vector index file;
[0073] The abnormal data checking unit 28 is used to confirm whether the business corresponding to the abnormal data has an abnormality after the alarm is issued for the abnormal data. When the corresponding business has not an abnormality, the abnormal data is determined to be a false alarm, and the quantized vector of the false alarm time point indicator to be detected is added to the vector index file to update the vector index file.
[0074] The beneficial effects achieved by the present invention are as follows:
[0075] Through innovative vectorization of time series data and vector-based retrieval, we can quickly detect anomalies in time series data, achieve fast and accurate retrieval of abnormal data, and adapt well to the weekend effect or holiday effect of time series data, with good fault tolerance; in actual use, it is more practical.
[0076] The above technical solution of the embodiment of the present invention is described in detail below in conjunction with specific application examples. For technical details not introduced during the implementation process, please refer to the relevant description in the previous text.
[0077] The abbreviations and key terms used in the present invention are defined as follows:
[0078] Anomaly detection: Anomaly detection is an important data mining or machine learning method that can find outliers that are different from the "main data distribution", such as finding fraud cases from credit card transactions, finding intrusions from normal network data streams, and finding patients with rare diseases in normal cases. It has very broad commercial application value.
[0079] Anomaly Detection System: A system that automatically detects anomalies.
[0080] Vector retrieval: Finding a vector that is similar to the target vector in a large amount of vector data is called vector retrieval. Vector retrieval is widely used in computer vision, such as face recognition, object recognition, etc.
[0081] Time series data: Data that changes over time and can be reflected by numerical values is called time series data.
[0082] The present invention is a time series data anomaly detection system based on vector retrieval, which is an original anomaly detection system. It vectorizes time series data and then uses vector retrieval to achieve high-speed and accurate anomaly detection. It belongs to the technical fields of big data, anomaly detection, time series data analysis, etc.
[0083] The patent of this invention converts time series data into vector data. Then based on the vector retrieval method, it can effectively adapt to the weekend effect or holiday effect and avoid false alarms. It has good performance in practical use and is a very practical anomaly detection system.
[0084] Vector search essentially transforms various fluctuations into a vector fingerprint. Similar to music search, after normalizing different pitches, weekend or holiday patterns can be retrieved, thus adapting to weekend or holiday effects.
[0085] The technical solution of the present invention is described in detail as follows:
[0086] 1. The vectorization scheme for time series data is described as follows:
[0087] Obtain the index data of the first preset number of time points based on the time series within the preset time period, process the index data of the first preset number of time points based on the time series, and construct a vector index file in the vector engine through the processed index data to form a vector retrieval engine based on the time series. The time series data consists of two columns, the first column is time, and the second column is value, which is usually hourly time series data, as shown below: 2021101101 1296 2021101102 2011 2021101103 3027
[0091] …
[0092] (1) Generate an initial vector based on time series data
[0093] For the indicator data at each time point, obtain the indicator data of the second preset number of time points from this time point forward, form a numerical column of the indicator data of the second preset number of time points in chronological order, and use the formed numerical column as the initial vector of the indicator at this time point; wherein the dimension of the initial vector of the indicator at this time point is equal to the second preset number, and the second preset number is less than the first preset number. Specifically:
[0094] The initial vector is based on the data at each time point. The data of 24 time points before this time point is taken, the time column is removed, and the initial vector is formed based on the value column. The dimension of the initial vector is 24.
[0095] If 24 time data cannot be obtained forward, then this time data is discarded and not converted into an initial vector (if the indicator data obtained from this time point forward does not meet the second preset number, the indicator data at this time point is discarded).
[0096] Therefore, for the system to operate normally, it needs to be preheated. The number of time points can be adjusted, and at least 24 time points must be accumulated before it can operate normally.
[0097] (2) Quantize the initial vector
[0098] The process of generating the final vector is the process of quantizing the initial vector. The quantization value of the indicator data of the first dimension in the initial vector of the indicator at that time point is set to 1, and the ratio of the indicator data of other dimensions to the indicator data of the first dimension is used as the corresponding quantization value of the dimension; the quantization vector of the indicator at that time point is formed through the quantization values of each dimension of the indicator. Specifically:
[0099] The first dimension of the vector is quantized to 1; the other dimensions are quantized to the ratio of the first dimension. This process of vector quantization is to avoid the influence of the vector size. After quantization, the new vector generated reflects the trend of the vector change. Normalization can also be used for quantization.
[0100] Let's take a three-dimensional vector as an example. Using the data shown above, assuming that the three-dimensional vector is (1296, 2011, 3027), the final vector generated based on the above quantization process is (1, 1.5517, 2.3356).
[0101] (3) Obtain the final vector dataset
[0102] Based on the above operations, we convert the data of each time point of a set of time series data into a 24-dimensional vector to form a vector data set. Specifically, the data of each time point of a set of time series data is converted into a 24-dimensional vector. The dimension can usually be set flexibly. The 24-dimensional dimension is taken here mainly because a lot of business monitoring is hourly data, and there are exactly 24 hours in a day. It is more appropriate to convert each data point into a 24-dimensional vector. In other words, the quantitative vector of the indicator at each time point is taken as a row and arranged based on the time series to form an indicator vector data set based on the time series.
[0103] (4) Building vector index
[0104] In order to achieve fast and accurate vector retrieval, it is necessary to create a vector index. The present invention uses Facebook's open source vector engine Faiss, and creates a vector data set into an index file based on Faiss to build a new vector retrieval engine, and then quickly retrieves approximate vectors. Specifically, the time-series-based indicator vector data set is created into a vector index file through Facebook's open source vector engine Faiss, and the vector index file is used as the retrieval vector of Faiss to obtain a time-series-based vector retrieval engine.
[0105] In order to improve the accuracy of detection, the data in the vector are all normal data. The retrieval method provided by faiss is used to traverse and search the vectors in the created vector index file; when there are no isolated vectors in the vector index file created, and the corresponding business of the vectors in the vector index file has no abnormality, the created vector index file is determined to be successful; wherein, the isolated vector refers to the vector in the created vector index file, and the distance between the vector and the existing vector in the retrieval method provided by faiss is greater than the preset test threshold. Specifically:
[0106] That is, abnormal data cannot be added to the index. For the system to work correctly, it usually needs to be pre-run for 2 cycles, usually 14 days. And the vectors obtained from the pre-run will be traversed and searched. If there are no isolated vectors (isolated vectors refer to vectors that cannot be clustered with existing vectors at a threshold distance. In other words, no vectors with a distance less than the threshold can be found in the existing vectors), and there are no abnormalities in the business, it is considered that the preheating is successful. If an isolated vector is found, manual intervention is required to see if there is any abnormality in the isolated vector point. This is an innovation. Automatically determine whether the preheating data is normal.)
[0107] 2. This anomaly detection system:
[0108] This anomaly detection system mainly consists of two modules: Figure 3 As shown, this anomaly detection system is mainly composed of a time series data vectorization module and a vector indexing and retrieval module.
[0109] The time series data vectorization module is responsible for vectorizing the data at each time point of the time series data (for each time point, 24 data points are also taken forward).
[0110] The vector indexing and retrieval module is implemented based on Facebook's open source Faiss vector engine and is responsible for indexing and retrieving vector data.
[0111] Obtain the indicator data at the time point to be detected, quantize and vectorize the indicator data at the time point to be detected, and obtain the indicator vector data to be detected; put the indicator vector data to be detected into the time-series-based vector retrieval engine, and retrieve the similarity between the indicator vector data to be detected and the vector index file through the time-series-based vector retrieval engine; according to the similarity between the indicator vector data to be detected and the vector index file, determine whether the indicator data corresponding to the indicator vector data to be detected is abnormal data. The whole system works as follows Figure 4 As shown, the process is described as follows:
[0112] (1) First, the data points of the time series data to be detected are vectorized and converted into vector data. Specifically: the index data of the time point to be detected is obtained, and a third preset number of time series-based index data is obtained from the time point forward to form an initial vector of the index of the time point to be detected;
[0113] The quantization value of the indicator data of the first dimension in the initial vector of the indicator at the time point to be detected is set to 1, and the ratio of the indicator data of other dimensions to the indicator data of the first dimension is used as the quantization value corresponding to the dimension; the quantization vector of the indicator at the time point to be detected is formed by the quantization values of each dimension of the indicator, as the indicator vector data to be detected.
[0114] (2) Put the vector data into the constructed vector search engine for similarity search. Specifically: use the vector search engine to search the distance between the quantized vector of the time point indicator to be tested and each vector in the vector index file in the time-series-based vector search engine, and use the distance between the quantized vector of the time point indicator to be tested and each vector in the vector index file in the time-series-based vector search engine as the similarity between the vector data of the indicator to be tested and the vector index file.
[0115] (3) If the distance of the nearest vector retrieved exceeds the preset threshold, it is considered to be abnormal data, and an alarm needs to be issued. Specifically: when the distance between the quantized vector of the indicator at the time point to be detected and the nearest vector in the vector index file is not less than the preset distance threshold, the indicator data corresponding to the quantized vector of the indicator at the time point to be detected is determined to be abnormal data; an alarm is issued for the abnormal data.
[0116] (4) If the distance between the nearest vector retrieved is less than the preset threshold, it is considered normal data and no alarm is required. At the same time, this vector is also added to the vector search engine. Specifically: when the distance between the quantized vector of the indicator at the time point to be detected and the nearest vector in the vector index file is less than the preset distance threshold, the indicator data corresponding to the quantized vector of the indicator at the time point to be detected is determined to be normal data; and the quantized vector of the indicator at the time point to be detected is added to the vector index file.
[0117] In addition, for this system to work properly, it needs to accumulate at least 24 data points of time series data. Usually, accumulating more data will help the system monitor accurately. In actual operation, it is generally necessary to warm up the system with 50 data points before starting monitoring.
[0118] After normalization, the trend change of the vector becomes simpler and prone to false alarms. That is, this system may also have false alarms, but after a false alarm, you only need to add the false alarm vector to the vector index, and the same situation will not occur in the future. Specifically: confirm whether the business corresponding to the abnormal data has an abnormality. When the corresponding business has not an abnormality, the abnormal data is determined to be a false alarm, and the quantized vector of the false alarm time point indicator to be detected is added to the vector index file to update the vector index file.
[0119] The beneficial effects achieved by the present invention are as follows:
[0120] Through innovative vectorization of time series data and vector-based retrieval, we can quickly detect anomalies in time series data, achieve fast and accurate retrieval of abnormal data, and adapt well to the weekend effect or holiday effect of time series data, with good fault tolerance; in actual use, it is more practical.
[0121] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.
[0122] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.
[0123] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.
[0124] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".
[0125] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.
[0126] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.
[0127] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.
[0128] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.
[0129] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A data anomaly detection method based on time series vector retrieval, characterized in that: include: Obtaining indicator data of a first preset number of time points based on time series within a preset time period, processing the indicator data of the first preset number of time points based on time series, constructing a vector index file in a vector engine through the processed indicator data, and forming a vector search engine based on time series; Obtain the index data of the time point to be detected, perform quantization processing and quantization on the index data of the time point to be detected, and obtain the vector data of the index to be detected; Put the vector data of the indicator to be detected into a time-series-based vector retrieval engine, and retrieve the similarity between the vector data of the indicator to be detected and the vector index file through the time-series-based vector retrieval engine; Determine whether the indicator data corresponding to the indicator vector data to be detected is abnormal data according to the similarity search between the indicator vector data to be detected and the vector index file; The data anomaly detection method based on time series vector retrieval also includes: Use the retrieval method provided by faiss to traverse and search the vectors in the created vector index file; When there are no isolated vectors in the created vector index file and there is no abnormality in the business corresponding to the vectors in the vector index file, the created vector index file is determined to be successful; wherein, the isolated vector refers to the vector in the created vector index file whose distance from the existing vector in the retrieval method provided by faiss is greater than the preset test threshold.
2. The data anomaly detection method based on time series vector retrieval according to claim 1 is characterized in that: The step of processing the indicator data of a first preset number of time points based on the time series, constructing a vector index file in the vector engine through the processed indicator data, and forming a vector search engine based on the time series specifically includes: For the indicator data at each time point, obtain the indicator data of a second preset number of time points from the time point forward, form a numerical column of the indicator data of the second preset number of time points in chronological order, and use the formed numerical column as the initial vector of the indicator at the time point; wherein the dimension of the initial vector of the indicator at the time point is equal to the second preset number, and the second preset number is less than the first preset number; The quantized value of the indicator data of the first dimension in the initial vector of the indicator at the time point is set to 1, and the ratio of the indicator data of other dimensions to the indicator data of the first dimension is used as the corresponding quantized value of the dimension; the quantized vector of the indicator at the time point is formed by the quantized values of each dimension of the indicator; The quantitative vector of the indicator at each time point is taken as a row and arranged based on time series to form a time series-based indicator vector data set; The time-series-based indicator vector data set is created into a vector index file through the vector engine faiss, and the vector index file is used as the retrieval vector of faiss to obtain a time-series-based vector retrieval engine.
3. The data anomaly detection method based on time series vector retrieval according to claim 2 is characterized in that: The step of obtaining the indicator data of a second preset number of time points from the time point forward for the indicator data at each time point specifically includes: If the indicator data acquired from the time point forward does not meet the second preset quantity, the indicator data at the time point is discarded.
4. The data anomaly detection method based on time series vector retrieval according to claim 1 is characterized in that: The step of obtaining the index data at the time point to be detected, performing quantization processing and quantization on the index data at the time point to be detected, and obtaining the index vector data to be detected specifically includes: Acquire the indicator data of the time point to be detected, and acquire a third preset number of time-series-based indicator data from the time point forward to form an initial vector of the indicator of the time point to be detected; The quantization value of the indicator data of the first dimension in the initial vector of the indicator at the time point to be detected is set to 1, and the ratio of the indicator data of other dimensions to the indicator data of the first dimension is used as the quantization value corresponding to the dimension; the quantization vector of the indicator at the time point to be detected is formed by the quantization values of each dimension of the indicator, as the indicator vector data to be detected.
5. The data anomaly detection method based on time series vector retrieval according to claim 4 is characterized in that: The step of placing the to-be-detected indicator vector data into a time-series-based vector search engine and searching for similarities between the to-be-detected indicator vector data and the vector index file through the time-series-based vector search engine specifically includes: Retrieving the distance between the quantized vector of the indicator at the time point to be inspected and each vector in the vector index file in the time-series-based vector retrieval engine through the vector retrieval engine, and taking the distance between the quantized vector of the indicator at the time point to be inspected and each vector in the vector index file in the time-series-based vector retrieval engine as the similarity between the vector data of the indicator to be inspected and the vector index file; The step of searching and determining whether the indicator data corresponding to the indicator vector data to be detected is abnormal data based on the similarity between the indicator vector data to be detected and the vector index file specifically includes: When the distance between the quantized vector of the indicator at the time point to be detected and the nearest vector in the vector index file is not less than a preset distance threshold, the indicator data corresponding to the quantized vector of the indicator at the time point to be detected is determined to be abnormal data; an alarm is issued for the abnormal data; When the distance between the quantized vector of the indicator at the time point to be detected and the nearest vector in the vector index file is less than a preset distance threshold, it is determined that the indicator data corresponding to the quantized vector of the indicator at the time point to be detected is normal data; and the quantized vector of the indicator at the time point to be detected is added to the vector index file; After the alarm is issued for abnormal data, the method further includes: Confirm whether the business corresponding to the abnormal data has an abnormality. When the corresponding business has not an abnormality, the abnormal data is determined to be a false alarm, and the quantized vector of the false alarm time point indicator to be detected is added to the vector index file to update the vector index file.
6. A data anomaly detection system based on time series vector retrieval, characterized in that: include: A vector search engine construction unit, used to obtain index data of a first preset number of time points based on time series within a preset time period, process the index data of the first preset number of time points based on time series, and construct a vector index file in the vector engine through the processed index data to form a vector search engine based on time series; A data processing unit to be detected is used to obtain the index data at the time point to be detected, perform quantization processing and quantization on the index data at the time point to be detected, and obtain the index vector data to be detected; The data retrieval unit to be detected is used to put the indicator vector data to be detected into the time-series-based vector retrieval engine, and retrieve the similarity between the indicator vector data to be detected and the vector index file through the time-series-based vector retrieval engine; A determination unit, used for determining whether the indicator data corresponding to the indicator vector data to be detected is abnormal data according to the similarity between the indicator vector data to be detected and the vector index file; The verification unit is used to traverse and search the vectors in the created vector index file by using the retrieval method provided by faiss; when there are no isolated vectors in the vector index file created and there is no abnormality in the business corresponding to the vectors in the vector index file, the created vector index file is determined to be successful; wherein, the isolated vector refers to a vector in the created vector index file, the distance between which and the existing vectors in the retrieval method provided by faiss is greater than a preset verification threshold.
7. The data anomaly detection system based on time series vector retrieval according to claim 6 is characterized in that: The vector search engine construction unit includes: an initial vector generating subunit, for obtaining, for each indicator data at a time point, indicator data at a second preset number of time points from the time point forward, forming a numerical sequence of the indicator data at the second preset number of time points in chronological order, and using the formed numerical sequence as an initial vector of the indicator at the time point; wherein the dimension of the initial vector of the indicator at the time point is equal to the second preset number, and the second preset number is less than the first preset number; The initial vector quantization subunit is used to set the quantization value of the indicator data of the first dimension in the initial vector of the indicator at the time point to 1, and use the ratio of the indicator data of other dimensions to the indicator data of the first dimension as the corresponding quantization value of the dimension; and form the quantization vector of the indicator at the time point through the quantization values of each dimension of the indicator; A data set generation subunit, used for treating the quantitative vector of the indicator at each time point as a row and arranging them based on time series to form an indicator vector data set based on time series; The vector search engine subunit is used to create a vector index file from a time-series-based indicator vector data set through the vector engine faiss, and use the vector index file as a search vector of faiss to obtain a time-series-based vector search engine.
8. The data anomaly detection system based on time series vector retrieval according to claim 7 is characterized in that: The initial vector generating subunit is specifically used for: If the indicator data acquired from the time point forward does not meet the second preset quantity, the indicator data at the time point is discarded.
9. The data anomaly detection system based on time series vector retrieval according to claim 6, characterized in that: The to-be-detected data processing unit is specifically used for: Acquire the indicator data of the time point to be detected, and acquire a third preset number of time-series-based indicator data from the time point forward to form an initial vector of the indicator of the time point to be detected; The quantization value of the indicator data of the first dimension in the initial vector of the indicator at the time point to be detected is set to 1, and the ratio of the indicator data of other dimensions to the indicator data of the first dimension is used as the quantization value corresponding to the dimension; the quantization vector of the indicator at the time point to be detected is formed by the quantization values of each dimension of the indicator, as the indicator vector data to be detected.
10. The data anomaly detection system based on time series vector retrieval according to claim 9, characterized in that: The to-be-detected data retrieval unit is specifically used for: Retrieving the distance between the quantized vector of the indicator at the time point to be inspected and each vector in the vector index file in the time-series-based vector retrieval engine through the vector retrieval engine, and taking the distance between the quantized vector of the indicator at the time point to be inspected and each vector in the vector index file in the time-series-based vector retrieval engine as the similarity between the vector data of the indicator to be inspected and the vector index file; The determination unit is specifically used for: When the distance between the quantized vector of the indicator at the time point to be detected and the nearest vector in the vector index file is not less than a preset distance threshold, the indicator data corresponding to the quantized vector of the indicator at the time point to be detected is determined to be abnormal data; an alarm is issued for the abnormal data; When the distance between the quantized vector of the indicator at the time point to be detected and the nearest vector in the vector index file is less than a preset distance threshold, it is determined that the indicator data corresponding to the quantized vector of the indicator at the time point to be detected is normal data; and adding the quantized vector of the indicator at the time point to be detected to the vector index file; The data anomaly detection system based on time series vector retrieval also includes: An alarm unit, configured to issue an alarm for abnormal data after determining that the data corresponding to the quantized vector of the indicator at the time point to be detected is abnormal data; The vector search engine updating unit adds the quantized vector of the indicator at the time point to be detected to the vector index file after determining that the data corresponding to the quantized vector of the indicator at the time point to be detected is normal data; The abnormal data checking unit is used to confirm whether the business corresponding to the abnormal data has an abnormality after the alarm is issued for the abnormal data. When the corresponding business has not an abnormality, the abnormal data is determined to be a false alarm, and the quantized vector of the false alarm time point indicator to be detected is added to the vector index file to update the vector index file.