Abnormality detection method and device

Abstract

The invention discloses an anomaly detection method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of determining a target object, and obtaining an original data stream related to a to-be-detected attribute from data related to the target object; for each moment, processing the data stream through a preprocessing strategy to obtain a window data stream, and coding the window data stream through a preset self-isolation strategy to obtain an outlier degree representation sequence; performing memory space retrieval processing based on the outlier degree representation sequence to obtain data in a memory block, comparing the data in the memory block with the outlier degree representation sequence, and determining a minimum error score; and acquiring an abnormal water level line at the previous moment, determining an abnormal score, and determining that the data stream at each moment is abnormal in response to the abnormal score exceeding a preset threshold. According to the embodiment, the abnormal condition in the data stream can be accurately identified, and the accuracy and reliability of abnormal detection are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an anomaly detection method and device. Background Art

[0002] In the e-commerce Internet era, various shopping marketing methods and promotion methods emerge in an endless stream. It is difficult for platforms or merchants to ensure that there are no mistakes when setting up promotional activities, which leads to some products being sold at super low prices. In particular, popular products, once targeted by the black market, may be snatched up in just a few minutes. If it is not discovered in time, the loss will be huge. For some small businesses, the result may be direct bankruptcy. Such price gate incidents have also occurred on major e-commerce platforms, so it is particularly important to be able to detect abnormal price fluctuations in a timely manner. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides an anomaly detection method and device, which can at least solve the problem of untimely and low efficiency anomaly detection in the prior art.

[0004] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, there is provided an anomaly detection method, comprising: determining a target object, obtaining an original data stream related to a property to be detected from data related to the target object; wherein the original data stream includes data streams at multiple moments; for each moment, processing the data stream by a preprocessing strategy to obtain a window data stream, encoding the window data stream by a preset self-isolation strategy to obtain an outlier degree representation sequence; performing memory space retrieval processing based on the outlier degree representation sequence to obtain data in a memory block, comparing the data in the memory block with the outlier degree representation sequence to determine a minimum error score; obtaining an abnormal water level line at a previous moment, determining an abnormal score based on the minimum error score and the abnormal water level line, and determining that the data stream at each moment is abnormal in response to the abnormal score exceeding a preset threshold.

[0005] Optionally, the data stream is processed by a preprocessing strategy to obtain a window data stream, including: determining a preset dimension, calculating the mean and standard deviation of the data in the same dimension in the data stream; using the mean and standard deviation to standardize each data in each dimension to obtain a first data stream; using a preset convolution kernel and a preset step size to perform mean pooling on the first data stream to obtain a second data stream; and using a preconfigured sequence window size to process the second data stream to obtain a window data stream.

[0006] Optionally, the calculating of the mean and standard deviation of the data in the same dimension in the data stream includes: for the initial moment, respectively calculating the mean and standard deviation of all the data in each dimension; for each moment except the initial moment, obtaining the amount of data in the data stream at each moment, and obtaining the mean and standard deviation of each dimension in the data stream at the previous moment, so as to calculate the mean and standard deviation of each dimension in the data stream at each moment.

[0007] Optionally, the encoding process of the window data stream is performed through a preset self-isolation strategy to obtain an outlier degree representation sequence, including: for any position point data in the window data stream, determining the distance to other position point data in the window data stream through a preset sequence element distance determination strategy; sorting the distances based on the position order of each position point data in the window data stream to obtain the distance vector of any position point data; determining the relative position distance encoding of any position point data based on the position of any position point data in the window data stream and a pre-configured sequence window size; determining the paradigm distance metric and of any position point data based on the distance vector and relative position distance encoding of any position point data, and obtaining the outlier degree representation of any position point data by scaling transformation; combining the outlier degree representations of all position point data according to the position order of each position point data in the window data stream to obtain an outlier degree representation sequence.

[0008] Optionally, the scaling transformation obtains the outlier degree representation of any one of the location point data, including the following methods: accumulating the sum of the paradigm distance metrics of all the location point data, and determining the outlier degree representation of any one of the location point data based on the outlier degree representation of any one of the location point data and the paradigm distance metrics of any one of the location point data; accumulating the sum of the paradigm distance metrics of all the location point data, and determining the outlier degree representation of any one of the location point data based on the outlier degree representation of any one of the location point data, the paradigm distance metrics of any one of the location point data, and a pre-configured sequence window size and number of dimensions.

[0009] Optionally, the outlier degree representation sequence is a one-dimensional vector, and the sum of all outlier degree representations in the outlier degree representation sequence is 1.

[0010] Optionally, the memory space retrieval processing is performed based on the outlier degree characterization sequence to obtain data in the memory block, including: encoding the outlier degree characterization sequence to obtain a sequence pattern index, and constructing an index tree based on the sequence pattern index; performing nearest neighbor retrieval on the sequence pattern index using the index tree to obtain a first sequence pattern index, and determining a first memory block based on the first sequence pattern index; and stacking the outlier degree characterization sequence in the first memory block to obtain data in the memory block.

[0011] Optionally, the encoding process of the outlier degree representation sequence to obtain a sequence pattern index includes: sorting the outlier degree representations in the outlier degree representation sequence in order from large to small; determining a preset variable value, extracting outlier degree representations of the preset variable value in the sorting order, and determining a position index of each extracted outlier degree representation in the outlier degree representation sequence; mapping based on the extracted outlier degree representation and the position index to obtain multiple elements, and then generating a sequence pattern index.

[0012] Optionally, the method further includes: forming a string based on the numerical concatenation of each element in the sequence pattern index, generating a second memory block, using the string as an index of the second memory block, and storing the outlier degree characterization sequence in the second memory block.

[0013] Optionally, the method of constructing an index tree based on a sequence pattern index includes: obtaining a historical index tree at a previous moment, determining a root node in the historical index tree, and obtaining a node index key set of the root node; wherein the node index key set includes an index key set of child nodes in a next layer; for the first element in the sequence pattern index, in response to the node index key set not including the first element, creating a new child node under the root node based on the first element; looping through each element in the sequence pattern index, determining the position of each element in the sequence pattern index, and determining a node at a level where the position is minus 1; obtaining the node index key set of the node, and in response to the node index key set not including each element, creating a new child node under the node based on each element to obtain a new index tree.

[0014] Optionally, the method of using the index tree to perform nearest neighbor retrieval on the sequence pattern index to obtain a first sequence pattern index includes: initializing an empty first sequence pattern index; wherein the structure of the first sequence pattern index is the same as the structure of the sequence pattern index; looping through the sequence pattern index, and for each element, determining the target node corresponding to each element in the index tree; and determining the position of each element in the sequence pattern index; determining other nodes with the shortest distance from the target node from the index tree level where the target node is located, and writing the elements corresponding to the other nodes into the said position in the first sequence pattern index.

[0015] Optionally, the method also includes: obtaining a maximum capacity size pre-configured for the memory block, and determining the actual size of the data in the memory block; in response to the actual size being smaller than the maximum capacity size, adding an outlier degree characterization sequence to the data in the memory block, and setting a position index to obtain a new data in the memory block; in response to the actual size being equal to the maximum capacity size, determining the position index corresponding to the minimum error score, and replacing the original outlier degree characterization sequence of the position index with the outlier degree characterization sequence.

[0016] Optionally, the process of determining the abnormal water level line at any moment includes: determining the moment before any moment, obtaining the average water level line of the reconstruction error at the previous moment, and the fluctuation size of the reconstruction error water level line; based on the average water level line, the fluctuation size and the minimum error score at any moment, determining the average water level line of the reconstruction error at any moment, and the fluctuation size of the reconstruction error water level line; based on the average water level line of the reconstruction error at any moment, the fluctuation size of the reconstruction error water level line and a preset fluctuation coefficient, determining the abnormal water level line at any moment.

[0017] Optionally, determining the anomaly score based on the minimum error score and the anomaly waterline includes: determining a quotient of the minimum error score and the anomaly waterline, determining a product of the quotient and a preset mapping coefficient, and processing the product using an activation function to obtain the anomaly score.

[0018] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, there is provided an anomaly detection device, comprising: an acquisition module, used to determine a target object, and to acquire an original data stream related to a property to be detected from data related to the target object; wherein the original data stream includes data streams at multiple moments; a processing module, used to process the data stream at each moment through a preprocessing strategy to obtain a window data stream, and to encode the window data stream through a preset self-isolation strategy to obtain an outlier degree representation sequence; a retrieval module, used to perform memory space retrieval processing based on the outlier degree representation sequence to obtain data in a memory block, and to compare the data in the memory block with the outlier degree representation sequence to determine a minimum error score; an anomaly module, used to obtain an anomaly waterline at a previous moment, determine an anomaly score based on the minimum error score and the anomaly waterline, and determine that an anomaly occurs in the data stream at each moment in response to the anomaly score exceeding a preset threshold.

[0019] Optionally, the processing module is used to: determine a preset dimension, calculate the mean and standard deviation of the data in the same dimension in the data stream; use the mean and standard deviation to standardize each data in each dimension to obtain a first data stream; use a preset convolution kernel and a preset step size to perform mean pooling on the first data stream to obtain a second data stream; use a pre-configured sequence window size to process the second data stream to obtain a window data stream.

[0020] Optionally, the processing module is used to: for the initial moment, calculate the mean and standard deviation of all data in each dimension respectively; for each moment other than the initial moment, obtain the data amount of the data stream at each moment, and obtain the mean and standard deviation of each dimension in the data stream at the previous moment, so as to calculate the mean and standard deviation of each dimension in the data stream at each moment.

[0021] Optionally, the processing module is used to: for any position point data in the window data stream, determine the distance to other position point data in the window data stream by using a preset distance determination strategy between sequence elements; sort the distances based on the position order of each position point data in the window data stream to obtain the distance vector of any position point data; determine the relative position distance encoding of any position point data based on the position of any position point data in the window data stream and a pre-configured sequence window size; determine the normal form distance metric and of any position point data based on the distance vector and relative position distance encoding of any position point data, and obtain the outlier degree representation of any position point data by scaling transformation; combine the outlier degree representations of all position point data according to the position order of each position point data in the window data stream to obtain an outlier degree representation sequence.

[0022] Optionally, the scaling transformation obtains the outlier degree representation of any one of the location point data, including the following methods: accumulating the sum of the paradigm distance metrics of all the location point data, and determining the outlier degree representation of any one of the location point data based on the outlier degree representation of any one of the location point data and the paradigm distance metrics of any one of the location point data; accumulating the sum of the paradigm distance metrics of all the location point data, and determining the outlier degree representation of any one of the location point data based on the outlier degree representation of any one of the location point data, the paradigm distance metrics of any one of the location point data, and a pre-configured sequence window size and number of dimensions.

[0023] Optionally, the outlier degree representation sequence is a one-dimensional vector, and the sum of all outlier degree representations in the outlier degree representation sequence is 1.

[0024] Optionally, the retrieval module is used to: encode the outlier degree characterization sequence to obtain a sequence pattern index to construct an index tree based on the sequence pattern index; use the index tree to perform nearest neighbor retrieval on the sequence pattern index to obtain a first sequence pattern index, and determine a first memory block based on the first sequence pattern index; and stack the outlier degree characterization sequence in the first memory block to obtain data within the memory block.

[0025] Optionally, the retrieval module is used to: sort the outlier degree representations in the outlier degree representation sequence in order from large to small; determine the preset variable value, extract the outlier degree representations of the preset variable value in the sorting order, and determine the position index of each extracted outlier degree representation in the outlier degree representation sequence; map based on the extracted outlier degree representation and the position index to obtain multiple elements, and then generate a sequence pattern index.

[0026] Optionally, the device further includes: forming a string based on the numerical concatenation of each element in the sequence pattern index, generating a second memory block, using the string as an index of the second memory block, and storing the outlier degree characterization sequence in the second memory block.

[0027] Optionally, the retrieval module is used to: obtain the historical index tree of the previous moment, determine the root node in the historical index tree, and obtain the node index key set of the root node; wherein the node index key set includes the index key set of the next layer of child nodes; for the first element in the sequence pattern index, in response to the node index key set not including the first element, create a new child node under the root node based on the first element; loop through each element in the sequence pattern index, determine the position of each element in the sequence pattern index, and determine the node at the level where the position is minus 1; obtain the node index key set of the node, and in response to the node index key set not including each element, create a new child node under the node based on each element to obtain a new index tree.

[0028] Optionally, the retrieval module is used to: initialize an empty first sequence pattern index; wherein the structure of the first sequence pattern index is the same as the structure of the sequence pattern index; loop through the sequence pattern index, and for each element, determine the target node corresponding to each element in the index tree; and determine the position of each element in the sequence pattern index; from the index tree level where the target node is located, determine other nodes with the shortest distance from the target node, and write the elements corresponding to the other nodes into the said position in the first sequence pattern index.

[0029] Optionally, the device also includes an update module, which is used to: obtain the maximum capacity size pre-configured for the memory block, and determine the actual size of the data in the memory block; in response to the actual size being smaller than the maximum capacity size, add the outlier degree characterization sequence to the data in the memory block, and set the position index to obtain a new data in the memory block; in response to the actual size being equal to the maximum capacity size, determine the position index corresponding to the minimum error score, and replace the original outlier degree characterization sequence of the position index with the outlier degree characterization sequence.

[0030] Optionally, the process of determining the abnormal water level line at any moment includes: determining the moment before any moment, obtaining the average water level line of the reconstruction error at the previous moment, and the fluctuation size of the reconstruction error water level line; based on the average water level line, the fluctuation size and the minimum error score at any moment, determining the average water level line of the reconstruction error at any moment, and the fluctuation size of the reconstruction error water level line; based on the average water level line of the reconstruction error at any moment, the fluctuation size of the reconstruction error water level line and a preset fluctuation coefficient, determining the abnormal water level line at any moment.

[0031] Optionally, the anomaly module is used to: determine a quotient of a minimum error score and an anomaly waterline, determine a product of the quotient and a preset mapping coefficient, and process the product using an activation function to obtain an anomaly score.

[0032] To achieve the above objective, according to another aspect of an embodiment of the present invention, an abnormality detection electronic device is provided.

[0033] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned anomaly detection methods.

[0034] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above-mentioned abnormality detection methods is implemented.

[0035] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer program product is provided. A computer program product according to an embodiment of the present invention includes a computer program, and when the program is executed by a processor, the anomaly detection method provided by the embodiment of the present invention is implemented.

[0036] According to the scheme provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: through preprocessing and encoding strategies, it is possible to effectively extract abnormal information related to the attribute to be detected from the original data stream; through the generation of the outlier degree characterization sequence and the retrieval and comparison of the memory space, the abnormal situation in the data stream can be accurately identified, thereby improving the accuracy and reliability of anomaly detection. The abnormal score is calculated based on the abnormal water level line and the minimum error score at the previous moment, which can respond to the changes in the data stream in a timely manner, thereby realizing the rapid identification and processing of abnormal situations. This idea based on multidimensional data streams solves the problems of existing non-multidimensional time series streams, short memory cycles, and unstable scoring outputs. It can identify potential anomalies in advance, reduce risks and reduce property losses, and has strong real-time and adaptability.

[0037] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0039] Figure 1 It is a schematic diagram of the existing invalid warning;

[0040] Figure 2A schematic diagram of the main process of an anomaly detection method according to an embodiment of the present invention;

[0041] Figure 3 is a flow chart of an optional anomaly detection method according to an embodiment of the present invention;

[0042] Figure 4 It is a flowchart for generating a sequence representing the degree of outliers;

[0043] Figure 5 is a flowchart of another optional anomaly detection method according to an embodiment of the present invention;

[0044] FIG6( a ) is a schematic diagram of generating a sequence pattern index;

[0045] FIG6( b ) is a schematic diagram of a stacked outlier degree characterization sequence;

[0046] Figure 7 is a flowchart of another optional anomaly detection method according to an embodiment of the present invention;

[0047] Figure 8 It is a schematic diagram of concept drift;

[0048] Fig. 9 is a flowchart of a specific anomaly detection method according to an embodiment of the present invention;

[0049] FIG10( a ) is a schematic diagram of sequence distribution of a normal data stream;

[0050] Figure 10(b) shows the risk event data flow in each dimension calculated for 5 minutes;

[0051] Figure 10(c) Schematic diagram of abnormality scoring;

[0052] FIG10( d ) is a schematic diagram of data before the present solution is used;

[0053] Figure 10(e) is a schematic diagram of the data after using this solution;

[0054] Fig.11 A schematic diagram of main modules of an abnormality detection device according to an embodiment of the present invention;

[0055] Fig.12 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0056] Fig.13 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0058] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0059] Among the existing time series flow anomaly detection technologies, the more typical and commonly used algorithm is RRCF (Robust Random Cut Forest) proposed by Amazon in 2016. RRCF is an unsupervised online learning algorithm that can be used for time series flow anomaly detection and can adaptively learn changes brought about by data distribution. However, it still has the following shortcomings in practical applications:

[0060] 1. Short memory cycle, resulting in invalid warnings. For example, based on minute-level data detection, the sequence window size of the algorithm input is set to 600 (i.e. 10 hours). At this time, the latest data is judged to be abnormal based on the data distribution of the past 10 hours. When a certain product is planned to be sold at 20:00 every night, there will be ultra-low price sales. At this time, it will be abnormal based on the window sequence data of the past 10 hours. However, if the sequence cycle is extended, it is normal, resulting in invalid warnings. See Figure 1 shown.

[0061] 2. Lack of anomaly detection for multi-dimensional time series. Existing technologies are mainly applicable to sequence data of single-dimensional indicators, but real risk anomaly events often need to be comprehensively judged by combining the performance results of multiple dimensions in time series to improve the accuracy of anomaly capture. It is not enough to simply consider a certain dimension (such as commodity price) below a certain threshold as an anomaly with price risk, and the performance of dimensions such as commodity gross profit and sales flow rate are equally important.

[0062] 3. The output results of anomaly scoring are unstable. Due to the random characteristics of the existing algorithms, the output results of anomaly scoring are also unstable and will fluctuate to a certain extent, sometimes with large fluctuations, which will also have a partial impact on anomaly coverage.

[0063] See also Figure 2, which shows a main flow chart of an abnormality detection method provided by an embodiment of the present invention, comprising the following steps:

[0064] S201: Determine a target object, and obtain an original data stream related to the attribute to be detected from data related to the target object; wherein the original data stream includes data streams at multiple moments;

[0065] S202: at each moment, the data stream is processed by a preprocessing strategy to obtain a window data stream, and the window data stream is encoded by a preset self-isolation strategy to obtain an outlier degree representation sequence;

[0066] S203: performing memory space retrieval processing based on the outlier degree representation sequence to obtain data in the memory block, and comparing the data in the memory block with the outlier degree representation sequence to determine a minimum error score;

[0067] S204: Acquire the abnormal water level line at the previous moment, determine the abnormal score based on the minimum error score and the abnormal water level line, and in response to the abnormal score exceeding a preset threshold, determine that the data flow at each moment is abnormal.

[0068] In the above implementation, the risk anomaly detection algorithm provided by this solution is universal and can be applied to anomaly detection needs in various scenarios, including but not limited to abnormal price fluctuations. It can also be applied to other scenarios, such as user behavior anomaly detection, such as monitoring user login, transaction and other behavior patterns, and identifying potential fraud activities. For example, network traffic anomaly detection, such as monitoring network traffic, timely discovering DDoS (Distributed Denial of Service) attacks or other malicious activities. For example, system performance anomaly detection, such as monitoring server performance indicators, such as CPU usage, memory usage, etc., to warn of system failures in advance. For example, medical health data anomaly detection, such as analyzing patients' vital signs data, timely discovering abnormal situations, and assisting doctors in making diagnoses. For example, environmental monitoring, using data collected by sensors to monitor changes in environmental factors such as air quality and water quality, and warn of environmental pollution events.

[0069] This solution mainly uses commodity prices as an example to illustrate, that is, the object to be monitored here is the commodity, the attribute to be detected is the price, and the dimensions can be the average transaction price, total sales volume, and total gross profit. For other scenarios, adaptive adjustments can be made. For example, in user behavior anomaly detection, the object is the user, the attribute is the behavior, and the dimension is login, transaction, browsing, favorites, etc. In network traffic anomaly detection, the object is network traffic, the attribute is traffic, and the dimension can be inbound traffic, outbound traffic, request frequency, etc. In system performance anomaly detection, the object is the server, the attribute is the performance indicator, and the dimension can be CPU usage, memory usage, disk I / O, etc. In medical health data anomaly detection, the object is the patient, the attribute is the vital signs, and the dimension can be heart rate, blood pressure, blood oxygen saturation, etc. In environmental monitoring, the object is the environment, the attribute is the environmental factor, and the dimension can be air quality, water quality, temperature, humidity, etc. In this way, the algorithm can be flexibly applied to different business scenarios to meet diverse anomaly detection needs.

[0070] For step S201, for the commodity, first obtain the original data stream related to the commodity and the attribute to be detected. For example, a self-operated commodity has a low-price risk event due to the superposition of multiple coupons for each full-reduction promotion, with a cumulative loss of 50W+, then the self-operated commodity is the object to be monitored. Based on the self-operated commodity in which a price risk event occurs, obtain its normal sales data from the operating system, upstream system, cloud or server, that is, the normal sales data here is the original data stream. Since the original data stream is continuously flowing in, this solution sets a certain window step size, such as a 5-minute time granularity, then the original data stream includes data streams at multiple times, and each data stream includes the average transaction price, total sales volume, total gross profit and other data related to the self-operated commodity price, such as data stream X within 5 minutes based on Flink statistics.

[0071] For step S202, first, according to preset dimensions, such as average transaction price, total sales volume, total gross profit, etc., the data stream X within 5 minutes is split into data of multiple dimensions, such as the average transaction price of 5 minutes, the order volume of 5 minutes, and the gross profit of goods in 5 minutes. Therefore, for the data stream at each moment, the size is (r*d), r represents the number of data stream sample records, that is, the data volume, and d represents the dimension size, where d=3. Assume that the normal sales data collected is from the past day, with a total of 256 sample data. The 256 sample data here are not strictly tested according to 288 data in one day (24 hours a day, once every 5 minutes, 12 times an hour, and 24*12=288 sample data are required in one day).

[0072] Get the mean of each dimension of data stream X and standard deviation For the initial moment, that is, the first moment, we can directly base and Calculate the mean and standard deviation of each dimension at the initial moment. For the remaining moments, such as the second 5 minutes, it is necessary to use the data stream parameter dynamic update method to determine the mean and standard deviation of each dimension. The specific calculation formula is as follows:

[0073]

[0074] Among them, z t Represents a single sample data at time t. For example, the single sample data above refers to the time series data of 1*3 size (5-minute average transaction price, 5-minute order volume, 5-minute commodity gross profit). N represents the size of the sample capacity involved in the calculation of the estimation at time t, that is, the amount of data. Represents the mean value of the data stream in a certain dimension at time t, Represents the standard deviation of the data stream in a certain dimension at time t.

[0075] After calculating the mean of each dimension at time t and standard deviation After that, you need to do the following steps:

[0076] 1. Zscore standardization: Among them, Xt represents the data stream at the tth moment. For example, the original sequence stream X is formed at a time granularity of 5 minutes, and the size is (r*d), where r represents the number of data stream sample records and d represents the dimension size. Z: sequence stream data after Zscore standardization processing, size (r*d).

[0077] 2. Average pooling: Z′=avgPool(Z, kernel_size, stride). kernel_size: convolution kernel size. If it is set to 6, it represents the pooling result of every 5 minutes of data within 30 minutes. The default stride step size is 1. Z': represents the data flow after pooling, size (r*d).

[0078] 3. Get the sliding window data stream: Z t =slide(Z′,n). Z t Represents a windowed data stream at time t, with a size of (n*d), where n represents the window sample size and d represents the dimension size. t Represents a time series with a window size of n that slides backward at time t. It can be single-dimensional or multi-dimensional.

[0079] After calculating the window data flow Z at each moment t After that, for the window data stream Z t Vectorize the sequence pattern E tIn order to characterize the sequence pattern, this scheme proposes a self-isolation strategy to improve the ability to capture abnormal sequence patterns, as shown below: t =SelfIsolation(Z t ). Among them, the degree of outlier characterizes the sequence E t Represents sequence Z t The sequence is embedded after encoding through the self-isolation strategy. For details, see Figure 3 The description shown is not repeated here.

[0080] For step S203, after calculating the outlier degree characterization sequence E t After that, based on E t Search the memory space and obtain the data E in the memory block mb , E mb Represents the embedding vector data set after the memory blocks are stacked. The memory space (MemSpace) consists of a series of memory blocks (MemBlock), each of which has a corresponding index (memKey), which is accessed through HashMap<memKey,MemBlock> For details, see Figure 5 shown.

[0081] Get the minimum error s of the sequence mode t ,s t =min||E mb -E t ||2. s t Represents the degree of outlier characterization sequence E t With the embedding vector dataset E mb The minimum error score of ||E mb -E t ||2 refers to E mb and E t The L2 norm (or Euclidean distance) between them represents the distance between the two vectors. Assume that the degree of outliers characterizes the sequence E t For [e1, e2, .., e n ], then E mb is m*n, where m is the number of outlier representation sequences retrieved, for example, 3, then E mb is 3*n. To calculate the distance here, we need to convert E mb The degree of outliers in the sequence characterization is related to E t Calculate the L2 norm and then select the E corresponding to the minimum L2 norm mb .

[0082] For step S204, the abnormality score is standardized based on the abnormal water level line, and the 0-1 mapping is performed using an activation function based on tanh. The relevant logic is as follows:

[0083]

[0084] Among them, s t represents the minimum error score at time t, l t-1 Represents the highest water level at time t-1, that is, the abnormal water level. ω represents the mapping coefficient of the abnormal score standardization process (such as ω = 0.1). When the mapping coefficient is 0.1, the score corresponding to the abnormal water level is also around 0.1. See Table 1:

[0085]

[0086] Water level 1 is the abnormal water level corresponding to the normalized processing. The mapping coefficient at water level 1 is 0.1, and the abnormal score is 0.099668. At this time, the score of the abnormal water level is close to 0.1, that is, the score below 0.1 belongs to the safe area, and the score above 0.1 belongs to the abnormal area. The larger the score, the higher the probability of abnormal risk.

[0087] The method provided in the above embodiment can effectively extract abnormal information related to the attribute to be detected from the original data stream through systematic preprocessing and encoding strategies. By generating a sequence representing the degree of outliers and searching and comparing the memory space, abnormal situations in the data stream can be accurately identified, thereby improving the accuracy and reliability of anomaly detection. The anomaly score is calculated based on the abnormal water level line at the previous moment and the minimum error score, which can respond to changes in the data stream in a timely manner, thereby realizing rapid identification and processing of abnormal events, and has strong real-time and adaptability.

[0088] See also Figure 3 , shows a schematic flow chart of an optional anomaly detection method according to an embodiment of the present invention, comprising the following steps:

[0089] S301: For any position point data in the window data stream, determine the distance from other position point data in the window data stream by using a preset distance determination strategy between sequence elements;

[0090] S302: Sort the distances based on the position sequence of each position point data in the window data stream to obtain a distance vector of any position point data;

[0091] S303: Determine the relative position distance code of any position point data based on the position of any position point data in the window data stream and the pre-configured sequence window size;

[0092] S304: determining a normal distance metric sum of the any position point data based on the distance vector and the relative position distance code of the any position point data, and obtaining an outlier degree representation of the any position point data by scaling transformation;

[0093] S305: According to the position sequence of each position point data in the window data stream, the outlier degree representations of all position point data are combined to obtain an outlier degree representation sequence.

[0094] In the above implementation, after calculating the window data flow Z at each moment t After that, for the window data stream Z t Vectorize the sequence pattern E t In order to characterize the sequence pattern, this scheme proposes a self-isolation strategy to improve the ability to capture abnormal sequence patterns, as shown below: t =SelfIsolation(Z t ). Among them, E t Represents sequence Z t The sequence is embedded after encoding through self-isolation strategy. t Represents a time series with a window size of n that slides backward at time t. It can be single-dimensional or multi-dimensional. Figure 4 The description shown is not repeated here.

[0095] n represents the window sample size, and the aforementioned Z t That is, a sequence {Z1, Z2, .., Z n}. The self-isolation strategy is to capture the sequence {Z1, Z2, .., Z n} is a method to measure the degree of mutual outlier between internal elements, and also a representation of sequence patterns. n}After encoding, it is converted into an Embedding vector [e1, e2, .., e n ], which satisfies the condition ∑e i =1, the relevant internal processing logic is as follows:

[0096] 1. Distance between sequence elements: D i = Dif(Z t , Z i )=Z t -Z i . Z i represents the sequence {Z1, Z2, .., Z n}, the point data at the i-th position, size (1*d), d represents the dimension size, corresponding to the transaction price, sales volume, gross profit and other dimensions. i Represents Z i With Z t The distance vector of each element in can be obtained by sorting the distances according to the position order of each location point data in the window data stream.

[0097] 2. Relative position distance:

[0098]

[0099] Where n represents the pre-configured window sample size, RP i Represents the relative position distance encoding of the i-th position in the window sequence.

[0100] 3. L-norm distance metric and: l i =||D i +RP i || k . i Represents Z i With Z t The overall k-norm distance metric of and, k=1.

[0101] 4. Characterization of outlier degree:

[0102]

[0103] Among them, e i Represents Z i In sequence Z t The overall outlier level in the i After scaling, softmax transformation is performed.

[0104] In addition to the above methods, the outlier degree representation can also be calculated in the following ways:

[0105]

[0106] e i =softmax(λ*l i )

[0107] Among them, λ represents the scaling factor corresponding to softmax, e i Represents Z i In sequence Z t The overall outlier level in the i After scaling, softmax transformation is performed. Considering that the previous implementation may be abnormal when encountering a maximum value, this solution preferably uses this implementation to calculate the outlier degree representation.

[0108] The method provided in the above embodiment is based on the self-isolation mechanism, and through precise distance calculation and sorting, it can comprehensively evaluate the degree of abnormality of the data at each position point in the window data stream. Through the combination of distance, relative position distance encoding and paradigm distance measurement, it is ensured that the characterization of the degree of outliers not only considers the absolute distance between the data, but also takes into account the positional relationship of the data in the sequence. This method has the ability to detect multi-dimensional time series streams, improves the ability to capture abnormal patterns of object attribute time series, and enables the system to better capture potential abnormal patterns, thereby achieving more effective monitoring and response in a dynamic data environment, so as to detect abnormalities early.

[0109] See also Figure 5 , shows a schematic flow chart of another optional anomaly detection method according to an embodiment of the present invention, comprising the following steps:

[0110] S501: Encoding the outlier degree representation sequence to obtain a sequence pattern index, and constructing an index tree based on the sequence pattern index;

[0111] S502: Perform nearest neighbor search on the sequence pattern index using the index tree to obtain a first sequence pattern index, and determine a first memory block based on the first sequence pattern index;

[0112] S503: stacking the outlier degree representation sequence in the first memory block to obtain data in the memory block.

[0113] In the above implementation mode, based on E t Search the memory space and obtain the data E in the memory block mb The memory space (MemSpace) is composed of a series of memory blocks (MemBlock), each of which has a corresponding index (memKey), which is accessed through HashMap<memKey,MemBlock> The relevant internal processing logic is as follows:

[0114] 1. Sequence pattern index, memKey←K t =topKHash(E t ), see Figure 6(a), where P represents the internal position index information of the sequence, K t Represents E t After encoding the sequence pattern index through topKHash, the internal implementation logic of topKHash is as follows:

[0115] memKey←K t =topKHash(E t )

[0116]

[0117] in, Represents E t The Large element values, is a variable, Represents 1 maximum, Represents 2 as the maximum, and so on. Represents E t Middle Large element value position index. k i Represents the sequence pattern index K t As shown in Figure 6, when TOPK = 3 (i.e. ), select the maximum 3 E t The data and the corresponding position index are mapped, that is, 9-0.2882, 0-0.2055, 8-0.1219, and the size of K is 7 (2*3+1) t .

[0118] 2. Index tree construction. The index tree (IndexTree) is constructed based on the sequence pattern index data set. The basic algorithm logic is as follows:

[0119] 1) Initialize an empty node set Nodes, the basic structure Map<int, subNodes>;

[0120] 2) Traverse the index data set K through a for loop * , get the sequence pattern index K at time t t ; Among them, the index data set K * Including the sequence pattern index of historical moments and the sequence pattern index of the current moment t;

[0121] 3) Based on the previous moment, i.e., moment t-1, determine the historical index tree closest to the current one, obtain the root node Node0 from the historical index tree, and then obtain Node0.keys of the root node; wherein Node0.keys includes the index key set of the next layer of child nodes;

[0122] 4) Get index K t The first index value k1 in Node0.keys is used to determine whether k1 is included in Node0.keys. If so, the child node Node corresponding to k1 is directly determined. 1j If not included, a new child node Node is created directly in the first level based on k1 1j . Among them, 1 represents the first level under the root node, and 1j represents the jth position in the first level.

[0123] 5) Continue the for loop to traverse the index code Kt , get the i-th element k i , determine Node i-1 Does .keys include k? i , if k i ∈Node i_1 .keys, directly based on Node i-1 Get k i Corresponding child node Node i . Add a Node-based i-1 Next k i The corresponding new child node Node i . This can also be Node ij , represents the jth node in the i-th layer child node set.

[0124] 6) The traversal is finished and the index tree based on the node set Nodes is constructed.

[0125] The above steps 1 to 6 involve the entire initialization process, while updating the index tree under the Streaming data flow only involves steps 3 to 5.

[0126] 3. Nearest neighbor search: K′ t =Query(K t ). Based on the sequence pattern index K t Perform nearest neighbor search to find the index K′ of the closest PatternBlock t , the basic algorithm logic is as follows:

[0127] 1) Initialize an empty K′ t , Shape size and K t same;

[0128] 2) for loop traverses the sequence pattern index K t , get the i-th element k i , determine from the index tree the number of nodes corresponding to k i The corresponding target node, such as Node ij ;

[0129] 3) Matching Node based on index tree ij The other child nodes with the smallest distance ir , where r represents the node at the rth position. That is, among the nodes at the i-th level, the distance between the node at the rth position and the node at the i-th position is the smallest.

[0130] 4) The aforementioned k i With Node ij Correspondingly, there are also elements and Node irCorrespondingly, assuming k′ i , write K′ t ;

[0131] 5) End the traversal and output K′ t .

[0132] 4. Get memKey and memory block memBlock

[0133] memKey=StringJoiner(K t )

[0134] memBlock=MemSpace(memKey)

[0135] E mb =Vstack(memBlock)

[0136] Among them, memKey represents the sequence pattern index K t All elements k in i The concatenated string. memBlock represents the sequence pattern index K t Memory block. mb Represents the embedding vector dataset after the memory blocks are stacked.

[0137] For example, for the sequence pattern index K at time t t , a unique index can be generated through the above memKey, a memory block (i.e., the second memory block) is generated based on memBlock, and then the outlier degree at the tth moment is represented as a sequence E t Write into the second memory block. Similarly, when determining K′ t After that, we can use K′ t Determine the first memory block, and then obtain the outlier degree representation sequence in these memory blocks to stack and obtain the above E mb As shown in Figure 6(b), based on K′ t Three memory blocks are matched, assuming they are ABC. The outlier degree representation sequence in A is [0.45, ..., 0.16, ..., 0.33], the outlier degree representation sequence in B is [0.45, ..., 0.17, ..., 0.32], and the outlier degree representation sequence in C is [0.46, ..., 0.15, ..., 0.32]. The effect after stacking is shown in Figure 6(b).

[0138] 5. Memory block update. When the capacity of the memory block (memBlock) is limited, update and replacement are performed based on the principle of maximum similarity matching to maintain maximum information memory. The main logic is as follows:

[0139]

[0140] memBlock(i mb )=E t

[0141] Among them, i mb Represents E t The position information with the minimum reconstruction error in the memory block (memBlock). mb Represents the actual size of the memory block. mbs represents the maximum capacity of the memory block. mb )=E t , representing the sequence number i in the memory block mb data, using the latest E t To update and replace the original data, this is equivalent to reassigning. In mathematics and optimization, argmin is a commonly used symbol, which represents the value of the independent variable that makes a function reach the minimum value.

[0142] Taking Figure 6(b) as an example, the outlier degree characterization sequence in A is [0.45, ..., 0.16, ..., 0.33], the outlier degree characterization sequence in B is [0.45, ..., 0.17, ..., 0.32], and the outlier degree characterization sequence in C is [0.46, ..., 0.15, ..., 0.32]. Assume that the outlier degree characterization sequence at time t is represented as D[0.46, ..., 0.16, ..., 0.32], the maximum capacity of the memory block is set to 5 in advance, and each outlier degree characterization sequence occupies 1 place. For E mb , currently only three outlier degree characterization sequences are stored, so the actual size is 3, which is less than the maximum capacity size of 5. In this case, D[0.46,...,0.16,...,0.32] is directly added to E mb At the end of the D, set a new number mb for D, such as 4. But suppose the maximum capacity is 3. In this case, E mb The actual size is equal to the maximum capacity size. Assuming that the error score between A and D is the smallest, A needs to be replaced by D, and the position index of D is set to the position index of A. For example, if A is ranked first, then the position index of D is 1.

[0143] The method provided in the above embodiment proposes a memory space (memSpace) idea. By constructing an index tree and performing nearest neighbor retrieval, it is possible to efficiently manage and query the outlier degree characterization sequence, quickly locate historical patterns similar to the current data stream, and significantly improve the speed and accuracy of data retrieval. The availability of data in the memory block is enhanced through stacking processing, so that the system can more effectively analyze and identify potential abnormal behaviors. In addition, the memory block data is updated based on the maximum similarity matching principle, aiming to maximize information memory and solve the problem of invalid warning caused by short memory cycle. This method provides strong support for real-time monitoring and decision-making, and improves the overall performance and responsiveness of the system.

[0144] See also Figure 7 , shows a schematic flow chart of another optional anomaly detection method according to an embodiment of the present invention, comprising the following steps:

[0145] S701: Determine the moment before any moment, and obtain the average water level of the reconstruction error at the previous moment and the fluctuation of the reconstruction error water level;

[0146] S702: Determine the average waterline of the reconstruction error at any moment and the fluctuation size of the reconstruction error waterline based on the average waterline, the fluctuation size and the minimum error score at any moment;

[0147] S703: Determine an abnormal water level at any moment based on the average water level of the reconstruction error at any moment, the fluctuation size of the reconstruction error water level and a preset fluctuation coefficient.

[0148] In the above implementation, a standardized anomaly score (AnomalyScore) is obtained based on the scorer. The relevant concepts and scoring logic are as follows:

[0149] 1. Concept drift mainly refers to the situation where the data distribution changes over time under the same data source. At this time, the range of reconstruction error will also become larger. If the anomaly detection model or strategy does not adapt to this change, the result may continue to be judged as abnormal. For example, when a certain brand of mobile phone is newly released, the price is about 10,000 yuan, and in the middle and late stages, the price drops to 5,000 to 6,000 yuan. At this time, in addition to the changes in the price and gross profit of the product, the sales data distribution of the product order also changes significantly. See Figure 8 shown.

[0150] 2. Abnormal waterline. In addition to the concept drift in the data stream, there is still the problem of inconsistent reconstruction error range between the distribution of different commodity data sources, which leads to limitations in the application layer and lack of relatively stable threshold judgment. In order to perform standardized processing, the concept of an abnormal waterline is proposed. Below the waterline is normal (safe area), above the waterline is abnormal (potential risk area), the higher the waterline, the greater the degree of abnormality, and the higher the risk probability. The following is the online update logic of the adaptive waterline under data streaming:

[0151]

[0152] Among them, μ t Represents the average water level of the reconstruction error at time t; σ t Represents the fluctuation size (standard deviation) of the reconstruction error waterline at time t; Represents the adaptive water level cycle size, which can be customized.

[0153] In calculating μ t and σ t After that, the abnormal water level can be calculated. The calculation method is: t =μ t +η*σ t Among them, l t Represents the highest water level at time t, i.e., the abnormal water level. η represents the fluctuation coefficient, and this scheme is preferably applicable to a value range of 3 to 6.

[0154] The method provided in the above embodiment can dynamically determine the abnormal water level by systematically analyzing the average water level of the reconstruction error and its fluctuation size, thereby effectively identifying and evaluating abnormal situations. This method not only improves the output stability of the scorer, but also can adapt to the drift of data, ensure the accuracy and reliability of the scoring results, provide a more accurate basis for anomaly detection, and enhance the system's response capability to abnormal events and decision support.

[0155] See also Fig. 9 FIG. 1 is a flow chart of a specific abnormality detection method according to an embodiment of the present invention, comprising the following steps:

[0156] 1. Determine the commodity to be tested, and obtain the original data stream related to the price to be tested from the data related to the commodity; wherein the original data stream includes data streams at multiple times;

[0157] 2. For each moment, the data stream is processed by the preprocessing strategy to obtain the window data stream Z t ;

[0158] 3. By presetting the self-isolation strategy, the window data stream is encoded and processed to obtain the outlier degree representation sequence E t ;

[0159] 4. Perform memory space retrieval processing based on the outlier degree representation sequence to obtain the data E in the memory block mb ;

[0160] 5. Compare the data in the memory block and the outlier degree characterization sequence to determine the minimum error score s t ;

[0161] 6. Obtain the abnormal water level line at the previous moment, and determine the abnormal score based on the minimum error score and the abnormal water level line;

[0162] 7. In response to the anomaly score exceeding a preset threshold, determining that the data flow at each moment is abnormal.

[0163] Case background: Due to the superposition of multiple coupons for each full discount promotion, the self-operated goods caused a low-price risk event, with a cumulative loss of 500,000+. Main dimensions: 5-minute average transaction price, 5-minute order volume, and 5-minute product gross profit. Algorithm initialization data flow: Based on the normal sales data of the goods that have price risk events, the algorithm model is initialized for the past day, with a total of 256 sample data. The sequence distribution of its normal data flow is shown in Figure 10(a). Taking 5 minutes as a moment, the risk event data flow under each dimension is calculated (caliber: including the current day + the previous day), as shown in Figure 10(b). The calculated anomaly score is shown in Figure 10(c). The anomaly of the sequence pattern at the right circle in the figure (corresponding to the right circle in Figure 10(b)) reaches the highest. At this time, intervention is carried out. The cumulative loss of the product is within 2,500, while the actual loss amount reaches 500,000+, involving an order volume of 20,000+. The detailed data is shown in the table shown in Figure 10(d) below (only part is shown), and the abnormal price is shown in the figure.

[0164] For the left circle shown in Figure 10(c), it just exceeds the abnormal water level line of 0.1. The time at this time is between 2 and 3 in the morning, and the superimposed coupons take effect at 0:00. At this time, the order volume is relatively small, but the average transaction price has dropped to 12.75, indicating that a small number of users have discovered that the superimposed coupons have caused the goods to be sold at ultra-low prices, and finally at 4 pm on the same day (the right circle) began to explode. The detailed data after the intervention is shown in Figure 10(e) below (only part of it is shown), which shows abnormal prices.

[0165] Traditional anomaly detection methods are mostly based on a global perspective, and are most abnormal when the loss is at its peak, lacking contextual information based on the time series flow. This solution solves these problems and can detect abnormal changes in sequence patterns in the time series flow earlier.

[0166] See also Fig.11 , shows a schematic diagram of the main modules of an anomaly detection device 900 provided by an embodiment of the present invention, including: an acquisition module 1101, used to determine the target object, and obtain the original data stream related to the attribute to be detected from the data related to the target object; wherein the original data stream includes data streams at multiple moments; a processing module 1102, used to process the data stream by a preprocessing strategy at each moment to obtain a window data stream, and encode the window data stream by a preset self-isolation strategy to obtain an outlier degree representation sequence; a retrieval module 1103, used to perform memory space retrieval processing based on the outlier degree representation sequence to obtain data in the memory block, and compare the data in the memory block with the outlier degree representation sequence to determine the minimum error score; an abnormality module 1104, used to obtain the abnormal water level line of the previous moment, determine the abnormality score based on the minimum error score and the abnormal water level line, and determine that the data stream at each moment is abnormal in response to the abnormality score exceeding the preset threshold.

[0167] In the implementation device of the present invention, the processing module 1102 is used to: determine a preset dimension, calculate the mean and standard deviation of the data in the same dimension in the data stream; use the mean and standard deviation to standardize each data in each dimension to obtain a first data stream; use a preset convolution kernel and a preset step size to perform mean pooling on the first data stream to obtain a second data stream; use a pre-configured sequence window size to process the second data stream to obtain a window data stream.

[0168] In the implementation device of the present invention, the processing module 1102 is used to: for the initial moment, respectively calculate the mean and standard deviation of all data in each dimension; for each moment other than the initial moment, obtain the data volume of the data stream at each moment, and obtain the mean and standard deviation of each dimension in the data stream at the previous moment, so as to calculate the mean and standard deviation of each dimension in the data stream at each moment.

[0169] In the implementation device of the present invention, the processing module 1102 is used to: for any position point data in the window data stream, determine the distance with other position point data in the window data stream by using a preset distance determination strategy between sequence elements; sort the distances based on the position order of each position point data in the window data stream to obtain the distance vector of any position point data; determine the relative position distance coding of any position point data based on the position of any position point data in the window data stream and a pre-configured sequence window size; determine the normal form distance metric and of any position point data based on the distance vector and relative position distance coding of any position point data, and obtain the outlier degree representation of any position point data by scaling transformation; combine the outlier degree representations of all position point data according to the position order of each position point data in the window data stream to obtain an outlier degree representation sequence.

[0170] In the implementation device of the present invention, the scaling transformation obtains the outlier degree representation of any one of the position point data, including the following methods: accumulating the sum of the paradigm distance metrics of all the position point data, and determining the outlier degree representation of any one of the position point data based on the outlier degree representation of any one of the position point data and the paradigm distance metrics of any one of the position point data; accumulating the sum of the paradigm distance metrics of all the position point data, and determining the outlier degree representation of any one of the position point data based on the outlier degree representation of any one of the position point data, the paradigm distance metrics of any one of the position point data, and a pre-configured sequence window size and number of dimensions.

[0171] In the implementation device of the present invention, the outlier degree representation sequence is a one-dimensional vector, and the sum of all outlier degree representations in the outlier degree representation sequence is 1.

[0172] In the implementation device of the present invention, the retrieval module 1103 is used to: encode the outlier degree representation sequence to obtain a sequence pattern index to build an index tree based on the sequence pattern index; use the index tree to perform nearest neighbor retrieval on the sequence pattern index to obtain a first sequence pattern index, and determine the first memory block based on the first sequence pattern index; stack the outlier degree representation sequence in the first memory block to obtain data in the memory block.

[0173] In the implementation device of the present invention, the retrieval module 1103 is used to: sort the outlier degree representations in the outlier degree representation sequence in order from large to small; determine the preset variable value, extract the outlier degree representations of the preset variable value according to the sorting order, and determine the position index of each extracted outlier degree representation in the outlier degree representation sequence; map based on the extracted outlier degree representation and the position index to obtain multiple elements, and then generate a sequence pattern index.

[0174] The implementation device of the present invention also includes: forming a character string based on the numerical concatenation of each element in the sequence pattern index, and generating a second memory block, using the character string as the index of the second memory block, and storing the outlier degree representation sequence in the second memory block.

[0175] In the implementation device of the present invention, the retrieval module 1103 is used to: obtain the historical index tree of the previous moment, determine the root node in the historical index tree, and obtain the node index key set of the root node; wherein the node index key set includes the index key set of the next layer of child nodes; for the first element in the sequence pattern index, in response to the node index key set not including the first element, create a new child node under the root node based on the first element; loop through each element in the sequence pattern index, determine the position of each element in the sequence pattern index, and determine the node at the level where the position is minus 1; obtain the node index key set of the node, and in response to the node index key set not including each element, create a new child node under the node based on each element to obtain a new index tree.

[0176] In the implementation device of the present invention, the retrieval module 1103 is used to: initialize an empty first sequence pattern index; wherein the structure of the first sequence pattern index is the same as the structure of the sequence pattern index; loop through the sequence pattern index, and for each element, determine the target node corresponding to each element in the index tree; and determine the position of each element in the sequence pattern index; from the index tree level where the target node is located, determine other nodes with the shortest distance from the target node, and write the elements corresponding to the other nodes into the said position in the first sequence pattern index.

[0177] The implementation device of the present invention also includes an update module, which is used to: obtain the maximum capacity size pre-configured for the memory block, and determine the actual size of the data in the memory block; in response to the actual size being smaller than the maximum capacity size, add the outlier degree representation sequence to the data in the memory block, and set the position index to obtain a new data in the memory block; in response to the actual size being equal to the maximum capacity size, determine the position index corresponding to the minimum error score, and replace the original outlier degree representation sequence of the position index with the outlier degree representation sequence.

[0178] In the implementation device of the present invention, the process of determining the abnormal water level line at any moment includes: determining the moment before any moment, obtaining the average water level line of the reconstruction error at the previous moment, and the fluctuation size of the reconstruction error water level line; based on the average water level line, the fluctuation size and the minimum error score at any moment, determining the average water level line of the reconstruction error at any moment, and the fluctuation size of the reconstruction error water level line; based on the average water level line of the reconstruction error at any moment, the fluctuation size of the reconstruction error water level line and a preset fluctuation coefficient, determining the abnormal water level line at any moment.

[0179] In the implementation device of the present invention, the anomaly module 1104 is used to: determine the quotient of the minimum error score and the anomaly water level, and determine the product of the quotient and the preset mapping coefficient, and use the activation function to process the product to obtain the anomaly score.

[0180] In addition, the specific implementation content of the device described in the embodiment of the present invention has been described in detail in the method described above, so the repeated content will not be described again here.

[0181] Fig.12 An exemplary system architecture 1200 to which embodiments of the present invention may be applied is shown, including terminal devices 1201 , 1202 , 1203 , a network 1204 and a server 1205 (only an example).

[0182] Terminal devices 1201, 1202, and 1203 can be various electronic devices with display screens and supporting web browsing, and are installed with various communication client applications. Users can use terminal devices 1201, 1202, and 1203 to interact with server 1205 through network 1204 to receive or send messages, etc.

[0183] The network 1204 is used to provide a medium for communication links between the terminal devices 1201, 1202, 1203 and the server 1205. The network 1204 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0184] The server 1205 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using the terminal devices 1201, 1202, and 1203 (for example only). The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal device. It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 1205, and accordingly, the device is generally set in the server 1205.

[0185] It should be understood that Fig.12 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0186] Reference below Fig.13 , which shows a schematic diagram of the structure of a computer system 1300 of a terminal device suitable for implementing an embodiment of the present invention. Fig.13 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0187] like Fig.13As shown, the computer system 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage part 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the system 1300 are also stored. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0188] The following components are connected to the I / O interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed, so that a computer program read therefrom is installed into the storage section 1308 as needed.

[0189] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1309, and / or installed from the removable medium 1311. When the computer program is executed by the central processing unit (CPU) 1301, the above-mentioned functions defined in the system of the present invention are executed.

[0190] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0191] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0192] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor. For example, they may be described as follows: a processor includes an acquisition module, a processing module, a retrieval module, and an exception module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, an exception module may also be described as an "anomaly detection module."

[0193] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device executes any of the above-mentioned abnormality detection methods.

[0194] The computer program product of the present invention comprises a computer program, and when the computer program is executed by a processor, the abnormality detection method in the embodiment of the present invention is implemented.

[0195] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An anomaly detection method, characterized in that: include: Determine the target object, and obtain the original data stream related to the attribute to be detected from the data related to the target object; wherein the original data stream includes data streams at multiple moments; At each moment, the data stream is processed by a preprocessing strategy to obtain a window data stream, and the window data stream is encoded by a preset self-isolation strategy to obtain a sequence representing the degree of outliers; Perform memory space retrieval processing based on the outlier degree representation sequence to obtain data in the memory block, and compare the data in the memory block with the outlier degree representation sequence to determine the minimum error score; The abnormal water level line of the previous moment is obtained, the abnormal score is determined based on the minimum error score and the abnormal water level line, and in response to the abnormal score exceeding a preset threshold, it is determined that the data flow at each moment is abnormal.

2. The method according to claim 1, characterized in that The process of processing the data stream by the preprocessing strategy to obtain the window data stream includes: Determine the preset dimension and calculate the mean and standard deviation of the data in the same dimension in the data stream; Using the mean and standard deviation, each data of each dimension is standardized to obtain a first data stream; Using a preset convolution kernel and a preset step size, perform mean pooling processing on the first data stream to obtain a second data stream; The second data stream is processed using a preconfigured sequence window size to obtain a window data stream.

3. The method according to claim 2, characterized in that The step of calculating the mean and standard deviation of the data in the same dimension in the data stream includes: For the initial moment, calculate the mean and standard deviation of all data in each dimension respectively; For each moment except the initial moment, the data volume of the data stream at each moment is obtained, and the mean and standard deviation of each dimension in the data stream at the previous moment are obtained to calculate the mean and standard deviation of each dimension in the data stream at each moment.

4. The method according to claim 2, characterized in that: The window data stream is encoded and processed by presetting a self-isolation strategy to obtain an outlier degree characterization sequence, including: For any position point data in the window data stream, the distance to other position point data in the window data stream is determined by presetting the distance determination strategy between sequence elements; Sort the distances based on the position order of each position point data in the window data stream to obtain a distance vector of any position point data; Determine the relative position distance coding of any one of the position point data based on the position of any one of the position point data in the window data stream and the pre-configured sequence window size; Based on the distance vector and relative position distance encoding of any one of the position point data, determine the normal distance metric sum of any one of the position point data, and obtain the outlier degree representation of any one of the position point data by scaling transformation; According to the position order of each position point data in the window data stream, the outlier degree representations of all position point data are combined to obtain an outlier degree representation sequence.

5. The method according to claim 4, characterized in that The scaling transformation obtains the outlier degree representation of the data of any one of the position points, including the following methods: Accumulating the sum of the normal distance metrics of all the location point data, and determining the outlier degree representation of any location point data based on the outlier degree representation of any location point data and the normal distance metric sum of any location point data; Accumulate the sum of the paradigm distance metrics of all location point data, and determine the outlier degree representation of any location point data based on the outlier degree representation of any location point data, the paradigm distance metric of any location point data, and the pre-configured sequence window size and number of dimensions.

6. The method according to claim 4 or 5, characterized in that: The outlier degree representation sequence is a one-dimensional vector, and the sum of all outlier degree representations in the outlier degree representation sequence is 1.

7. The method according to claim 1, characterized in that The memory space retrieval process is performed based on the outlier degree characterization sequence to obtain the data in the memory block, including: Encoding the outlier degree representation sequence to obtain a sequence pattern index, so as to construct an index tree based on the sequence pattern index; Performing nearest neighbor search on the sequence pattern index using the index tree to obtain a first sequence pattern index, and determining a first memory block based on the first sequence pattern index; The outlier degree representation sequence in the first memory block is stacked to obtain the data in the memory block.

8. The method according to claim 7, characterized in that The encoding process of the outlier degree representation sequence to obtain a sequence pattern index includes: Sort the outlier degree representations in the outlier degree representation sequence in descending order; Determine a preset variable value, extract outlier degree representations of the preset variable value in a sorted order, and determine a position index of each extracted outlier degree representation in the outlier degree representation sequence; Mapping is performed based on the extracted outlier degree representation and position index to obtain multiple elements, thereby generating a sequence pattern index.

9. The method according to claim 7 or 8, characterized in that: The method further comprises: A string is formed based on the numerical concatenation of each element in the sequence pattern index, and a second memory block is generated. The string is used as an index of the second memory block, and the outlier degree representation sequence is stored in the second memory block.

10. The method according to claim 7 or 8, characterized in that: The step of constructing an index tree based on a sequence pattern index includes: Obtain the history index tree at the previous moment, determine the root node in the history index tree, and obtain the node index key set of the root node; wherein the node index key set includes the index key set of the next layer of child nodes; For a first element in the sequence pattern index, in response to the node index key set not including the first element, creating a new child node under the root node based on the first element; Loop through each element in the sequence pattern index, determine the position of each element in the sequence pattern index, and determine the node at the level where the position is minus 1; A node index key set of the node is obtained, and in response to the node index key set not including each element, a new child node is created under the node based on each element to obtain a new index tree.

11. The method according to claim 7, characterized in that The method of performing nearest neighbor search on the sequence pattern index using the index tree to obtain the first sequence pattern index includes: Initialize an empty first sequence pattern index; wherein the structure of the first sequence pattern index is the same as the structure of the sequence pattern index; Loop through the sequence pattern index, for each element, determine the target node corresponding to each element in the index tree; and determine the position of each element in the sequence pattern index; From the index tree level where the target node is located, determine other nodes with the shortest distance from the target node, and write the elements corresponding to the other nodes into the positions in the first sequence pattern index.

12. The method according to claim 1 or 7, characterized in that: The method further comprises: Obtaining the maximum capacity size pre-configured for the memory block and determining the actual size of the data in the memory block; In response to the actual size being smaller than the maximum capacity, the outlier degree representation sequence is added to the data in the memory block, and the position index is set to obtain a new data in the memory block; In response to the actual size being equal to the maximum capacity size, a position index corresponding to the minimum error score is determined, and an original outlier degree characterization sequence of the position index is replaced with the outlier degree characterization sequence.

13. The method according to claim 1, characterized in that The process of determining the abnormal water level at any time includes: Determine the moment before any moment, obtain the average water level of the reconstruction error at the previous moment, and the fluctuation of the reconstruction error water level; Based on the average water level, the fluctuation size and the minimum error score at any moment, determine the average water level of the reconstruction error at any moment and the fluctuation size of the reconstruction error water level; Based on the average water level of the reconstruction error at any moment, the fluctuation size of the reconstruction error water level and the preset fluctuation coefficient, the abnormal water level at any moment is determined.

14. The method according to claim 1 or 13, characterized in that: The determining of the abnormality score based on the minimum error score and the abnormal water level line includes: The quotient of the minimum error score and the anomaly waterline is determined, and the product of the quotient and the preset mapping coefficient is determined, and the product is processed using an activation function to obtain an anomaly score.

15. An abnormality detection device, characterized in that: include: An acquisition module is used to determine a target object and acquire an original data stream related to the attribute to be detected from data related to the target object; wherein the original data stream includes data streams at multiple moments; A processing module is used to process the data stream at each moment through a preprocessing strategy to obtain a window data stream, and encode the window data stream through a preset self-isolation strategy to obtain an outlier degree representation sequence; A retrieval module is used to perform memory space retrieval processing based on the outlier degree representation sequence to obtain data in the memory block, and compare the data in the memory block with the outlier degree representation sequence to determine the minimum error score; The anomaly module is used to obtain the anomaly waterline of the previous moment, determine the anomaly score based on the minimum error score and the anomaly waterline, and determine that the data flow at each moment is abnormal in response to the anomaly score exceeding a preset threshold.

16. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 14.

17. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

18. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images