Abnormal data detection method and device, electronic device and storage medium

Abstract

The present invention discloses a method for detecting abnormal data, an apparatus thereof, an electronic device, and a storage medium, and relates to the field of big data. The method comprises: receiving parameter information, and after configuring an abnormality detection environment based on the configuration parameters, collecting a data set to be detected; and, based on operating parameters, using a preset detection algorithm to detect each data point in the data set to be detected, obtaining a detection result; and determining that the data set to be detected does not contain abnormal points if the number of preset points corresponding to all data points is greater than or equal to a preset point threshold. The present invention solves the technical problem in related technologies of poor timeliness in performing anomaly detection on real-time data in environments where high-performance computing platforms cannot be deployed.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a method for detecting abnormal data, a device thereof, an electronic device, and a storage medium. Background Art

[0002] Currently, commonly used algorithms for anomaly detection include pattern-based algorithms (such as the Luminol algorithm, a lightweight Python anomaly detection library for time series data analysis), neural network-based algorithms (such as the TS-Bert algorithm, a bidirectional encoding representation algorithm), and distance-based algorithms (such as the Nearest Neighbor (NN) algorithm, a nearest neighbor data association algorithm). These algorithms are primarily targeted at high-performance computer server platforms and cannot be run on portable or low-power platforms.

[0003] However, in actual production environments, some sensors generating real-time data lack the availability of high-performance, high-power computing environments. Related technologies, for environments requiring real-time monitoring and alerting of sensor data, can only transmit real-time data back over the network, with data processing performed only after detection is complete. This places high demands on low-latency networks and wastes network bandwidth.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present invention provide a method for detecting abnormal data, an apparatus thereof, an electronic device, and a storage medium, so as to at least solve the technical problem in the related art of poor timeliness of anomaly detection on real-time data in an environment where a high-performance computing platform cannot be deployed.

[0006] According to one aspect of an embodiment of the present invention, a method for detecting abnormal data is provided, comprising: receiving parameter information, wherein the parameter information comprises operating parameters and configuration parameters, and the operating parameters comprise at least a preset point threshold; collecting a data set to be detected when an abnormality detection environment is configured based on the configuration parameters; and detecting each data point in the data set to be detected using a preset detection algorithm based on the operating parameters to obtain a detection result, wherein the preset detection algorithm is an algorithm running in a preset detection device based on the abnormality detection environment, and the detection result comprises at least the number of preset points corresponding to each of the data points; and determining that no abnormal points exist in the data set to be detected when the number of preset points corresponding to all the data points is greater than or equal to the preset point threshold.

[0007] Optionally, the preset detection device is deployed with a first interface and a second interface, wherein the first interface is used to receive the parameter information transmitted by the preset platform, and the second interface is used to collect the data set to be detected, and the step of receiving parameter information includes: based on the first interface, receiving the parameter information transmitted by the preset platform, wherein the preset platform provides a parameter input interface and generates the parameter information based on the input parameters, and the operating parameters in the parameter information also include: a preset radius value, a preset window value, and a preset slider value.

[0008] Optionally, when the anomaly detection environment is configured based on the configuration parameters, the step of collecting the data set to be detected includes: when the anomaly detection environment is configured, opening the second interface; establishing a communication connection with a preset data source based on the second interface, wherein the preset data source is used to generate the data to be detected; and collecting the data set to be detected based on the communication connection.

[0009] Optionally, the step of collecting the data set to be detected based on the communication connection includes: determining each time point in the time period to be detected; based on the communication connection, collecting the time series data generated by the preset data source at each time point, and adding the time series data to the data set to be detected, wherein each time series data is represented as a data point.

[0010] Optionally, based on the operating parameters, the step of using a preset detection algorithm to detect each data point in the data set to be detected includes: dividing the data set to be detected according to the preset slider value to obtain multiple slider data; adding the slider data to the current processing window, wherein the window value of the current processing window is the preset window value; based on the current processing window, using the preset detection algorithm to detect each data point in the slider data.

[0011] Optionally, based on the current processing window, the step of using the preset detection algorithm to detect each data point in the slider data includes: determining all core points in the slider data and adding all core points to a preset core point list; detecting whether there is a core point in the preset core point list that is a preset distance away from the data point, wherein the preset distance is the product value of the preset radius value and a preset ratio; in the case that there is a core point that is a preset distance away from the data point, associating the data point to the preset distance list corresponding to the core point; or, in the case that there is no core point that is a preset distance away from the data point, adding the data point as a new core point to the preset core point list; and determining the preset number of data points based on the preset distance list.

[0012] Optionally, after determining the preset point number of the data point based on the preset distance list, it also includes: when the number value of the slider data in the current processing window is greater than the preset window value, sorting the slider data in the current processing window according to the time when the slider data was added to the current processing window to obtain a sorting result; based on the sorting result, determining the target slider data that was first added to the current processing window; eliminating the target slider data; when the preset point number of any data point in the target slider data is less than the preset point threshold, determining that the data point is the abnormal point.

[0013] According to another aspect of an embodiment of the present invention, a device for detecting abnormal data is also provided, including: a receiving unit for receiving parameter information, wherein the parameter information includes: operating parameters, configuration parameters, and the operating parameters include at least: a preset point threshold; a collection unit for collecting a data set to be detected when the abnormal detection environment is configured based on the configuration parameters; a detection unit for detecting each data point in the data set to be detected using a preset detection algorithm based on the operating parameters to obtain a detection result, wherein the preset detection algorithm is an algorithm running in the preset detection device based on the abnormal detection environment, and the detection result includes at least: the number of preset points corresponding to each of the data points; a determination unit for determining that there are no abnormal points in the data set to be detected when the number of preset points corresponding to all the data points is greater than or equal to the preset point threshold.

[0014] Optionally, the preset detection device is deployed with a first interface and a second interface, wherein the first interface is used to receive the parameter information transmitted by the preset platform, and the second interface is used to collect the data set to be detected, and the receiving unit includes: a first receiving module, used to receive the parameter information transmitted by the preset platform based on the first interface, wherein the preset platform provides a parameter input interface and generates the parameter information based on the input parameters, and the operating parameters in the parameter information also include: a preset radius value, a preset window value, and a preset slider value.

[0015] Optionally, the acquisition unit includes: a first opening module, used to open the second interface when the abnormality detection environment is configured; a first establishment module, used to establish a communication connection with a preset data source based on the second interface, wherein the preset data source is used to generate the data to be detected; and a first acquisition module, used to collect the data set to be detected based on the communication connection.

[0016] Optionally, the first acquisition module includes: a first determination submodule, used to determine each time point in the time period to be detected; a first acquisition submodule, used to collect time series data generated by the preset data source at each time point based on the communication connection, and add the time series data to the data set to be detected, wherein each time series data is represented as a data point.

[0017] Optionally, the detection unit includes: a first segmentation module, used to segment the data set to be detected according to the preset slider value to obtain multiple slider data; a first adding module, used to add the slider data to the current processing window, wherein the window value of the current processing window is the preset window value; a first detection module, used to detect each data point in the slider data based on the current processing window using the preset detection algorithm.

[0018] Optionally, the first detection module includes: a second determination submodule, for determining all core points in the slider data and adding all core points to a preset core point list; a first detection submodule, for detecting whether there is a core point in the preset core point list that is a preset distance away from the data point, wherein the preset distance is the product value of the preset radius value and a preset ratio; a first association submodule, for associating the data point to the preset distance list corresponding to the core point when there is a core point that is a preset distance away from the data point; a first addition submodule, for adding the data point as a new core point to the preset core point list when there is no core point that is a preset distance away from the data point; and a third determination submodule, for determining the preset number of the data points based on the preset distance list.

[0019] Optionally, the detection device also includes: a first sorting module, which is used to sort the slider data in the current processing window according to the time when the slider data was added to the current processing window to obtain a sorting result after determining the preset point number of the data point based on the preset distance list, when the number value of the slider data in the current processing window is greater than the preset window value; a first determination module, which is used to determine the target slider data that was added to the current processing window earliest based on the sorting result; a first elimination module, which is used to eliminate the target slider data; and a second determination module, which is used to determine that the data point is the abnormal point when the preset point number of any data point in the target slider data is less than the preset point threshold.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned abnormal data detection method.

[0021] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for detecting abnormal data.

[0022] In the present disclosure, parameter information is received, and when the anomaly detection environment is configured based on the configuration parameters, a data set to be detected is collected, and based on the operating parameters, a preset detection algorithm is used to detect each data point in the data set to be detected to obtain a detection result. When the number of preset points corresponding to all data points is greater than or equal to the preset point threshold, it is determined that there are no anomalies in the data set to be detected. In the present disclosure, after the anomaly detection environment is configured according to the received configuration parameters, the data set to be detected can be collected, and then based on the operating parameters, a preset detection algorithm that can be run in a preset detection device is used to detect each data point in the data set to be detected. If it is detected that the number of preset points corresponding to all data points is greater than or equal to the preset point threshold, it can be determined that there are no anomalies in the data set to be detected. Through the portable preset detection device, abnormal data detection can be performed in real time, effectively reducing the dependence on low-latency networks and the waste of communication channels, thereby solving the technical problem in the related art of poor timeliness of anomaly detection for real-time data in environments where high-performance computing platforms cannot be deployed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 is a flow chart of an optional abnormal data detection method according to an embodiment of the present invention;

[0025] Figure 2 is a schematic diagram of an optional abnormal data detection architecture according to an embodiment of the present invention;

[0026] Figure 3 is an optional flow chart for detecting data points according to an embodiment of the present invention;

[0027] Figure 4is a schematic diagram of an optional abnormal data detection device according to an embodiment of the present invention;

[0028] Figure 5 The figure is a hardware structure block diagram of an electronic device (or mobile device) for detecting abnormal data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] To facilitate those skilled in the art to understand the present invention, some of the terms or nouns involved in the embodiments of the present invention are explained below:

[0032] An algorithm is an accurate and complete description of a solution to a problem. It is a set of clear instructions for solving a problem. An algorithm represents a systematic description of a problem-solving strategy. In other words, it is able to produce a desired output within a finite time given a certain set of inputs.

[0033] Anomaly detection: The identification of items, events, or observations that do not match expected patterns or datasets. Anomalous items often represent problems such as financial fraud, structural defects, medical issues, and textual errors. Anomalies are also known as outliers, novelties, noise, deviations, and exceptions.

[0034] CPOD (Core-Point based Outlier Detection) algorithm: is an abnormal data detection algorithm based on N-order nearest neighbor relationship.

[0035] Euclidean distance (Euclidean metric) is a commonly used distance definition, referring to the true distance between two points in m-dimensional space, or the natural length of a vector, that is, the distance from the point to the origin. The Euclidean distance in two-dimensional and three-dimensional space is the actual distance between two points.

[0036] Abnormal data points: refers to data points that are inconsistent with the actual situation due to various reasons such as system noise and data errors.

[0037] Normal data points: refers to the data points fed back by the system that are consistent with the actual situation.

[0038] It should be noted that the abnormal data detection method and device in the present disclosure can be used in the field of big data when detecting abnormal data, and can also be used in any field other than the field of big data when detecting abnormal data. The present disclosure does not limit the application field of the abnormal data detection method and device.

[0039] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display and analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0040] The following embodiments of the present invention can be applied to various systems / applications / devices for detecting abnormal data. The present invention can combine anomaly detection algorithms (such as the CPOD algorithm, the DBSCAN algorithm (i.e., a density-based spatial clustering algorithm), the one-class SVM algorithm (i.e., a single-classification algorithm), etc.) with small computing platforms (i.e., preset detection devices, such as the Raspberry Pi platform, the Banana pro platform (i.e., an open source tablet computer), and other portable platforms), thereby moving data anomaly detection from the high-performance computing end to the front-end platform device that generates data in real time, thereby improving the real-time detection of errors generated in the data, reducing dependence on low-latency networks and waste of communication channels, improving production efficiency, and reducing system overhead.

[0041] The present invention will be described in detail below with reference to various embodiments.

[0042] Example 1

[0043] According to an embodiment of the present invention, an embodiment of a method for detecting abnormal data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] Figure 1 is a flow chart of an optional abnormal data detection method according to an embodiment of the present invention, such as Figure 1 As shown, the method includes the following steps:

[0045] Step S101: receiving parameter information, wherein the parameter information includes: operating parameters, configuration parameters, and the operating parameters at least include: a preset point threshold.

[0046] Step S102: After the abnormality detection environment is configured based on the configuration parameters, a data set to be detected is collected.

[0047] In step S103, based on the operating parameters, a preset detection algorithm is used to detect each data point in the detection data set to obtain a detection result, wherein the preset detection algorithm is an algorithm based on an abnormal detection environment and running in a preset detection device, and the detection result includes at least: the number of preset points corresponding to each data point.

[0048] Step S104 : When the number of preset points corresponding to all data points is greater than or equal to the preset point threshold, it is determined that there are no abnormal points in the data set to be detected.

[0049] Through the above steps, parameter information can be received, and when the abnormality detection environment is configured based on the configuration parameters, the data set to be detected is collected, and based on the operating parameters, a preset detection algorithm is used to detect each data point in the data set to be detected to obtain a detection result. When the number of preset points corresponding to all data points is greater than or equal to the preset point threshold, it is determined that there are no abnormal points in the data set to be detected. In an embodiment of the present invention, after the abnormality detection environment is configured according to the received configuration parameters, the data set to be detected can be collected, and then, based on the operating parameters, a preset detection algorithm that can be run in a preset detection device is used to detect each data point in the data set to be detected. If it is detected that the number of preset points corresponding to all data points is greater than or equal to the preset point threshold, it can be determined that there are no abnormal points in the data set to be detected. Through the portable preset detection device, abnormal data detection can be performed in real time, effectively reducing the dependence on low-latency networks and the waste of communication channels, thereby solving the technical problem in the related art of poor timeliness of abnormality detection of real-time data in environments where high-performance computing platforms cannot be deployed.

[0050] The following embodiments of the present invention are described in detail in conjunction with the above steps. The following steps can be applied to a preset detection device, such as a Raspberry Pi platform.

[0051] Step S101: receiving parameter information, wherein the parameter information includes: operating parameters, configuration parameters, and the operating parameters at least include: a preset point threshold.

[0052] Optionally, the preset detection device is deployed with a first interface and a second interface, wherein the first interface is used to receive parameter information transmitted by the preset platform, and the second interface is used to collect the data set to be detected. The step of receiving parameter information includes: based on the first interface, receiving parameter information transmitted by the preset platform, wherein the preset platform provides a parameter input interface and generates parameter information based on the input parameters. The operating parameters in the parameter information also include: preset radius value, preset window value, and preset slider value.

[0053] In an embodiment of the present invention, the preset detection device can provide a CPU with ARM technology (i.e., the preset detection device supports ARM (Advanced RISC Machine) architecture) and running memory (such as 4GB of running memory). The preset detection device can be compatible with a series of general-purpose operating systems (such as Linux, etc.), which makes it extremely versatile. By implementing a preset detection algorithm (such as CPOD algorithm) in a running environment (i.e., an abnormality detection environment, such as JAVA1.8), the preset detection algorithm can be run on this preset detection device. In addition, the preset detection device also provides multiple network interfaces (i.e., the preset detection device is deployed with a first interface and a second interface), so that the system running on the preset detection device can obtain data generated in the production environment in a variety of ways, and has extremely high scalability. The preset detection device mainly uses a network interface as a data receiving method.

[0054] In this embodiment, the first interface is used to receive parameter information transmitted by a preset platform (such as the SpringBoot (a framework) platform), and the second interface is used to collect the data set to be tested. This embodiment can receive parameter information (including operating parameters, configuration parameters, etc.) transmitted by the preset platform based on the first interface. The preset platform can provide a parameter input interface and generate parameter information based on the input parameters. The configuration parameters are used to configure the operating environment. The operating parameters include: a preset radius value, a preset window value, a preset slider value, a preset point threshold value, etc.

[0055] In an embodiment of the present invention, a series of parameters required by the preset detection algorithm (i.e., operating parameters) and system setting parameters (i.e., configuration parameters) can be pre-set in the network page interface provided by the preset platform (such as the SpringBoot platform). After that, the data source can communicate with the second interface (such as the Socket interface) in the preset detection device opened according to the parameter settings through the UDP (User Datagram Protocol) or TCP (Transmission Control Protocol) protocol, so that the data can be submitted to the preset detection algorithm of the preset detection device for anomaly detection. After the detection is completed, the data is returned to the data source or sent to other platforms according to the settings.

[0056] Step S102: After the abnormality detection environment is configured based on the configuration parameters, a data set to be detected is collected.

[0057] Optionally, when the anomaly detection environment is configured based on the configuration parameters, the step of collecting the data set to be detected includes: when the anomaly detection environment is configured, opening the second interface; establishing a communication connection with a preset data source based on the second interface, wherein the preset data source is used to generate the data to be detected; and collecting the data set to be detected based on the communication connection.

[0058] In an embodiment of the present invention, after the configuration parameters are configured to complete the anomaly detection environment, the data set to be detected can be collected. Specifically, after the anomaly detection environment is configured, the second interface can be opened, and then a communication connection between the second interface and the preset data source can be established. Then, through the communication connection, the data to be detected generated in real time by the preset data source can be collected.

[0059] Figure 2 is a schematic diagram of an optional abnormal data detection architecture according to an embodiment of the present invention, such as Figure 2 As shown, after the parameter setting is completed, the obtained parameter information is transmitted to the preset detection device through the first interface for parameter management, and the managed parameter information can be transmitted to the preset detection algorithm, and the data source can transmit the parameters to be detected to the preset detection algorithm through the first interface for detection.

[0060] Optionally, the step of collecting the dataset to be detected based on the communication connection includes: determining each time point in the time period to be detected; collecting time series data generated by the preset data source at each time point based on the communication connection, and adding the time series data to the dataset to be detected, wherein each time series data is represented as a data point. In an embodiment of the present invention, each time point in the time period to be detected can be first determined, and then the time series data generated by the preset data source at each time point can be collected through the communication connection, and the time series data can be added to the dataset to be detected.

[0061] In this embodiment, each time series data can be represented as a data point. For example, the preset data source is a wind power base station. During operation, the wind power base station will generate real-time data such as motor speed (r), motor temperature (T), and voltage (V). The time series data generated by a wind power base station at each time point is Tst = {rt, Tt, Vt}, and the time series data set (i.e., the data set to be detected) of this wind power base station within a period of time is Ts = {Ts1, Ts2, Ts3, ..., Tst}. Afterwards, the preset detection device can receive the data set to be detected collected by the sensor from the network through the second interface, or the second interface can be developed to directly read the data set to be detected from the sensor.

[0062] In step S103, based on the operating parameters, a preset detection algorithm is used to detect each data point in the detection data set to obtain a detection result, wherein the preset detection algorithm is an algorithm based on an abnormal detection environment and running in a preset detection device, and the detection result includes at least: the number of preset points corresponding to each data point.

[0063] Optionally, based on the operating parameters, the step of using a preset detection algorithm to detect each data point in the data set to be detected includes: dividing the data set to be detected according to a preset slider value to obtain multiple slider data; adding the slider data to the current processing window, wherein the window value of the current processing window is the preset window value; based on the current processing window, using a preset detection algorithm to detect each data point in the slider data.

[0064] Figure 3 is an optional flow chart for detecting data points according to an embodiment of the present invention, such as Figure 3 As shown, the method includes the following steps:

[0065] Step S301, dividing the data set to be detected according to the preset slider value to obtain multiple slider data;

[0066] Step S302, adding the slider data to the current processing window, wherein the window value of the current processing window is a preset window value;

[0067] Step S303 : Based on the current processing window, a preset detection algorithm is used to detect each data point in the slider data.

[0068] In an embodiment of the present invention, the preset detection algorithm may be a CPOD algorithm. The CPOD algorithm is a data anomaly detection algorithm based on an N-order nearest neighbor relationship. The CPOD algorithm assumes that most data generated in a natural environment should be in piles or clusters, that is, normal data should have a certain number of data points within a certain distance (generally Euclidean distance or any other distance algorithm). Therefore, a fixed maximum nearest neighbor radius R and minimum number of nearest neighbors K can be set, and these two parameters can be used to detect abnormal data points.

[0069] Data anomaly detection algorithms based on N-order nearest neighbor relationships have a high detection rate for abnormal data. However, because the algorithm's basic principle requires calculating the distance between all data points and other data points, its overall algorithm time performance is O(n²). Since time series data is often generated at the second level, based on the timeliness of the data, it can be determined that only data within a certain period of time is valid. However, by setting window parameters and only calculating the data within the window, although good detection performance can be achieved, the time efficiency is not satisfactory, especially on portable platforms (i.e., pre-set detection equipment) where storage and computing resources are very scarce.

[0070] In this embodiment, based on the fundamental principles of trigonometric functions and the principle that the sum of two sides must be greater than the third (i.e., the triangle theorem), data points are associated with a key point by setting core data points. This method effectively prunes the data within the entire calculation window, allowing for rapid detection of distant data points that require no calculation, while achieving O(n) time performance without increasing storage requirements.

[0071] In an embodiment of the present invention, a preset detection algorithm (the preset detection algorithm is an algorithm based on an abnormality detection environment and running in a preset detection device) can be used to detect each data point in the detection data set according to the operating parameters to obtain a detection result (the detection result includes: the preset number of points corresponding to each data point), specifically:

[0072] The data set to be detected can be divided according to the preset slider value (i.e., the slider size S) to obtain multiple slider data (i.e., the received data set to be detected can be divided according to the set slider size S to obtain multiple sliders of size S), and then the slider data is added to the current processing window (i.e., one slider is added to the current processing window each time), and the window value of the current processing window is the preset window value (i.e., the window size W), and then according to the current processing window, the preset detection algorithm is used to detect each data point in the slider data (i.e., after the slider enters the current processing window, the preset detection algorithm will detect each data point).

[0073] Optionally, based on the current processing window, a preset detection algorithm is used to detect each data point in the slider data, including: determining all core points in the slider data and adding all core points to a preset core point list; detecting whether there is a core point in the preset core point list that is a preset distance away from the data point, wherein the preset distance is the product of a preset radius value and a preset ratio; if there is a core point that is a preset distance away from the data point, associating the data point to the preset distance list corresponding to the core point; or, if there is no core point that is a preset distance away from the data point, adding the data point as a new core point to the preset core point list; and determining the preset number of data points based on the preset distance list.

[0074] In an embodiment of the present invention, before detecting each data point, all core points in the current slider data may be determined and added to a preset core point list (i.e., a core point list). The preset core point list may then be checked to see if there is a core point at a preset distance from the data point, where the preset distance is the product of a preset radius value (i.e., a neighbor radius R) and a preset ratio (e.g., 1 / 2, 1, 2 / 3, 2) (i.e., R / 2, R, R3 / 2, 2R). If there is a core point at a preset distance from the data point, the data point is associated with the preset distance list corresponding to the core point (i.e., if there is a core point at a distance of R / 2 from the data point, the data point is associated with the LR / 2 list; if there is a core point at a distance of R from the data point, the data point is associated with the LR list; if there is a core point at a distance of R3 / 2 from the data point, the data point is associated with the LR3 / 2 list; if there is a core point at a distance of 2R from the data point, the data point is associated with the L2R list). If there is no core point with a preset distance from the data point, the data point is added as a new core point to the preset core point list. The preset number of points for the data point can then be determined based on the preset distance list (i.e., the number of all data points in all preset distance lists associated with the data point is accumulated to obtain the preset number of points (i.e., the number of neighbors) for the data point).

[0075] For example, the triangle theorem shows that for any data point, its nearest neighbors within the R distance must be in the four lists of its associated core points (LR / 2, LR, LR3 / 2, L2R). Therefore, if a data point is within the R / 2 range of its core point, the data points in the core point's LR / 2 list must be its nearest neighbors in terms of R distance. Then, simply checking the distances between the data points in LR and LR3 / 2 will yield the number of its nearest neighbors. If a data point is within the R range of its core point, then checking the data points in the four lists of its associated core points (LR / 2, LR, LR3 / 2, L2R) will yield the number of its nearest neighbors. If a data point is within the 2R range of its core point, then checking the data points in the two lists of its associated core points (LR / 2, LR) will yield the number of its nearest neighbors.

[0076] Optionally, after determining the preset number of data points based on the preset distance list, the method further includes: when the number of slider data in the current processing window is greater than the preset window value, sorting the slider data in the current processing window according to the time when the slider data was added to the current processing window to obtain a sorting result; based on the sorting result, determining the target slider data that was added to the current processing window earliest; eliminating the target slider data; and when the preset number of any data point in the target slider data is less than the preset point threshold, determining that the data point is an abnormal point.

[0077] In an embodiment of the present invention, when the number of sliders currently being processed exceeds the window size W, the old sliders are eliminated. If any of the eliminated sliders still have points that do not meet the K nearest neighbors (i.e., the preset point threshold), they are considered to be outliers and an alarm is issued. Specifically, if the number of slider data in the current processing window is greater than the preset window value (i.e., the window size W), the slider data in the current processing window can be sorted based on the time when the slider data was added to the current processing window to obtain a sorting result. Then, based on the sorting result, the target slider data that was added to the current processing window the earliest is determined and the target slider data is eliminated. If the preset number of points of any data point in the target slider data is less than the preset point threshold (i.e., the nearest neighbor threshold K), the data point can be determined to be an outlier.

[0078] Step S104 : When the number of preset points corresponding to all data points is greater than or equal to the preset point threshold, it is determined that there are no abnormal points in the data set to be detected.

[0079] In an embodiment of the present invention, if the number of preset points corresponding to all data points is greater than or equal to the preset point threshold, it can be determined that there are no abnormal points in the data set to be detected (that is, only data points whose number of preset points is less than the preset point threshold are confirmed as abnormal points).

[0080] The following describes in detail another optional specific implementation.

[0081] The anomaly detection method in this embodiment can be applied in scenarios with low bandwidth requirements and low power consumption and with real-time data detection requirements, such as wind power base stations and intelligent driving systems.

[0082] Taking the wind power environment as an example, with the further development of renewable energy, the demand for wind power base stations is gradually increasing. It is obviously difficult to build a high-speed Internet network for all wind turbines. Generally, only 2G network bandwidth is provided. At the same time, it is not feasible to set up a dedicated high-power and high-performance real-time detection platform for each wind turbine for data analysis. This requires a tool that can effectively detect anomalies in the data generated by various sensors in the wind turbine in a low-power environment.

[0083] During operation, a wind power base station generates real-time data such as motor speed (r), motor temperature (T), and voltage (V). The time series data generated by a wind power base station at each time point is Tst = {rt, Tt, Vt}, and the time series data set of this wind power base station over a period of time (i.e., the data set to be detected) is Ts = {Ts1, Ts2, Ts3, ..., Tst}. Afterwards, the preset detection device can receive the data set to be detected collected by the sensor from the network through the second interface (such as receiving data collected by the sensor from the network through the Socket interface), or the second interface can be developed (such as secondary development of the Raspberry Pi's JTag interface) to directly read the data set to be detected from the sensor.

[0084] In this embodiment, the detection algorithm requires four parameters: the neighbor radius R, the window size W, the slider size S, and the neighbor threshold K. When real-time data generated by the power generation system enters the detection algorithm, it segments the received data set according to the configured slider size (S), adding a slider to the current processing window each time the detection algorithm is executed.

[0085] After the slider enters the current processing window, the detection algorithm determines the slider's core points before detecting each data point. It then determines whether the slider's core points exist within a distance of R / 2, R, R3 / 2, or 2R from the data point. If so, the data point is associated with the corresponding core point list (LR / 2, LR, LR3 / 2, L2R). If not, the data point is added as a new core point to the core point list. The triangle trilateral theorem states that for any data point, its nearest neighbors within a distance of R must be in the four associated core point lists (LR / 2, LR, LR3 / 2, L2R). Therefore, if a data point is within the R / 2 range of its core point, the data points in the core point LR / 2 list must be its nearest neighbors in terms of R distance. Then, it is only necessary to detect the distances of the data points in LR and LR3 / 2 to obtain the number of all the nearest neighbors of the data point. If a data point is within the R range of its core point, it is necessary to detect the data points in the four lists of its associated core points (LR / 2, LR, LR3 / 2, L2R) to obtain the number of all the nearest neighbors of the data point. If a data point is within the 2R range of its core point, it is necessary to detect the data points in the two lists (LR / 2, LR) associated with the remaining core points to obtain the number of the nearest neighbors of the data point.

[0086] When the number of currently processed sliders is greater than the window size (W), the old sliders will be eliminated. If there are still points among the eliminated sliders that do not meet the K nearest neighbors, they are considered as abnormal points and an alarm is issued.

[0087] In an embodiment of the present invention, a portable device (i.e., a preset detection device) is used to process the real-time data fed back from each sensor, and only the real-time status of the current wind power base station is returned to the control center through a low-bandwidth network. This ensures that effective information is returned while controlling energy consumption, providing effective assistance for the detection and maintenance of such base stations.

[0088] In an embodiment of the present invention, an anomaly detection algorithm (such as the CPOD algorithm, etc.) is combined with a small computing platform (i.e., a preset detection device), which solves the problem of poor timeliness in detecting abnormal data at the data source end, and realizes the movement of data anomaly detection from the high-performance computing end to the front-end platform device that generates data in real time, thereby improving the real-time detection of errors generated in the data, reducing dependence on low-latency networks and waste of communication channels, improving production efficiency, and reducing system overhead. In addition, the preset detection device that supports the ARM architecture has a high reuse rate. If there is further development of the anomaly detection algorithm in the future, it only needs to replace the algorithm module to achieve an upgrade of the system detection performance.

[0089] The following describes it in detail with reference to another embodiment.

[0090] Example 2

[0091] The abnormal data detection device provided in this embodiment includes multiple implementation units, each implementation unit corresponding to each implementation step in the above-mentioned embodiment 1.

[0092] Figure 4 is a schematic diagram of an optional abnormal data detection device according to an embodiment of the present invention, such as Figure 4 As shown, the detection device may include: a receiving unit 40, a collecting unit 41, a detecting unit 42, and a determining unit 43, wherein:

[0093] The receiving unit 40 is configured to receive parameter information, wherein the parameter information includes: operating parameters and configuration parameters, and the operating parameters include at least: a preset point threshold;

[0094] The collection unit 41 is used to collect the data set to be detected when the abnormality detection environment is configured based on the configuration parameters;

[0095] A detection unit 42 is configured to detect each data point in the test data set using a preset detection algorithm based on the operating parameters to obtain a detection result, wherein the preset detection algorithm is an algorithm running in a preset detection device based on an abnormality detection environment, and the detection result includes at least: the number of preset points corresponding to each data point;

[0096] The determining unit 43 is configured to determine that no abnormal points exist in the data set to be detected when the number of preset points corresponding to all data points is greater than or equal to a preset point threshold.

[0097] The above-mentioned detection device can receive parameter information through the receiving unit 40, and collect the data set to be detected by the collection unit 41 when the abnormality detection environment is configured based on the configuration parameters. The detection unit 42 uses a preset detection algorithm to detect each data point in the data set to be detected based on the operating parameters to obtain a detection result. The determination unit 43 determines that there are no abnormal points in the data set to be detected when the number of preset points corresponding to all data points is greater than or equal to the preset point threshold. In an embodiment of the present invention, after the abnormality detection environment is configured according to the received configuration parameters, the data set to be detected can be collected, and then, based on the operating parameters, a preset detection algorithm that can be run in the preset detection device can be used to detect each data point in the data set to be detected. If the number of preset points corresponding to all data points is greater than or equal to the preset point threshold, it can be determined that there are no abnormal points in the data set to be detected. The portable preset detection device can perform abnormal data detection in real time, effectively reducing the dependence on low-latency networks and the waste of communication channels, thereby solving the technical problem in the related art of poor timeliness of anomaly detection for real-time data in environments where high-performance computing platforms cannot be deployed.

[0098] Optionally, the preset detection device is deployed with a first interface and a second interface, wherein the first interface is used to receive parameter information transmitted by the preset platform, and the second interface is used to collect the data set to be detected, and the receiving unit includes: a first receiving module, used to receive parameter information transmitted by the preset platform based on the first interface, wherein the preset platform provides a parameter input interface and generates parameter information based on the input parameters, and the operating parameters in the parameter information also include: a preset radius value, a preset window value, and a preset slider value.

[0099] Optionally, the acquisition unit includes: a first opening module, used to open the second interface when the abnormality detection environment is configured; a first establishment module, used to establish a communication connection with a preset data source based on the second interface, wherein the preset data source is used to generate data to be detected; and a first acquisition module, used to collect the data set to be detected based on the communication connection.

[0100] Optionally, the first acquisition module includes: a first determination submodule, used to determine each time point in the time period to be detected; a first acquisition submodule, used to collect time series data generated by a preset data source at each time point based on a communication connection, and add the time series data to the data set to be detected, wherein each time series data is represented as a data point.

[0101] Optionally, the detection unit includes: a first segmentation module, used to segment the data set to be detected according to a preset slider value to obtain multiple slider data; a first adding module, used to add the slider data to the current processing window, wherein the window value of the current processing window is a preset window value; a first detection module, used to detect each data point in the slider data based on the current processing window using a preset detection algorithm.

[0102] Optionally, the first detection module includes: a second determination submodule, used to determine all core points in the slider data and add all core points to a preset core point list; a first detection submodule, used to detect whether there is a core point in the preset core point list that is a preset distance away from the data point, wherein the preset distance is the product value of a preset radius value and a preset ratio; a first association submodule, used to associate the data point to the preset distance list corresponding to the core point when there is a core point that is a preset distance away from the data point; a first addition submodule, used to add the data point as a new core point to the preset core point list when there is no core point that is a preset distance away from the data point; and a third determination submodule, used to determine the preset number of data points based on the preset distance list.

[0103] Optionally, the detection device also includes: a first sorting module, which is used to determine the preset number of data points based on a preset distance list, and when the number of slider data in the current processing window is greater than the preset window value, sort the slider data in the current processing window according to the time when the slider data was added to the current processing window to obtain a sorting result; a first determination module, which is used to determine the target slider data that was added to the current processing window earliest based on the sorting result; a first elimination module, which is used to eliminate the target slider data; and a second determination module, which is used to determine that the data point is an abnormal point when the preset number of any data point in the target slider data is less than the preset point threshold.

[0104] The above-mentioned detection device may also include a processor and a memory. The above-mentioned receiving unit 40, acquisition unit 41, detection unit 42, determination unit 43, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0105] The processor includes a kernel that retrieves the corresponding program unit from memory. One or more kernels can be configured, and kernel parameters are adjusted to determine that the detected dataset contains no outliers when the number of preset points corresponding to all data points is greater than or equal to a preset point threshold.

[0106] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0107] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program initialized with the following method steps: receiving parameter information, collecting a data set to be detected after completing the anomaly detection environment configuration based on the configuration parameters, and using a preset detection algorithm based on the operating parameters to detect each data point in the data set to be detected to obtain a detection result, and determining that there are no anomalies in the data set to be detected when the number of preset points corresponding to all data points is greater than or equal to a preset point threshold.

[0108] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored computer program, wherein when the computer program runs, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned abnormal data detection method.

[0109] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the above-mentioned method for detecting abnormal data.

[0110] Figure 5 FIG. 1 is a hardware structure diagram of an electronic device (or mobile device) for detecting abnormal data according to an embodiment of the present invention. Figure 5 As shown, the electronic device may include one or more (illustrated as 502a, 502b, ..., 502n in the figure) processors 502 (the processor 502 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 504 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 5 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components than shown, or with Figure 5 Different configurations shown.

[0111] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0112] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0114] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0115] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0117] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for detecting abnormal data, characterized in that: include: Receive parameter information, wherein the parameter information includes: operating parameters, configuration parameters, and the operating parameters at least include: a preset point threshold; When the anomaly detection environment is configured based on the configuration parameters, collecting a data set to be detected; Based on the operating parameters, a preset detection algorithm is used to detect each data point in the data set to be detected to obtain a detection result, wherein the preset detection algorithm is an algorithm running in a preset detection device based on the abnormality detection environment, and the detection result at least includes: the number of preset points corresponding to each data point; the preset detection device is deployed with a first interface and a second interface, the first interface is used to receive the parameter information transmitted by a preset platform, and the second interface is used to collect the data set to be detected, and receiving the parameter information includes: based on the first interface, receiving the parameter information transmitted by the preset platform, wherein the preset platform provides a parameter input interface and generates the parameter information based on the input parameters, and the operating parameters in the parameter information also include: a preset radius value, a preset window value, and a preset slider value; When the number of the preset points corresponding to all the data points is greater than or equal to the preset point threshold, determining that there are no abnormal points in the data set to be detected; Wherein, based on the operating parameters, each data point in the to-be-detected data set is detected using a preset detection algorithm, including: According to the preset slider value, the data set to be detected is divided to obtain a plurality of slider data; the slider data is added to a current processing window, wherein the window value of the current processing window is the preset window value; Based on the current processing window, each data point in the slider data is detected using the preset detection algorithm, including: determining all core points in the slider data and adding all core points to a preset core point list; detecting whether there is a core point in the preset core point list that is a preset distance away from the data point, wherein the preset distance is a product value of the preset radius value and a preset ratio; if there is a core point that is a preset distance away from the data point, associating the data point with the preset distance list corresponding to the core point; or, if there is no core point that is a preset distance away from the data point, adding the data point as a new core point to the preset core point list; and determining the preset number of the data points based on the preset distance list.

2. The detection method according to claim 1, wherein When the anomaly detection environment is configured based on the configuration parameters, the step of collecting the data set to be detected includes: When the anomaly detection environment is configured, opening the second interface; establishing a communication connection with a preset data source based on the second interface, wherein the preset data source is used to generate the data to be detected; Based on the communication connection, the data set to be detected is collected.

3. The detection method according to claim 2, characterized in that The step of collecting the data set to be detected based on the communication connection includes: Determine each time point in the time period to be detected; Based on the communication connection, the time series data generated by the preset data source at each time point is collected, and the time series data is added to the data set to be detected, wherein each time series data is represented as a data point.

4. The detection method according to claim 1, wherein After determining the preset number of data points based on the preset distance list, the method further includes: When the number of slider data in the current processing window is greater than the preset window value, sorting the slider data in the current processing window according to the time when the slider data was added to the current processing window to obtain a sorting result; Based on the sorting result, determining the target slider data to be added to the current processing window earliest; Eliminating the target slider data; When the preset point number of any data point in the target slider data is less than the preset point threshold, the data point is determined to be the abnormal point.

5. A device for detecting abnormal data, used to implement the method for detecting abnormal data according to claim 1, characterized in that: include: A receiving unit, configured to receive parameter information, wherein the parameter information includes: operating parameters, configuration parameters, and the operating parameters at least include: a preset point threshold; A collection unit, configured to collect a data set to be detected when the anomaly detection environment is configured based on the configuration parameters; a detection unit, configured to detect each data point in the to-be-detected data set using a preset detection algorithm based on the operating parameters, and obtain a detection result, wherein the preset detection algorithm is an algorithm running in the preset detection device based on the abnormality detection environment, and the detection result at least includes: the number of preset points corresponding to each data point; The determining unit is configured to determine that no abnormal points exist in the data set to be detected when the number of the preset points corresponding to all the data points is greater than or equal to the preset point threshold.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the abnormal data detection method according to any one of claims 1 to 4.

7. An electronic device, characterized in that: The device comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the abnormal data detection method according to any one of claims 1 to 4.

Images