Data Anomaly Detection System
Using PCA to generate a new coordinate system and apply sigma rules effectively addresses the inefficiencies of existing anomaly detection methods, ensuring accurate and reliable data processing.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- EBAY INC
- Filing Date
- 2025-04-24
- Publication Date
- 2026-07-24
Smart Images
Figure 0007894977000001 
Figure 0007894977000002 
Figure 0007894977000003
Abstract
Description
Technical Field
[0001] The subject matter disclosed herein generally relates to data anomaly detection. Specifically, the present disclosure addresses systems and methods for detecting data anomalies by generating a new coordinate system, transforming data into the new coordinate system, and applying sigma statistical rules.
Background Art
[0002] A data platform can have tens of thousands of pipeline jobs running daily. These jobs generate large amounts of data. This leads to the problem of how to efficiently monitor the data and accurately detect abnormal data or data anomalies. If data anomalies are not detected and corrected, they may cause inaccurate results in downstream processes or systems.
Brief Description of the Drawings
[0003] [Figure 1A] A diagram showing the 3-sigma rule for data anomaly detection. [Figure 1B] A diagram showing an exemplary plot of data anomalies that cannot be detected using the sigma rule. [Figure 2] A diagram showing an exemplary plot on the optimal coordinate system generated using principal component analysis. [Figure 3] A diagram showing an exemplary network environment suitable for detecting data anomalies using the sigma rule according to an exemplary embodiment. [Figure 4] A diagram showing the components of a data analysis system and an exemplary downstream system according to an exemplary embodiment. [Figure 5] A flowchart showing the processing of a method for detecting data anomalies using the sigma rule according to an exemplary embodiment. [Figure 6]This is a block diagram showing several components of a machine, as illustrated by some examples, that are capable of reading multiple instructions from a machine storage medium and performing any one or more of the methods described herein. [Modes for carrying out the invention]
[0004] The following description describes systems, methods, techniques, instruction sequences, and computing machine program products that illustrate multiple examples of this subject. The following description includes numerous specific details for illustrative purposes to provide an understanding of the various examples of this subject. However, it will be apparent to those skilled in the art that the various examples of this subject can be implemented without some or others of these specific details. The examples are merely representative of possible variations. Unless otherwise specified, the multiple structures (e.g., multiple structural components) are optional and may be combined or subdivided, and the multiple operations (e.g., in a procedure, algorithm, or other function) may have their sequences changed, combined, or subdivided.
[0005] The exemplary embodiment addresses the technical challenge of efficiently and accurately detecting anomalies in data generated on a data platform. Sigma statistical rules (also known as “sigma rules”) are a common technique for detecting anomalous data. In particular, the 3-sigma rule can be used to detect data anomalies, thereby defining normal data as being between μ-3σ and μ+3σ (e.g., the 3-sigma normal range), as shown in Figure 1A. Here, μ is the mean of the distribution and σ is its standard deviation. These two parameters are derived statistically and can be used to identify any outliers (e.g., data point 102) outside the 3-sigma normal range (indicated by the dotted line). While the exemplary embodiment is described herein with reference to the 3-sigma rule, other sigma rules may also be used. For example, instead of the 3-sigma rule, the 2-sigma rule, 4-sigma rule, 5-sigma rule, etc., can be used.
[0006] However, there are situations where the sigma rule cannot accurately detect data anomalies. In statistics, correlation is any statistical relationship between two random variables, usually referring to the degree to which a pair of variables are linearly related. Referring to Figure 1B, an exemplary plot is shown that illustrates a data anomaly that cannot be detected using the sigma rule. To a human observing Figure 1B, it is clear that data point A is an outlier, but the computer cannot detect the outlier using the known sigma rule. The plot shows the relationship between variable X and variable Y. For example, if the 3-sigma rule is used to detect X, the normal range is 7 to 9. For Y, the normal range is 0.9 to 1.15. Therefore, data point A is not considered an outlier because its X and Y values are between their respective normal ranges. However, the "actual" normal range is along the solid line 104.
[0007] To address the shortcomings of the 3-sigma rule or the general sigma rule, exemplary embodiments use principal component analysis (PCA) to generate a new optimal coordinate system. In one exemplary embodiment, the optimal coordinate system is generated based on maximizing the variance of data in two or more dimensions. In an alternative embodiment, the optimal coordinate system is generated based on linear regression on two-dimensional data. Referring now to Figure 2, an exemplary plot on the optimal coordinate system generated using principal component analysis is shown. Using new X and Y axes and applying the sigma rule (e.g., shown as a dashed line), point A is clearly outside the range of the sigma rule and is an outlier. While the examples in Figures 1A, 1B, and 2 show two-dimensional data points, the exemplary embodiments are also applicable to any multi-dimensional data points (e.g., three-dimensional data points). The process for generating the optimal coordinate system is described in more detail below.
[0008] Accordingly, the exemplary embodiments address the technical challenge of efficiently and accurately detecting data anomalies in large datasets generated on a data platform. Unlike typical PCA techniques, a new coordinate system is first constructed using PCA without reducing the number of dimensions. The data to be analyzed is then transformed into the new coordinate system. After transformation, sigma rules are applied to detect and output data anomalies. In some embodiments, data anomalies can be removed before the corrected data is sent to downstream systems for further operation. Thus, the exemplary embodiments provide a technical solution to improve computer functionality and operation by accurately detecting data anomalies and correcting these anomalies so as not to adversely affect downstream components, processing, and results.
[0009] Figure 3 shows an exemplary network environment 300 suitable for detecting data anomalies using sigma (statistical) rules, according to an exemplary embodiment. The network system 302 provides server-side functions to multiple user devices 306 via a communication network 304 (e.g., the Internet, a wireless network, a cellular network, or a wide area network (WAN)). The network system 302 can comprise any entity having a data platform that generates large amounts of data. For example, the network system 302 can be associated with a banking site, an e-commerce site, a travel-related site, a social networking site, and so on.
[0010] In various cases, user device 306 is a device associated with a user account of a user of network system 302. In some cases, user device 306 is a device of an individual who uses network system 302 to perform searches, transactions, or other processes, and therefore triggers the generation of data in network system 302. In other cases, user device 306 is a device associated with an individual who is an operator or administrator of network system 302, and uses that user device 306 to monitor and analyze data and / or correct (e.g., remove) data anomalies.
[0011] The user device 306 interfaces with the network system 302 via a connection to the network 304. Depending on the configuration of the user device 306, various types of connections and networks 304 can be used. For example, this connection may be a CDMA (Code Division Multiple Access) connection, a GSM (Global System for Mobile communications) (registered trademark) connection, or another type of cellular connection. Such a connection may implement any of various types of data transfer technologies, such as single-carrier radio transmission technology (1xRTT), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, or other data transfer technologies (e.g., 4th generation wireless, 4G network, 5G network). When such technologies are used, the network 304 includes a cellular network with multiple cell sites having overlapping geographic coverage, interconnected by cellular telephone exchanges. These cellular telephone exchanges are connected to a network backbone (for example, a public switched telephone network (PSTN), a packet-switched data network, or other types of networks).
[0012] In another example, the connection to network 304 is a WirelessFidelity (Wi-Fi, IEEE 802.11x type) connection, a WiMAX (Worldwide Interoperability for Microwave Access) connection, or another type of wireless data connection. In such an example, network 304 includes one or more wireless access points connected to a local area network (LAN), a wide area network (WAN), the Internet, or another packet-switched data network. In yet another example, the connection to network 304 is a wired connection (e.g., an Ethernet® link), and network 304 is a LAN, WAN, the Internet, or another packet-switched data network. Thus, a variety of different configurations are explicitly considered.
[0013] The user device 306 may include, but is not limited to, a smartphone, tablet, laptop, multiprocessor system, microprocessor-based or programmable consumer electronics, game console, set-top box, server, or any other communication device capable of accessing the network system 302. The user device 306 may include a display component (not shown) for displaying information (e.g., in the form of a user interface). The user device 306 may be operated by a human user and / or a machine user.
[0014] Referring specifically to network system 302, application programming interface (API) server 308 and web server 310 are coupled to one or more networking servers 312, providing a programmatic interface and a web interface, respectively. One or more networking servers 312 host various systems, including data processing system 314 and data analysis system 316, each comprising multiple components that can be embodied as hardware, software, firmware, or any combination thereof.
[0015] In particular, the data processing system 314 includes components that generate data in the network system 302. The data may include logistics data, financial data, social networking data, transaction data, or any other type of data, and may be structured or unstructured. For example, if the network system 302 is associated with a banking site, the data processing system 314 can generate data related to banking transactions, account referral operations, loan applications, etc. In another example, if the network system 302 is associated with a commercial transaction site, the data processing system 314 can generate data related to sales transactions, revenue information, the number of users performing various operations (e.g., search, add to wishlist, purchase, return), etc. The data may be generated by different components or sources.
[0016] The data analysis system 316 analyzes the data generated by the data processing system 314. Specifically, the data analysis system 316 aggregates and monitors the data generated by the data processing system 314 and detects any data anomalies. Although the data analysis system 316 is shown within the network system 302, in an alternative embodiment, the data analysis system 316 may be located outside the network system 302 but may be connected to communicate via network 304. The data analysis system 316 will be described in more detail with reference to Figure 4 below.
[0017] One or more networking servers 312 are connected to one or more database servers 318, which enable access to one or more storage repositories or data storage devices 320. The data storage device 320 is a storage device that stores user accounts, including user profiles, and data generated by the data processing system 314.
[0018] Any of the systems, servers, data storage devices, or devices shown in or associated with Figure 3 (collectively referred to as the “Multiple Components”) may be, include, or otherwise implement a special-purpose (e.g., special or otherwise non-general-purpose) computer that can be modified (e.g., composed of or programmed by software such as one or more software components from applications, operating systems, firmware, middleware, or other programs) to perform one or more of the functions described herein for that system or machine. For example, a special-purpose computer system capable of implementing any one or more of the methods described herein is described below with respect to Figure 6, and such a special-purpose computer is a means of performing any one or more of the methods described herein. Within the art of such special-purpose computers, a special-purpose computer modified by the structures described herein to perform the functions described herein is technically improved compared to other special-purpose computers that lack the structures described herein or are otherwise unable to perform the functions described herein. Accordingly, the special-purpose machines configured according to the systems and methods described herein offer improvements to the technology of similar special-purpose machines.
[0019] Furthermore, any two or more of the components shown in Figure 3 may be combined, and any multiple functions described herein for any single component may be subdivided among multiple components. Multiple functions of a particular system may, in alternative examples, be embodied in different systems. For example, any number of user devices 306 or data storage devices 320 may be embodied within the network environment 300. Although only a single network system 302 is shown, alternatively, two or more network systems 302 may be included (e.g., localized to a specific area).
[0020] Figure 4 shows components of a data analysis system 316 that communicates with a downstream system, according to an exemplary embodiment. The data analysis system 316 accesses data generated by the data processing system 314 and analyzes the data to detect one or more data anomalies. To enable these processes, the data analysis system 316 comprises a data access component 402, a coordinate component 404, an anomaly detection component 406, a notification component 408, a data correction component 410, and a downstream data component 412, all of which are configured to communicate with each other (e.g., via a bus, shared memory, or switch). The data analysis system 316 may include other components that are not necessary for the operation of the inspection system embodiment.
[0021] The data access component 402 is configured to access data generated by the data processing system 314. The data may be accessed periodically (e.g., nightly, once a week) when a certain amount of data is generated, and / or when triggered (e.g., by an administrator via the user device 306). The data access component 402 may retrieve data directly from the data processing system 314, from the data storage device 320, or a combination of both. The data accessed (e.g., aggregated) may come from different sources or components of the data processing system 314.
[0022] The coordinate component 404 is configured to generate a new optimal coordinate system and transform the data into the new coordinate system. An exemplary embodiment uses principal component analysis (PCA) to generate a new coordinate system without reducing the number of dimensions. PCA is generally used to reduce the number of dimensions of data. However, the coordinate component 404 sets a strict condition that it constructs a new coordinate system using PCA but does not reduce the number of dimensions of the data. Assume that the data has m rows and each row of the data has n features. The input data is a matrix, m is the number of observations, and n is the number of dimensions for each observation. Thus, if the number of features of the input data is n, the output of PCA is also n. Referring back to FIG. 2, if PCA is used to reduce the number of dimensions of the data, PCA picks up the values on the X-axis and ignores the values on the Y-axis. However, the outlier (data point A) is on the Y-axis. Thus, using PCA, the coordinate component 404 ensures that the number of input dimensions and the number of output dimensions remain the same.
[0023] In one embodiment, the generation of the new optimal coordinate system is based on maximizing the variance. To find a new X-axis, the coordinate component 404 assumes that X is a vector having three unknown variables (u, v, w). Next, the coordinate component 404 maps all the data points onto X. The values on X and the variance on X are determined. The variance is an equation having the unknown variable X. For example, if the equation is -3X 2 +4X - 5, then X is max(-3X 2 +4X - 5). As a result, a straight line with the maximum variance that becomes the X-axis is obtained.
[0024] Using a similar approach, the Y-axis and Z-axis can be determined. For example, when the X-axis is determined, the coordinate component 404 can identify a plane perpendicular to the X-axis. The coordinate component 404 can map the data points onto this plane and maximize the variance to construct the Y-axis. When the Y-axis is determined, the Z-axis can be determined in the same way.
[0025] While exemplary embodiments are described above based on maximizing variance, alternative embodiments may use linear regression for two-dimensional data. Linear regression attempts to obtain a linear function such as y = ax + b by determining the values of parameters a and b. The values of a and b are determined by minimizing the loss. For example, the loss for a particular observation or point is the distance along the Y-axis. The smaller the loss, the better the function fits. Linear regression obtains the sum of the losses for all observations. Thus, the values of a and b are obtained by minimizing the sum of the losses for all data points in the data. A new X-axis is determined by determining the linear function. A new Y-axis perpendicular to the new X-axis is then identified.
[0026] When a new coordinate system is generated by coordinate component 404, the coordinate component transforms the data into the new coordinate system. This is done without reducing the number of dimensions. The anomaly detection component 406 detects data anomalies in the transformed data on the new coordinate system. In an exemplary embodiment, the anomaly detection component 406 applies a sigma rule to the transformed data on the new coordinate system. In one embodiment, the sigma rule is a 3-sigma rule. Any data point outside the range of the sigma rule is considered a data anomaly.
[0027] The notification component 408 is configured to provide notifications or alerts regarding detected data anomalies. In an exemplary embodiment, the notification component 408 generates a report or other form of alert indicating the data anomaly. The notification component 408 then sends the report or alert to the appropriate system or individual. For example, the notification component 408 can send an email to the administrator's user device 306 or trigger an alert to appear on the monitoring user interface of a device associated with the administrator.
[0028] The data correction component 410 is configured to remove or correct data anomalies detected by the anomaly detection component 406. In one embodiment, the data correction component 410 automatically removes data anomalies to generate corrected data. Automatic removal may be performed based on a set of rules indicating the circumstances under which data can be automatically removed and the circumstances under which the data needs to be reviewed before removal. In another embodiment, the data correction component 410 flags data anomalies, and a human or machine user can review the flagged data anomalies and trigger their removal.
[0029] Once data anomalies are removed and the data is corrected, the downstream data component 412 can send the corrected data to the downstream component for further processing or operation. For example, the downstream data component 412 can send the data to the machine learning system 414.
[0030] In embodiments where the data correction component 410 flags data anomalies but does not correct them, the downstream data component 412 may send the uncorrected data containing the data anomalies to a downstream component capable of correcting the uncorrected data (e.g., removing the data anomalies). For example, instead of sending notifications generated by the notification component 408, or in addition to sending notifications, the downstream data component 412 may send the uncorrected data flagged by the data correction component 410 to an external component of the data analysis system 316 (e.g., a component similar to the data correction component 410) for correction, or send the uncorrected data to a user (e.g., an administrator), a machine, or a machine learning system for pre-correction verification.
[0031] In embodiments where the corrected data is sent to the machine learning system 414, the machine learning system 414 is configured to train one or more machine learning (ML) models to determine probabilities for a particular task. The machine learning system 414 also refines the ML models by further retraining them with the corrected (training) data. The machine learning system 414 can then apply new data, which may be new corrected data from the data analysis system 316, to the trained ML models to obtain results. Thus, the machine learning system 414 includes a training component 416 and an evaluation component 418.
[0032] In some embodiments, the training component 416 trains one or more ML models using the corrected data. Since the corrected data has had data anomalies removed, the training results in more accurate ML models. Machine learning can be performed using artificial intelligence such as neural networks, and the training of one or more ML models may include training in probability estimation.
[0033] During runtime or inference, the evaluation component 418 of the machine learning system 414 may be configured to determine probabilities or other outcomes using a trained ML model. In some cases, the corrected data is the data to be evaluated by the evaluation component 418. As with training, removing data anomalies before evaluation by the evaluation component 418 yields more accurate results.
[0034] In the recommended embodiment, the trained ML model from the machine learning system 414 is the recommendation model. In these cases, the evaluation component 418 uses the recommendation model to generate recommendation information.
[0035] In some embodiments, notifications, uncorrected data, and / or corrected data generated by the notification component 408 are sent to the anomaly analysis system 420 (for example, by the downstream data component 412). The anomaly analysis system 420 performs an analysis of the data anomalies. For example, the anomaly analysis system 420 may attempt to identify what caused each data anomaly. For example, the anomaly analysis system 420 may attempt to identify a component or system that may have caused the anomaly. In other cases, the anomaly analysis system 420 may attempt to correlate the anomaly with trends, current events, or other data in order to identify the cause of the anomaly, or the relationship between the anomaly and trends, current events, and other data.
[0036] Figure 5 is a flowchart illustrating the process of Method 500 for detecting data anomalies using sigma rules, according to an exemplary embodiment. Each process in Method 500 may be performed by the data analysis system 316 using the components described above with respect to Figure 4. Thus, Method 500 is described exemplary with reference to the data analysis system 316. However, it should be noted that at least some of the processes of Method 500 may be deployed on various other hardware configurations or performed by similar components located elsewhere in the network environment 300. Thus, Method 500 is not intended to be limited to the data analysis system 316.
[0037] In process 502, the data access component 402 accesses data generated by the data platform (e.g., the data processing system 314). The data may be accessed periodically, when a certain amount of data is generated, or when triggered by a user. The data access component 402 may aggregate / collect data directly from the data processing system 314, from the data storage device 320, or a combination of both. In exemplary embodiments, the data access component 402 may be configured or instructed to access, for example, specific types of data, data within a specific date range, and / or data generated by specific components(s) for analysis. The data may come from any source associated with the data processing system 314.
[0038] In process 504, the coordinate component 404 generates a new optimal coordinate system. In an exemplary embodiment, the coordinate component 404 uses PCA to generate the optimal coordinate system without reducing the number of data dimensions. In one embodiment, the generation of the optimal coordinate system is based on maximizing the variance. To find a new X-axis, the coordinate component 404 maps all data points onto X. The values on X and the variance on X are identified. The variance is then maximized. Once the X-axis is determined, the coordinate component 404 identifies a plane perpendicular to the X-axis. Next, the coordinate component 404 maps the data points onto this plane and constructs the Y-axis by maximizing the variance. If necessary, the Z-axis can be found using a similar process.
[0039] In process 506, the coordinate component 604 transforms the data into the optimal coordinate system. This is done without reducing the number of dimensions of the data. In process 508, the anomaly detection component 406 detects anomalies by applying a sigma rule to the transformed data on the optimal coordinate system. The sigma rule provides a normal range for the data. Any data point outside the normal range is considered an anomaly. In one embodiment, the sigma rule applied to the transformed data is a 3-sigma rule.
[0040] In process 510, the notification component 408 provides a notification or alert regarding the detected data anomaly. In an exemplary embodiment, the notification component 408 generates a report or other form of notification or alert indicating the data anomaly. The notification component 408 then sends the notification to the appropriate system or displays the notification on the appropriate system. In some cases, the notification may be sent to the anomaly analysis system 420 along with the corresponding or corrected data. The anomaly analysis system 420 may then attempt to identify the cause of each data anomaly, or attempt to identify the cause or relationship by correlating the anomaly with trends, current events, or other data.
[0041] In process 512, the data correction component 410 removes data anomalies detected by the anomaly detection component 406. In one embodiment, the data correction component 410 automatically removes data anomalies to generate updated data. Automatic removal may be performed based on a set of rules indicating the circumstances under which data can be automatically removed and the circumstances under which data needs to be reviewed before removal. In some cases, the data correction component 410 flags data anomalies, and a human or machine user triggers the removal of the flagged data after review.
[0042] Please note that processes 510 and 512 may be optional. Furthermore, process 512 may be executed before process 510. Figure 6 shows several components of a machine 600 capable of reading multiple instructions from a mechanical storage medium (e.g., a mechanical storage device, a non-temporary mechanical storage medium, a computer storage medium, or any preferred combination thereof) in several exemplary embodiments, and executing any one or more of the methods described herein. Specifically, Figure 6 shows a schematic diagram of a machine 600 in an exemplary form of a computer device (e.g., a computer), in which multiple instructions 624 (e.g., software, a program, an application, an applet, an app, or other executable code) causing the machine 600 to execute any one or more of the methods described herein can be executed in whole or in part.
[0043] For example, multiple instructions 624 can cause machine 600 to execute the flowchart in Figure 5. In a particular embodiment, multiple instructions 624 can transform machine 600 into a specific machine (e.g., a specially configured machine) programmed to perform the described and illustrated functions in the described manner.
[0044] In alternative embodiments, machine 600 may operate as a standalone device or be connected to other machines (e.g., networked). In a networked configuration, machine 600 may operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 600 may be a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smartphone, a web appliance, a network router, a network switch, a network bridge, a compute beacon, or any machine capable of executing (sequentially or otherwise) multiple instructions 624 that specify multiple operations to be performed by the machine. Furthermore, although only a single machine is shown, the term “machine” should also be interpreted to include a collection of machines that individually or collectively execute multiple instructions 624 to perform any one or more of the methods described herein.
[0045] The machine 600 includes one of the processors 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any suitable combination thereof), main memory 604, and static memory 606, which are configured to communicate with each other via bus 608. The processor 602 may include microcircuits that can be temporarily or permanently configured by some or all of the instructions 624, so that the processor 602 can be configured to perform any one or more of the methods described herein, in whole or in part. For example, one or more microcircuits of a set of the processor 602 may be configured to perform one or more modules (e.g., software modules) described herein.
[0046] In some embodiments, the machine 600 may further include a graphics display 610 (e.g., a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT), or any other display capable of displaying graphics or moving images). The machine 600 may also include an input device 612 (e.g., a keyboard), a cursor control device 614 (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing device), a storage unit 616, a signal generation device 618 (e.g., a sound card, amplifier, speaker, headphone jack, or any suitable combination thereof), and a network interface device 620.
[0047] The storage unit 616 includes a mechanical storage medium 622 (e.g., a tangible mechanical storage medium) storing a plurality of instructions 624 (e.g., software) that embody any one or more of the methods or functions described herein. The plurality of instructions 624 may also be entirely or at least partially present in the main memory 604, in the processor 602 (e.g., in the processor's cache memory), or both, before or during the execution of those instructions by the machine 600. Thus, the main memory 604 and the processor 602 can be considered as mechanical storage media (e.g., tangible and non-temporary mechanical storage media). The plurality of instructions 624 may be transmitted over the network 626 via the network interface device 620.
[0048] In some exemplary embodiments, the machine 600 may be a portable computing device and may have one or more additional input components (e.g., sensors or gauges). Examples of such input components include image input components (e.g., one or more cameras), audio input components (e.g., microphones), direction input components (e.g., compass), position input components (e.g., GPS (global positioning system) receivers), orientation components (e.g., gyroscopes), motion detection components (e.g., one or more accelerometers), altitude detection components (e.g., altimeters), and gas detection components (e.g., gas sensors). Multiple inputs collected by any one or more of these input components may be accessible and available for use by any of the components described herein.
[0049] Executable instructions and machine storage media Various memories (e.g., 604, 606, and / or memories of one or more processors 602) and / or storage units 616 may store one or more sets of instructions and data structures (e.g., software) 624 that embody or utilize any one or more of the methods or functions described herein. When these instructions are executed by one or more processors 602, they cause various operations to be performed for carrying out the disclosed embodiments.
[0050] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” (collectively referred to as “machine storage medium 622”) have the same meaning and may be used interchangeably in this disclosure. These terms refer to one or more storage devices and / or media that store multiple executable instructions and / or data (e.g., centralized or distributed databases, and / or associated caches and servers), as well as cloud-based storage systems or storage networks that include multiple storage devices or devices. Accordingly, these terms shall be construed as including, but not limited to, solid-state memory, including internal or external memory of a processor, as well as optical and magnetic media. Specific examples of machine storage medium, computer storage medium, and / or device storage medium 622 include, for example, non-volatile memory including semiconductor memory devices such as EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), FPGAs, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The terms one or more “mechanical storage media,” one or more “computer storage media,” and one or more “device storage media” 622 specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term “signaling media” as set forth below. In this context, mechanical storage media are non-transient.
[0051] signal medium The terms “signaling medium” or “transmission medium” shall be interpreted to include any form such as modulated data signals, carrier waves, etc. The term “modulated data signal” means a signal in which one or more of its properties are set or modified in order to encode the information contained in the signal.
[0052] Computer-readable media The terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” mean the same thing and may be used interchangeably in this disclosure. These terms are defined to include both machine storage media (one or more) and signaling media (one or more). Accordingly, these terms include both storage devices / media and carrier / modulated data signals.
[0053] Multiple instructions 624 may further be transmitted or received over a communication network 626 using a transmission medium via a network interface device 620 and utilizing one of several well-known transfer protocols (e.g., HTTP). Multiple examples of communication networks 626 include local area networks (LANs), wide area networks (WANs), the Internet, mobile phone networks, POTS (plain old telephone service) networks, and wireless data networks (e.g., Wi-Fi, LTE, and WiMAX networks). The term “transmission medium” shall be interpreted as including any intangible medium capable of storing, encoding, or carrying multiple instructions 624 for execution by machine 600, and including digital or analog communication signals or other intangible medium for enabling communication of such software.
[0054] Throughout this specification, multiple instances may implement a component, operation, or structure described as a single instance. While individual operations of one or more methods are illustrated and described as separate operations, one or more of these operations may occur simultaneously, and it is not required that the operations occur in the order in which they are illustrated. Multiple structures and functions presented as separate components in exemplary configurations may be implemented as a combined structure or component. Similarly, structures and functions presented as a single component may be implemented as multiple separate components. These and other variations, modifications, additions, and improvements are included within the scope of the subject matter of this specification.
[0055] A “component” refers to a device, physical entity, or logic whose boundaries are defined by other technologies that provide, for example, function or subroutine calls, branching points, application programming interface APIs, or partitioning or modularization of specific processing or control functions. Multiple components can be combined through interfaces with other components to perform machine processing. A component is a packaged, functional hardware unit designed to be used together with other components, and is often part of a program that performs a specific function among several related functions. Multiple components can constitute either a software component (e.g., code embodied in a machine-readable medium) or a hardware component.
[0056] A “hardware component” is a tangible unit capable of performing a specific operation and can be configured or arranged in a specific physical manner. In various exemplary embodiments, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as hardware components that operate to perform a specific operation as described herein.
[0057] In some embodiments, hardware components may be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may comprise a dedicated circuit or logic permanently configured to perform a specific operation. For example, a hardware component may be a special-purpose processor such as an FPGA (field programmable gate array) or ASIC. A hardware component may comprise programmable logic or circuit configured by software to perform a specific operation temporarily. For example, a hardware component may comprise software contained within a general-purpose processor or other programmable processor. When configured by such software, the hardware component is no longer a general-purpose processor, as it becomes a specific machine (or a specific component of a machine) independently tuned to perform the configured function. It will be understood that the decision to implement a hardware component mechanically, in a dedicated and permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be made based on cost and time considerations.
[0058] Therefore, the term “hardware component” should be understood to encompass tangible entities that are physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) in order to operate in a particular way or to perform a particular operation as described herein. Considering multiple examples of hardware components being temporarily configured (e.g., programmed), each of the multiple hardware components does not need to be configured or instantiated in any single instance. For example, if a hardware component consists of a general-purpose processor configured by software to become a special-purpose processor, the general-purpose processor may be configured as a different special-purpose processor (e.g., with different hardware components) at different times. Thus, the software may, for example, configure the processor to constitute a specific hardware component in one instance at one time, while composing a different hardware component in a different instance at another time.
[0059] Hardware components can provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled. When multiple hardware components exist simultaneously, communication can be achieved by signal transmission between or within two or more of the hardware components (e.g., via appropriate circuits and buses). In multiple examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components may be achieved, for example, through the storage and retrieval of information in memory structures accessed by the multiple hardware components. For example, one hardware component can not only perform an operation but also store the output of that operation in a communicatively coupled memory device. Another hardware component can then retrieve and process the stored output by subsequently accessing the memory device. Multiple hardware components can also initiate communication with input or output devices, or they can operate on resources (e.g., collections of information).
[0060] Various operations of the exemplary methods described herein may be performed, at least partially, by one or more processors that are temporarily (e.g., by software) or permanently configured to perform the operations in question. Whether temporarily or permanently configured, such processors may constitute a processor implementation component that operates to perform one or more operations or functions described herein. As used herein, “processor implementation component” means a hardware component implemented using one or more processors.
[0061] Similarly, some of the methods described herein may be at least partially processor-implemented, and a processor is an example of hardware. For example, at least some of the operations of a method may be performed by one or more processors or processor implementation components. Furthermore, one or more processors may operate in a “cloud computing” environment or as “software as a service” (SaaS) to support the execution of the operations in question. For example, at least some of the operations may be performed by a group of computers (as an example of a machine containing multiple processors), and these operations may be accessible over a network (e.g., the Internet) and over one or more suitable interfaces (e.g., an API (application program interface)).
[0062] The specific performance characteristics of multiple operations may reside not only within a single machine, but may also be distributed among one or more processors deployed across multiple machines. In some exemplary embodiments, one or more processors or processor implementation components may be located in a single geographical location (e.g., a home environment, an office environment, or a server farm). In other exemplary embodiments, one or more processors or processor implementation components may be distributed across multiple geographical locations.
[0063] Multiple examples Example 1 is a method for detecting data anomalies. The method includes the steps of: a server accessing data generated on a data platform; the server detecting one or more data anomalies in the accessed data, which include using principal component analysis (PCA) to generate an optimal coordinate system without reducing the number of dimensions, transforming the data into the optimal coordinate system without reducing the number of dimensions, and applying sigma rules to the transformed data in the optimal coordinate system; and the server generating and transmitting notifications of one or more data anomalies.
[0064] In Example 2, the subject of Example 1 may optionally include the fact that the sigma rule includes the 3-sigma rule. In Example 3, any subject from Examples 1 or 2 may optionally include the sigma rule being one of the following: a 2-sigma rule, a 4-sigma rule, or a 5-sigma rule.
[0065] In Example 4, the subject matter of any of Examples 1-3 includes the step of generating and transmitting a notification, which includes providing an anomaly analysis system with a display of one or more data anomalies, and the anomaly analysis system may optionally include performing further analysis of one or more data anomalies.
[0066] In Example 5, any subject from Examples 1-4 may optionally include a step of automatically removing one or more data anomalies to generate corrected data. In Example 6, any subject from Examples 1-5 may optionally include a step of training a machine learning model using corrected data from which one or more data anomalies have been automatically removed.
[0067] In Example 7, any subject from Examples 1-6 may optionally include a step of training a machine learning model that includes training a recommendation model to provide recommendation information.
[0068] In Example 8, any of the themes from Examples 1-7 may optionally include the fact that generating an optimal coordinate system is done on the basis of maximizing variance. In Example 9, any of the themes from Examples 1-8 may optionally include the fact that the generation of the optimal coordinate system is performed based on linear regression.
[0069] Example 10 is a system for detecting data anomalies. The system comprises one or more processors and a memory for storing instructions, and when executed by one or more processors, the instructions cause one or more processors to perform a process which includes the steps of: accessing data generated on a data platform; detecting one or more data anomalies in the accessed data, the detection of one or more data anomalies which includes generating an optimal coordinate system without reducing the number of dimensions using principal component analysis (PCA); transforming the data into the optimal coordinate system without reducing the number of dimensions; applying sigma rules to the transformed data on the optimal coordinate system; and generating and transmitting notifications of one or more data anomalies.
[0070] In Example 11, the subject of Example 10 may optionally include the fact that the sigma rule has a 3-sigma rule. In Example 12, any subject from Examples 10-11 may optionally include the sigma rule having one of the following: a 2-sigma rule, a 4-sigma rule, or a 5-sigma rule.
[0071] In Example 13, the subject matter of any of Examples 10-12 may optionally include the step of generating and transmitting a notification, which provides an anomaly analysis system with a display of one or more data anomalies, and which may optionally include the anomaly analysis system performing further analysis of one or more data anomalies.
[0072] In Example 14, any subject from Examples 10-13 may optionally include a further step in which the process automatically removes one or more data anomalies to generate corrected data.
[0073] In Example 15, any subject from Examples 10–14 may optionally include a further step in which the process trains a machine learning model using corrected data from which one or more data anomalies have been automatically removed.
[0074] In Example 16, any subject from Examples 10–15 may optionally include a step of training a machine learning model that includes training a recommendation model to provide recommendation information.
[0075] In Example 17, any of the themes from Examples 10–16 may optionally include the fact that generating an optimal coordinate system is done based on maximizing variance. In Example 18, any of the themes from Examples 10-17 may optionally include the fact that the generation of the optimal coordinate system is based on linear regression.
[0076] Example 19 is a computer storage medium that includes instructions causing a machine to perform a process for detecting data anomalies when executed by one or more processors of the machine. The process includes the steps of: accessing data generated on a data platform; detecting one or more data anomalies in the accessed data, which include detecting one or more data anomalies using principal component analysis (PCA) to generate an optimal coordinate system without reducing the number of dimensions, transforming the data into the optimal coordinate system without reducing the number of dimensions, and applying sigma rules to the transformed data on the optimal coordinate system; and generating and transmitting notifications of one or more data anomalies.
[0077] In Example 20, the subject of Example 19 may optionally include the fact that the sigma rule has a 3-sigma rule. Some parts of this specification are presented in terms of algorithms or symbolic representations of multiple operations on data stored as bits or binary digital signals in machine memory (e.g., computer memory). These algorithms or symbolic representations are examples of techniques used by those skilled in the field of data processing to communicate the content of their work to others skilled in the field. As used herein, “algorithm” is a self-consistent sequence of multiple actions or similar operations to achieve a desired result. In this context, algorithms and operations include the physical operation of physical quantities. Usually, but not always, such quantities take the form of electrical, magnetic, or optical signals that can be stored, accessed, transferred, combined, compared, or otherwise operated (operated) by a machine. It is sometimes convenient, mainly for general use reasons, to refer to such signals using words such as “data,” “content,” “bit,” “value,” “element,” “symbol,” “character,” “term,” “number,” and “digit.” However, these terms are merely convenient labels and should be associated with the appropriate physical quantities.
[0078] Unless otherwise specified, descriptions herein using terms such as “processing,” “computing,” “calculating,” “determining,” “presenting,” and “displaying” refer to the operation or process of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities in one or more memories (e.g., volatile memory, non-volatile memory, or any preferred combination thereof), registers, or other machine components that receive, store, transmit, or display information. Furthermore, unless otherwise specified, the articles “a” or “an” are used herein to include one or more cases, as is common in multiple patent documents. Finally, where used herein, the conjunction “or” refers to a non-exclusive “or” unless otherwise specified.
[0079] While the outline of this subject matter has been described with reference to specific examples, various modifications and changes can be made to these examples without departing from the broader scope of the multiple examples of the invention. For example, the various examples or their features may be mixed and adapted or arbitrarily selected by those skilled in the art. Such examples of this subject matter may, for convenience only, be referred to individually or collectively as “inventions” in this specification, but where two or more are actually disclosed, the scope of this application is not intended to be arbitrarily limited to any single invention or concept.
[0080] The various examples provided herein are considered to be described in sufficient detail to enable those skilled in the art to implement the disclosed teachings. Other examples may be used and derived therefrom so that structural and logical substitutions and modifications may be made without departing from the scope of this disclosure. Accordingly, “modes for carrying out the invention” should not be interpreted in a restrictive sense, and the scope of the various examples is defined only by the appended claims, along with the entire scope of equivalents to which such claims are entitled.
[0081] Furthermore, multiple examples may be provided for multiple resources, multiple operations, or multiple structures described herein as a single example. In addition, the boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary, and certain operations are shown in the context of a particular exemplary configuration. Other assignments of functions are conceivable and may be included within the scope of various examples of the invention. In general, multiple structures and functions presented as separate resources in an exemplary configuration may be implemented as a combined set of structures or resources. Similarly, multiple structures and functionalities presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of the examples of the invention represented by the appended claims. Accordingly, this specification and the drawings should be considered illustrative rather than restrictive.
Claims
1. A method for improving computer detection of data anomalies that cannot be detected when a sigma rule is applied to unconverted data, The server's data analysis system accesses data, including the unconverted data, generated on the data platform. Based on the accessed data, the coordinate component of the data analysis system generates an optimal coordinate system using principal component analysis (PCA) without reducing the number of dimensions of the accessed data. The coordinate component performs the steps of transforming the accessed data onto the optimal coordinate system without reducing the number of dimensions of the accessed data, and obtaining a new set of coordinates for the transformed data. The steps include: applying the sigma rule to the transformed data on the optimal coordinate system to detect one or more data anomalies; A method comprising the step of the server generating and transmitting one or more notifications of data anomalies.
2. The method according to claim 1, wherein the sigma rule includes a 3-sigma rule.
3. The method according to claim 1, wherein the sigma rule includes one of the two-sigma rule, the four-sigma rule, or the five-sigma rule.
4. The method according to claim 1, wherein the step of generating and transmitting the notification includes providing an anomaly analysis system with a display of the one or more data anomalies, the anomaly analysis system performs further analysis on the one or more data anomalies.
5. The method according to claim 1, further comprising the step of automatically removing one or more of the aforementioned data anomalies to generate corrected data.
6. The method according to claim 5, further comprising the step of training a machine learning model using the corrected data from which one or more data anomalies have been automatically removed.
7. The method according to claim 6, wherein the step of training the machine learning model includes training it to become a recommendation model for providing recommendation information.
8. The method according to claim 1, wherein the generation of the optimal coordinate system is performed based on maximizing the variance.
9. The method according to claim 1, wherein the generation of the optimal coordinate system is performed based on linear regression.
10. A system for improving computer detection of data anomalies that cannot be detected when a sigma rule is applied to unconverted data, One or more processors, A memory for storing instructions, wherein when the instructions are executed by the one or more processors, the one or more processors, A step of accessing data including the unconverted data generated on the data platform, The steps include generating an optimal coordinate system using principal component analysis (PCA) based on the accessed data without reducing the number of dimensions of the accessed data, The steps include: transforming the accessed data onto the optimal coordinate system without reducing the number of dimensions of the accessed data, and obtaining a new set of coordinates for the transformed data; The steps include: applying the sigma rule to the transformed data on the optimal coordinate system to detect one or more data anomalies; A system that performs a process including the step of generating and sending notifications of one or more data anomalies.
11. The system according to claim 10, wherein the sigma rule includes a 3-sigma rule.
12. The system according to claim 10, wherein the sigma rule includes one of the two-sigma rule, the four-sigma rule, or the five-sigma rule.
13. The system according to claim 10, wherein the step of generating and transmitting the notification includes providing the display of the one or more data anomalies to an anomaly analysis system, the anomaly analysis system performing further analysis on the one or more data anomalies.
14. The aforementioned process is, The system according to claim 10, further comprising the step of automatically removing one or more of the aforementioned data anomalies and generating corrected data.
15. The aforementioned process is, The system according to claim 14, further comprising the step of training a machine learning model using the corrected data from which one or more data anomalies have been automatically removed.
16. The system according to claim 15, wherein the step of training the machine learning model includes training it to become a recommendation model for providing recommendation information.
17. The system according to claim 10, wherein the generation of the optimal coordinate system is performed on the basis of maximizing the variance.
18. The system according to claim 10, wherein the generation of the optimal coordinate system is performed based on linear regression.
19. A machine storage medium containing instructions, wherein, when executed by one or more processors of the machine, the instructions cause the machine to perform a process to improve computer detection of data anomalies that cannot be detected by applying sigma rules to unconverted data, and the process is: A step of accessing data including the unconverted data generated on the data platform, The steps include generating an optimal coordinate system using principal component analysis (PCA) based on the accessed data without reducing the number of dimensions of the accessed data, The steps include: transforming the accessed data onto the optimal coordinate system without reducing the number of dimensions of the accessed data, and obtaining a new set of coordinates for the transformed data; The steps include: applying the sigma rule to the transformed data on the optimal coordinate system to detect one or more data anomalies; A machine storage medium comprising the step of generating and transmitting a notification of one or more of the aforementioned data anomalies.
20. The mechanical storage medium according to claim 19, wherein the sigma rule includes a 3-sigma rule.