Data anomaly detection early warning processing method and device and terminal

Abstract

The invention discloses a data anomaly detection early warning processing method and device and a terminal. The method comprises the steps of collecting data from a plurality of data sources in real time; preprocessing the collected data to obtain preprocessed data; analyzing the preprocessed data in real time by using a stream processing technology, capturing a dynamic mode of data change, and performing anomaly detection on the data by using multiple built-in anomaly detection algorithms; and when it is detected that there is data deviating from the conventional data point, it is judged that there is abnormal data, and real-time early warning prompt is performed on the abnormal data according to a preset early warning threshold and rule. The invention provides an abnormal data detection and intelligent early warning method, which can automatically detect abnormal data in a data set and give an early warning when abnormal data is detected.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data anomaly detection and early warning processing method, device, intelligent terminal and storage medium. Background Art

[0002] In business operations, the importance of data is reflected in decision support, efficiency improvement, customer insight, risk management, innovation drive, competitive advantage and performance evaluation. With the development of technologies such as big data and artificial intelligence, the application of data will become more extensive and in-depth. Enterprises must attach importance to the collection, analysis and application of data to maintain competitiveness and sustainable development in a rapidly changing market environment.

[0003] In the process of enterprise operation, data anomalies are like hidden reefs, which may cause undercurrents in operation at any time. However, there is no good way to detect and discover abnormal data in the data set in the existing technology, nor can abnormal reminders be issued in time.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a data anomaly detection and early warning processing method, device, intelligent terminal and storage medium in response to the problems and defects of the above-mentioned prior art. The present invention provides an abnormal data detection and intelligent early warning method, which can automatically detect abnormal data in a data set and issue an early warning when an abnormality is detected.

[0006] The technical solution adopted by the present invention to solve the problem is as follows:

[0007] A data anomaly detection and early warning processing method, comprising:

[0008] Collect data from multiple data sources in real time;

[0009] Preprocessing the collected data to obtain preprocessed data;

[0010] Use stream processing technology to analyze pre-processed data in real time, capture dynamic patterns of data changes, and use built-in anomaly detection algorithms to detect anomalies in data;

[0011] When data points that deviate from normal data are detected, they are judged to be abnormal data, and real-time warning prompts are issued for the abnormal data based on preset warning thresholds and rules.

[0012] The data anomaly detection and early warning processing method, wherein the step of collecting data from multiple data sources in real time includes:

[0013] Pre-set warning thresholds and rules to detect abnormal data and trigger warnings.

[0014] The data anomaly detection and early warning processing method, wherein the multiple anomaly detection algorithms used to perform anomaly detection on data using multiple built-in anomaly detection algorithms include:

[0015] Use statistical methods, using standard deviation and box plots based on statistical principles, to identify outliers in the data;

[0016] Adopt machine learning methods, apply support vector machine and random forest of machine learning models to perform data anomaly detection;

[0017] Adopt deep learning methods and use autoencoders of deep neural networks to identify complex abnormal patterns.

[0018] The data anomaly detection and early warning processing method, wherein the step of providing a real-time early warning prompt for abnormal data according to preset early warning thresholds and rules:

[0019] According to the preset warning thresholds and rules, the abnormal data is sent out as warning notifications via email, SMS, and application push to provide real-time warning prompts.

[0020] The data anomaly detection and early warning processing method, wherein the abnormal data includes:

[0021] Abnormal data refers to the point where the data value deviates from the normal range at a specified time point;

[0022] Contextual anomaly data where a data point exhibits characteristics that are inconsistent with expectations under a given context or condition;

[0023] A cluster of outliers is a collection of data points that exhibit characteristics that are inconsistent with the overall data set.

[0024] The data anomaly detection and early warning processing method, wherein the step of using multiple built-in anomaly detection algorithms to perform anomaly detection on data includes:

[0025] Perform point anomaly detection on data and analyze time series data using statistical methods or machine learning models;

[0026] Perform contextual anomaly detection on data, combine contextual information, and use conditional probability models or association rule learning to identify anomalies;

[0027] Perform cluster anomaly detection on data, applying clustering algorithms or density-based methods to identify abnormal clusters in the data set.

[0028] The data anomaly detection and early warning processing method, wherein the step of collecting data from multiple data sources in real time includes:

[0029] Receive user's operation instructions to customize the content of warning notification, including detailed information of abnormal data and recommended action plans.

[0030] A data anomaly detection and early warning processing device, wherein the device comprises:

[0031] A pre-setting module is used to pre-set warning thresholds and rules to detect abnormal data and trigger warnings;

[0032] Custom module, used to receive user's operation instructions to customize the content of warning notification, including detailed information of abnormal data and recommended action plan;

[0033] A data collection module, used to collect data from multiple data sources in real time;

[0034] A preprocessing module is used to preprocess the collected data to obtain preprocessed data;

[0035] The anomaly detection module is used to use stream processing technology to perform real-time analysis on pre-processed data, capture dynamic patterns of data changes, and use built-in multiple anomaly detection algorithms to detect anomalies in data;

[0036] The early warning prompt module is used to determine that there is abnormal data when it detects that the data deviates from the normal data points, and to provide real-time early warning prompts for the abnormal data according to the preset early warning thresholds and rules.

[0037] An intelligent terminal includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, including the method for executing any one of the methods described above.

[0038] A computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the methods described above.

[0039] Beneficial effects of the present invention: The present invention provides a data anomaly detection and early warning processing method, device, intelligent terminal and storage medium, which collects data from multiple data sources in real time; preprocesses the collected data to obtain preprocessed data; uses stream processing technology to analyze the preprocessed data in real time, captures the dynamic pattern of data changes, and uses built-in multiple anomaly detection algorithms to detect anomalies in the data; when it is detected that there are data points that deviate from the normal data, it is determined that there is abnormal data, and a real-time early warning prompt is given for the abnormal data according to the preset early warning threshold and rules. The present invention provides an abnormal data detection and intelligent early warning method, which can automatically detect abnormal data in a data set and issue an early warning when an anomaly is detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 It is a flow chart of the data anomaly detection and early warning processing method provided in Example 1 of the present invention.

[0042] Figure 2 It is a flow chart of the data anomaly detection and early warning processing method provided in Example 2 of the present invention.

[0043] Figure 3 A principle block diagram of an embodiment of a data anomaly detection and early warning processing device provided by the present invention.

[0044] Figure 4 It is a block diagram of the internal structure principle of the intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0047] Faced with the increasing amount of data and complex indicators, the present invention introduces an intelligent early warning system that can automatically identify and warn of abnormal data when the data is running smoothly. Once a data point that deviates from the norm is detected, the system immediately issues an alarm through a robot to remind the operation team to verify. This not only improves the efficiency of data processing, but also ensures the accuracy of decision-making. Outliers, or outliers, although they account for a small proportion of the data set, may have a significant impact on the business.

[0048] like Figure 1 As shown, a data anomaly detection and early warning processing method of embodiment 1 of the present invention includes the following steps:

[0049] Step S100, collecting data from multiple data sources in real time;

[0050] In the embodiment of the present invention, general data inspection is used as an example for explanation. In specific implementation, data is collected from multiple data sources in real time. For example, different data sources can be integrated and connected to obtain real-time data to provide support for decision-making.

[0051] For example, collecting customer interaction data. For example, an e-commerce company wants to understand customers’ purchasing behavior and preferences in order to optimize product recommendations. The company can collect information in real time from the following data sources:

[0052] Website analysis tools: Use tools such as Google Analytics to track customers’ browsing history, click behavior, and dwell time on the website in real time.

[0053] Social media: Collect customer interaction data through the APIs of social media platforms (such as Facebook, Instagram, Weibo, etc.) to understand customers’ comments, likes, and sharing behaviors on the brand.

[0054] CRM system: Obtain the customer’s basic information, transaction history, and service records from the CRM system to build a comprehensive customer profile.

[0055] Another example is collecting marketing data. For example, a marketing company wants to monitor the effectiveness of its digital advertising. To achieve this goal, data can be collected in real time from the following data sources:

[0056] Advertising platform: Obtain advertising data from advertising platforms such as Google AdWords and Facebook Ads, including impressions, click-through rates, conversion rates, etc.

[0057] Email marketing tools: Get real-time email open and click rates through email marketing platforms such as MailChimp to evaluate the effectiveness of different email marketing campaigns.

[0058] Website traffic analysis: Use website traffic analysis tools to monitor the conversion from ad clicks to actual website visits.

[0059] It can be seen that by integrating information from different channels and data sources, enterprises can obtain a comprehensive and real-time data view. This not only helps enterprises gain insight into customer needs and market trends, but also improves response speed and decision-making accuracy. Connecting these data sources to a unified platform provides strong support for the strategic planning and execution of enterprises.

[0060] Step S200, preprocessing the collected data to obtain preprocessed data;

[0061] In this step, before data anomaly detection, the original data is cleaned and converted to improve data quality and applicability. This step includes data deduplication, missing value processing, format standardization, outlier detection and processing, etc.

[0062] Among them, data deduplication: that is, deleting duplicate records to ensure that each piece of data is unique and avoid bias in analysis.

[0063] Missing value processing: that is, analyzing the missing values ​​in the data set, you can choose to delete these records, fill them with the mean or median, or use a predictive model to fill them.

[0064] Format standardization is the process of standardizing data into a uniform format, such as standardizing dates into a uniform format, or standardizing different ways of writing the same information (such as standardizing "NY" and "New York" into "NewYork").

[0065] By removing duplicate and incomplete data, we ensure that the data used is reliable and high-quality, which is crucial for subsequent analysis and decision-making. The preprocessed data is neater and more consistent, which reduces the time analysts spend on data analysis and allows them to focus more on extracting insights. The preprocessed data can better reflect the real situation, reduce erroneous analysis and decision-making caused by data problems, and help improve the accuracy of the final results.

[0066] Step S300: Analyze the pre-processed data in real time using stream processing technology to capture the dynamic pattern of data changes, and use multiple built-in anomaly detection algorithms to detect anomalies in the data;

[0067] This step embodiment involves using advanced data stream processing technology to monitor and analyze data in real time and identify potential problems and abnormal situations in a timely manner.

[0068] The stream processing technology used in the present invention processes the continuously generated data stream in real time, and can analyze the data and respond to it instantly. Compared with traditional batch processing, stream processing can process data faster and provide real-time insights.

[0069] Regarding dynamic pattern capture, the present invention can identify dynamic patterns of changes in data through real-time analysis. These patterns reflect the trend and periodicity of the data, so that anomalies can be discovered in a timely manner, such as a drop in sales data or a surge in website visits.

[0070] The real-time data stream is then analyzed using a variety of built-in anomaly detection algorithms (such as methods based on statistics, machine learning, and time series analysis) to identify and flag anomalous data points that differ significantly from normal patterns.

[0071] For example, consider an online retail company using stream processing to monitor its sales data in real time:

[0072] When capturing dynamic patterns, specifically during large-scale promotions, you can analyze the changing trends of sales data in real time to find out which products have seen a surge in sales and which have flat sales, so that you can adjust inventory and marketing strategies in real time.

[0073] In the anomaly detection stage, if the sales volume of a certain product suddenly increases within a certain period of time, the present invention will identify this anomaly through the stream processing system, and use different anomaly detection algorithms to detect threshold violations or based on historical data patterns to confirm whether the product is a real transaction caused by online advertising or other channels, or a data error caused by a system failure.

[0074] It can be seen that the present invention can quickly respond to data changes and capture dynamic patterns by utilizing stream processing technology and real-time analysis, thereby optimizing business operations and decision-making. Combined with anomaly detection algorithms, it can effectively identify potential problems, improve the level of intelligent data processing, and thus promote continuous improvement and business innovation of enterprises.

[0075] The present invention uses real-time analysis to enable enterprises to promptly identify potential problems, such as equipment failures or network attacks, and take quick action to reduce losses. And by identifying dynamic patterns, enterprises can better predict trends and changes, optimize decision-making processes and resource allocation. And through anomaly detection, enterprises can promptly detect and respond to potential security threats or fraudulent behavior, improving overall data security.

[0076] Step S400: When it is detected that data deviates from the normal data point, it is determined to be abnormal data, and a real-time warning prompt is given for the abnormal data according to the preset warning threshold and rules.

[0077] This step embodiment refers to identifying data points that are significantly different from normal patterns through set standards and rules during the data analysis process, and issuing alarms in a timely manner so that relevant personnel can take prompt action.

[0078] Specifically, when performing data analysis, the present invention can use statistical models or machine learning algorithms to determine the "normal" or normal range of data. If some data points significantly exceed this range, they are considered "data deviation from normal".

[0079] In the embodiment of the present invention, once the data is detected to be out of the norm, these data points are automatically marked as abnormal. This determination depends on preset rules, including preset thresholds (for example, values ​​exceeding a certain range) or other statistical indicators (such as standard deviation).

[0080] When abnormal data is determined, the embodiment of the present invention will automatically generate a notification based on the set warning threshold. These notifications can be sent to the relevant responsible persons or teams through various means such as email, SMS or system alarms to ensure that they can respond and handle the problem quickly.

[0081] For example, a large network company is responsible for monitoring the server performance of its data center. In order to ensure the normal operation and high availability of the server, the embodiment of the present invention sets a parameter monitoring system to detect abnormal situations.

[0082] Reference for normal performance data: Under normal circumstances, the average hourly CPU usage of servers in a data center is between 20% and 75%. In practice, a CPU usage of more than 75% usually indicates a possible bottleneck, while a CPU usage of less than 20% may indicate that the server is not fully utilized.

[0083] Setting a warning threshold: For ease of monitoring, a warning threshold may be set in this embodiment: if the CPU usage of a server exceeds 85% or is less than 10% for 5 consecutive minutes, the present invention will mark this state as abnormal.

[0084] Monitoring process: In a monitoring cycle, the present embodiment detects a server named "Server_A" whose CPU usage rate remains at 90% for 10 consecutive minutes. Since 90% exceeds the set threshold of 85%, the present invention will immediately identify that this data deviates from the normal range.

[0085] Abnormal data determination: In the embodiment of the present invention, when it is confirmed that the CPU usage of "Server_A" is 90% and is determined to be abnormal data, a detailed report will be generated, including time, monitoring indicators and current CPU usage, and marked as "needs attention".

[0086] And an early warning message will be generated immediately, as follows:

[0087] Warning content: "Warning! It is detected that the CPU usage of server 'Server_A' has exceeded 90% for the past 10 minutes. Please check whether there is overload or potential problems."

[0088] The warning information is sent to the operation and maintenance team's email and monitoring system dashboard at the same time to ensure that relevant personnel receive the information in a timely manner.

[0089] This makes it convenient for relevant personnel to quickly start the diagnostic program and analyze the operation status of "Server_A" after receiving the warning. It is found that this server is processing a large number of concurrent requests, and attempts to disperse the traffic by optimizing load balancing, while checking whether there are background processes that occupy too many resources.

[0090] Through this example, we can see the key role of real-time monitoring and early warning prompts in data monitoring. Timely identification of abnormal server CPU usage not only helps prevent potential system failures, but also ensures the continuous availability of customer service. With this mechanism, the operation and maintenance team can respond quickly before the problem affects the user experience, thereby improving overall operational efficiency. This approach is of great significance in ensuring the health of IT infrastructure and can significantly reduce the risk of service interruptions.

[0091] The benefit of this embodiment is that the real-time early warning mechanism can detect and handle problems in the shortest possible time to prevent potential losses from expanding, which is particularly important in the financial and manufacturing industries. In addition, the automated anomaly detection and early warning can reduce the reliance on manual monitoring, reduce labor costs, and improve operational efficiency. Moreover, with the help of real-time anomaly monitoring, the relevant management can obtain timely information to make decisions quickly and avoid decision-making errors caused by data problems.

[0092] Furthermore, in another embodiment of the present invention, the data anomaly detection and early warning processing method, before the step of collecting data in real time from multiple data sources, includes: receiving the user's operating instructions to customize the content of the early warning notification, including detailed information of the abnormal data and a recommended action plan, that is, the embodiment of the present invention allows the user to customize the content of the early warning notification, including detailed information of the abnormal data and a recommended action plan.

[0093] In another embodiment, the specific implementation includes the steps of presetting warning thresholds and rules for detecting abnormal data and triggering warnings.

[0094] For example, an air quality monitoring company is responsible for monitoring environmental sensors across the country and collecting air quality data in real time. In order to promptly detect and respond to abnormal air pollution, reasonable warning thresholds and rules can be set.

[0095] The process of setting warning thresholds and rules:

[0096] Normal air quality data reference: According to environmental monitoring data, the air quality index (AQI) of a region is usually between 0 and 100, indicating good to moderate air quality. According to historical data, the AQI generally does not exceed 150. When it exceeds 150, it means that the air quality is poor and needs attention.

[0097] Setting warning thresholds: To effectively warn, this embodiment can set the following thresholds and rules:

[0098] Green warning: AQI ≥ 100, indicating that the air quality has deteriorated but is still acceptable. The public is advised to pay attention.

[0099] Yellow alert: AQI ≥ 150, indicating poor air quality, the public is advised to reduce outdoor activities.

[0100] Red alert: AQI ≥ 200, indicating that the air quality is extremely poor and the public is advised to avoid going out.

[0101] Monitoring and anomaly detection: During normal monitoring, sensor data shows that the AQI is always in the range of 50 to 90, which is good air quality. However, one day, the system receives sensor data from a city, and the AQI data of the city suddenly soars to 220.

[0102] Triggering an early warning: For example, if the AQI data reaches 220, exceeding the set red warning threshold, the system immediately triggers the early warning mechanism and generates an emergency notification with the following content:

[0103] Warning message: "Emergency alert! The air quality index AQI in city XYZ has reached 220 (red warning). The public is advised to avoid going out and relevant departments are requested to immediately initiate emergency response measures." This notification was sent in real time to environmental protection departments, public health departments and the public via email, text messages and social media platforms.

[0104] This allows for the following follow-up responses: For example, after receiving an early warning, the environmental protection department quickly organizes an emergency team to conduct an on-site investigation to understand the causes of the sharp drop in air quality (such as industrial emissions, traffic congestion, etc.), and releases relevant information to alert the public and the media.

[0105] In this way, through the pre-set AQI warning thresholds and rules, environmental pollution anomalies can be identified in a timely manner, and measures can be taken to respond quickly. This not only helps protect public health and reduce the harm caused by pollution, but also enhances credibility and social responsibility. This monitoring case emphasizes the necessity of data anomaly detection and the importance of timely response, which helps to build an efficient emergency response mechanism in the field of real-time monitoring.

[0106] In another embodiment, the data anomaly detection and early warning processing method, wherein the multiple anomaly detection algorithms used to perform anomaly detection on data using multiple built-in anomaly detection algorithms include:

[0107] Statistical methods are used to identify outliers in the data using statistical principles such as standard deviation and box plots; that is, statistical principles such as standard deviation and box plots are used to identify outliers in the data.

[0108] Using machine learning methods, support vector machines and random forests of machine learning models are applied to perform data anomaly detection; that is, machine learning models such as support vector machines (SVM), random forests, etc. are applied to perform anomaly detection.

[0109] Deep learning methods are used to identify complex abnormal patterns using autoencoders of deep neural networks, that is, deep neural networks, such as autoencoders, are used to identify complex abnormal patterns.

[0110] In another embodiment, the data anomaly detection and early warning processing method, wherein the step of providing a real-time early warning prompt for abnormal data according to the preset early warning threshold and rules: according to the preset early warning threshold and rules, the abnormal data can be sent through email, SMS, application push and multiple other methods to provide a real-time early warning prompt. This enables the detected abnormal data to be notified to the relevant person in charge through early warning prompts in a timely manner for timely processing.

[0111] In a further embodiment, in the data anomaly detection and early warning processing method, the abnormal data includes:

[0112] S11, abnormal data at a specified time point where the data value deviates from the normal range;

[0113] Abnormal data in the present invention refers to the value of a data point at a specific time point that significantly deviates from a preset normal range. This situation usually indicates a system failure, data input error or an emergency.

[0114] For example, a power company monitors the load of the power grid. Under normal circumstances, the load of a substation should be maintained between 1,000 and 2,000 kilowatts. However, at a certain time (for example, 3 p.m.), the monitoring system records that the load of the substation suddenly soared to 5,000 kilowatts. This data is obviously beyond the normal range and may be due to equipment failure or data collection errors.

[0115] S12. Contextual anomaly data where the data point exhibits characteristics that are inconsistent with expectations under specified contexts or conditions;

[0116] This type of abnormal data refers to the behavior of a data point that does not conform to the expected characteristics under a specific context or conditions. This usually requires analysis in combination with other conditions to determine whether the data is abnormal.

[0117] For example, consider the sales data monitored by an online retailer during a promotion. Normally, the daily sales of a certain product are about 50 units. However, on the first day of the promotion, the sales of the product reach 500 units, which is reasonable in the context. But if the sales of the product remain at 300 units on the second day after the promotion ends, this is abnormal because sales should fall back to normal levels without the promotion. In this case, the sales data shows anomalies that do not conform to the characteristics in a specific context, and timely warnings can be issued.

[0118] S13. Clustered anomaly data where a set of data points exhibits characteristics that are inconsistent with the overall data set.

[0119] The abnormal data in this step refers to a group of data points that have significant differences in characteristics or behaviors from the overall data set, forming a cluster. This situation may indicate a specific event or trend.

[0120] For example, a social media platform monitors the daily activity data of users. Under normal circumstances, the daily activity of users is between 1,000 and 5,000 times. However, one day, it was monitored that a group of users (for example, 100 users) were as active as 20,000 times in one day. The activity of this group of users is significantly higher than that of other users, which may indicate that they participated in a popular event or were attracted by a specific content. This cluster anomaly can help the platform identify potential hot topics or events.

[0121] As can be seen from the above, the embodiments of the present invention can detect and analyze abnormal data through the above three types of abnormal data. Whether it is a single point abnormality, a contextual abnormality or a cluster abnormality, timely identification of these data abnormalities can help quickly respond to potential problems, thereby improving system stability and user experience.

[0122] In a further embodiment, the data anomaly detection and early warning processing method, wherein the step of using multiple built-in anomaly detection algorithms to perform anomaly detection on data further includes:

[0123] S21. Detect point anomalies in the data and analyze time series data using statistical methods or machine learning models;

[0124] Point anomaly detection in this step embodiment refers to analyzing time series data through statistical methods or machine learning models to identify data points that significantly deviate from the normal pattern at a specific time point. This detection method usually uses historical data to build a model, determine the normal range, and thus find outliers.

[0125] For example, a meteorological bureau monitors daily temperature data. Through historical data analysis, it is found that the temperature in a certain area is usually between 15℃ and 30℃. However, the temperature on a certain day was recorded as 45℃, which is obviously beyond the normal range. Using statistical methods (such as Z-score or IQR method), the degree of abnormality of this temperature value can be calculated, thus marking it as a point anomaly. This anomaly may be caused by equipment failure or extreme weather events.

[0126] S22, perform contextual anomaly detection on the data, combine context information, and use conditional probability models or association rule learning to identify anomalies;

[0127] In the embodiment of this step, contextual anomaly detection combines the contextual information of the data point (such as time, location, environment, etc.) and uses conditional probability models or association rule learning to identify abnormal data that do not meet expected characteristics under specific conditions.

[0128] For example, an online shopping platform monitors the purchasing behavior of users. Under normal circumstances, the daily sales of a certain electronic product is 100 pieces on weekdays, while it may reach 300 pieces on weekends. If the sales of the product suddenly reach 500 pieces on a weekday, this may be a contextual anomaly. By using the conditional probability model analysis, it is found that the promotion on Monday led to an abnormal increase in sales. Combined with this contextual information, the present invention can identify this sales anomaly and conduct subsequent analysis and response in a targeted manner.

[0129] S23. Perform cluster anomaly detection on the data, apply clustering algorithms or density-based methods to identify abnormal clusters in the data set.

[0130] In this step, cluster anomaly detection is to use a clustering algorithm or a density-based method to identify abnormal clusters in a data set that exhibit characteristics inconsistent with the overall data set. This method is usually used to discover abnormal patterns or trends in data, especially for large-scale data sets.

[0131] For example, a bank monitors the transaction behavior of its customers. Under normal circumstances, the number of transactions per day by customers is between 1 and 10. However, when the system uses clustering algorithms (such as K-means or DBSCAN) to analyze the transaction data, it finds that there is a group of customers whose transaction times are concentrated between 50 and 100. This cluster is significantly higher than the behavior of other customers, which may indicate that these customers are conducting abnormal transactions (such as money laundering activities). By identifying this cluster, the bank can further investigate the transaction patterns of these customers to prevent potential financial risks.

[0132] It can be seen that through the explanations and examples of the above three steps, point anomaly detection, context anomaly detection and cluster anomaly detection each have unique application scenarios and methods. Effective use of these detection methods will help enterprises to identify and respond to anomalies in data in a timely manner, thereby improving the accuracy of decision-making and the security of the system.

[0133] The present invention is further described in detail below through a specific application example:

[0134] like Figure 2 As shown, a data anomaly detection and early warning processing method according to a specific application embodiment of the present invention comprises the following steps:

[0135] S31, start, and enter S32.

[0136] S32, data collection, control the data collection module to start data collection. Specifically as described in the previous embodiment, the present invention can collect data from multiple data sources in real time, perform preprocessing, and enter step S33.

[0137] S33: Data preprocessing, as described in the previous embodiment, and proceed to step S34.

[0138] S34, perform anomaly detection on the collected data through multiple built-in anomaly detection algorithms, including point anomaly detection, enter S35 to generate a first anomaly report; context anomaly detection, enter S36 to generate a second anomaly report; and cluster anomaly detection, enter S37 to generate a third anomaly detection report; pass the above three anomaly detections and enter step S38.

[0139] S38. Real-time warning notification, and proceed to step S39.

[0140] In the embodiment of the present invention, according to the set warning threshold and rules, once abnormal data is detected, the system will trigger a warning.

[0141] S39: Exception rule management, and proceed to step S40.

[0142] In the embodiment of the present invention, the user is allowed to customize the content of the warning notification, including detailed information of the abnormal data and a suggested action plan.

[0143] S40, the system performs adaptive learning and proceeds to step S41.

[0144] In the embodiment of the present invention, the system has learning capability and can adjust the monitoring strategy according to historical data to adapt to changes in data characteristics.

[0145] S41, end.

[0146] Of course, in another embodiment of the present invention, a rule setting iteration function is added to adapt to the ever-changing business needs, including:

[0147] 1) Provide a rule management interface: Provide a user-friendly interface for operators to manage and adjust warning rules.

[0148] 2) Provide historical data analysis: The system records and analyzes historical warning events to provide data support for rule optimization.

[0149] 3) Provide a feedback mechanism: Allow users to provide feedback on the effectiveness of the rules, and the system will automatically or manually adjust the rules accordingly.

[0150] As can be seen from the above, the embodiment of the present invention provides a data anomaly detection and early warning processing method, which can automatically detect abnormal data in a data set and issue an early warning when an anomaly is detected, thereby providing protection for data security.

[0151] Exemplary Devices

[0152] like Figure 3 As shown, an embodiment of the present invention provides a data anomaly detection and early warning processing device, the device comprising:

[0153] A presetting module 310 is used to pre-set warning thresholds and rules for detecting abnormal data and triggering warnings;

[0154] A customization module 320, for receiving a user's operation instruction to customize the content of the warning notification, including detailed information of the abnormal data and a suggested action plan;

[0155] A data collection module 330 is used to collect data from multiple data sources in real time;

[0156] A preprocessing module 340 is used to preprocess the collected data to obtain preprocessed data;

[0157] Anomaly detection module 350, used to use stream processing technology to perform real-time analysis on pre-processed data, capture dynamic patterns of data changes, and use built-in multiple anomaly detection algorithms to perform anomaly detection on data;

[0158] The early warning module 360 ​​is used to determine that there is abnormal data when it detects that there are data points that deviate from the normal data points, and to issue a real-time early warning prompt for the abnormal data according to the preset early warning thresholds and rules, as described above.

[0159] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 4 As shown. The intelligent terminal includes a processor, a memory, a network interface, a display screen, and a database connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data anomaly detection and early warning processing method is implemented. The database of the intelligent terminal is used to store a data anomaly detection and early warning processing program.

[0160] Those skilled in the art will understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0161] In one embodiment, a smart terminal is provided, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the following operations:

[0162] Collect data from multiple data sources in real time;

[0163] Preprocessing the collected data to obtain preprocessed data;

[0164] Use stream processing technology to analyze pre-processed data in real time, capture dynamic patterns of data changes, and use built-in anomaly detection algorithms to detect anomalies in data;

[0165] When it is detected that data deviates from normal data points, it is determined to be abnormal data, and a real-time warning prompt is given for the abnormal data according to the preset warning threshold and rules, as described above.

[0166] Wherein, the step of collecting data from multiple data sources in real time includes:

[0167] Pre-set warning thresholds and rules to detect abnormal data and trigger warnings.

[0168] The multiple anomaly detection algorithms used to perform anomaly detection on data using multiple built-in anomaly detection algorithms include:

[0169] Use statistical methods, using standard deviation and box plots based on statistical principles, to identify outliers in the data;

[0170] Adopt machine learning methods, apply support vector machine and random forest of machine learning models to perform data anomaly detection;

[0171] Adopt deep learning methods and use autoencoders of deep neural networks to identify complex abnormal patterns.

[0172] Among them, the step of providing real-time warning prompts for abnormal data according to preset warning thresholds and rules:

[0173] According to the preset warning thresholds and rules, the abnormal data is sent out as warning notifications via email, SMS, and application push to provide real-time warning prompts.

[0174] The abnormal data includes:

[0175] Abnormal data refers to the point where the data value deviates from the normal range at a specified time point;

[0176] Contextual anomaly data where a data point exhibits characteristics that are inconsistent with expectations under a given context or condition;

[0177] A cluster of outliers is a collection of data points that exhibit characteristics that are inconsistent with the overall data set.

[0178] The step of using multiple built-in anomaly detection algorithms to perform anomaly detection on data includes:

[0179] Perform point anomaly detection on data and analyze time series data using statistical methods or machine learning models;

[0180] Perform contextual anomaly detection on data, combine contextual information, and use conditional probability models or association rule learning to identify anomalies;

[0181] Perform cluster anomaly detection on data, applying clustering algorithms or density-based methods to identify abnormal clusters in the data set.

[0182] Wherein, the step of collecting data from multiple data sources in real time includes:

[0183] Receive the user's operation instructions to customize the content of the warning notification, including detailed information of the abnormal data and the recommended action plan, as described above.

[0184] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0185] In summary, the present invention provides a data anomaly detection and early warning processing method, device, intelligent terminal and storage medium. The present invention provides an abnormal data detection and intelligent early warning method, which can automatically detect abnormal data in a data set and issue an early warning when an abnormality is detected.

Claims

1. A data anomaly detection and early warning processing method, characterized in that: include: Collect data from multiple data sources in real time; Preprocessing the collected data to obtain preprocessed data; Use stream processing technology to analyze pre-processed data in real time, capture dynamic patterns of data changes, and use built-in anomaly detection algorithms to detect anomalies in data; When data points that deviate from normal data are detected, they are judged to be abnormal data, and real-time warning prompts are issued for the abnormal data based on preset warning thresholds and rules.

2. The data anomaly detection and early warning processing method according to claim 1 is characterized in that: The step of collecting data from multiple data sources in real time includes: Pre-set warning thresholds and rules to detect abnormal data and trigger warnings.

3. The data anomaly detection and early warning processing method according to claim 1 is characterized in that: The multiple anomaly detection algorithms used to perform anomaly detection on data using multiple built-in anomaly detection algorithms include: Use statistical methods, using standard deviation and box plots based on statistical principles, to identify outliers in the data; Adopt machine learning methods, apply support vector machine and random forest of machine learning models to perform data anomaly detection; Adopt deep learning methods and use autoencoders of deep neural networks to identify complex abnormal patterns.

4. The data anomaly detection and early warning processing method according to claim 1 is characterized in that: The step of providing real-time warning prompts for abnormal data according to preset warning thresholds and rules: According to the preset warning thresholds and rules, the abnormal data is sent out as warning notifications via email, SMS, and application push to provide real-time warning prompts.

5. The data anomaly detection and early warning processing method according to claim 1 is characterized in that: The abnormal data includes: Abnormal data refers to the point where the data value deviates from the normal range at a specified time point; Contextual anomaly data where a data point exhibits characteristics that are inconsistent with expectations under a given context or condition; A cluster of outliers is a collection of data points that exhibit characteristics that are inconsistent with the overall data set.

6. The data anomaly detection and early warning processing method according to claim 5 is characterized in that: The step of using multiple built-in anomaly detection algorithms to perform anomaly detection on data includes: Perform point anomaly detection on data and analyze time series data using statistical methods or machine learning models; Perform contextual anomaly detection on data, combine contextual information, and use conditional probability models or association rule learning to identify anomalies; Perform cluster anomaly detection on data, applying clustering algorithms or density-based methods to identify abnormal clusters in the data set.

7. The data anomaly detection and early warning processing method according to claim 2 is characterized in that: The step of collecting data from multiple data sources in real time includes: Receive user's operation instructions to customize the content of warning notification, including detailed information of abnormal data and recommended action plans.

8. A data anomaly detection and early warning processing device, characterized in that: The device comprises: A pre-setting module is used to pre-set warning thresholds and rules to detect abnormal data and trigger warnings; Custom module, used to receive user's operation instructions to customize the content of warning notification, including detailed information of abnormal data and recommended action plan; A data collection module, used to collect data from multiple data sources in real time; A preprocessing module is used to preprocess the collected data to obtain preprocessed data; The anomaly detection module is used to use stream processing technology to perform real-time analysis on pre-processed data, capture dynamic patterns of data changes, and use built-in multiple anomaly detection algorithms to detect anomalies in data; The early warning prompt module is used to determine that there is abnormal data when it detects that the data deviates from the normal data points, and to provide real-time early warning prompts for the abnormal data according to the preset early warning thresholds and rules.

9. An intelligent terminal, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include being used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as described in any one of claims 1 to 7.

Images