Data anomaly detection method based on deformation model, storage medium, and electronic device

Abstract

The present application provides a data anomaly detection method based on a deformation model, a storage medium, and an electronic device. The method comprises: acquiring service data, and determining a data temporal characteristic of the service data; on the basis of the data temporal characteristic and historical service data, determining a reference baseline curve of the service data; on the basis of the service data and the reference baseline curve, calculating a baseline deformation variable of the service data; and on the basis of the baseline deformation variable, determining an anomaly alert. According to the method provided by embodiments of the present application, a data anomaly can be effectively converted into an anomaly in mathematical logic, and anomaly conditions can be quickly identified from time sequence data according to the described method, thereby significantly improving the accuracy of anomaly recognition. By applying data transformation processing twice, the described method enhances the capability of recognizing a data anomaly, and also improves the efficiency and accuracy during a recognition process. In addition, the reusability of the method is enhanced, so that high-efficiency and accurate performance can be maintained in different data scenarios.

Description

Data change detection method, storage medium and electronic device based on deformation model

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application CN202311353473.5, entitled “Data anomaly detection method, storage medium and electronic device based on deformation model”, filed on October 18, 2023. The entire contents of that application are incorporated herein by reference. Technical Field

[0003] This application belongs to the field of Internet technology, and in particular to the field of Internet business data technology. Background Art

[0004] In the e-commerce and financial sectors, identifying the normality of business data is a crucial task, involving real-time monitoring and analysis of large amounts of complex data. Traditional identification solutions rely on expert experience and extensive human resources, which is not only inefficient but also difficult to adapt to changing business scenarios and data types.

[0005] Summary of the Invention

[0006] The present application provides a data change detection method, storage medium, and electronic device based on a deformation model, which are used to improve the reusability and efficiency of data change identification.

[0007] In a first aspect, an embodiment of the present application provides a method for detecting data changes based on a deformation model, comprising:

[0008] Acquiring business data and determining the data time characteristics of the business data;

[0009] Based on the data time characteristic, generating a reference benchmark curve corresponding to the historical business data, wherein the reference benchmark curve is used to characterize the performance of the historical business data with respect to the data time characteristic;

[0010] Calculating a reference deformation variable of the business data based on the business data and the reference baseline curve, wherein the reference deformation variable is used to characterize the difference between the business data and the reference baseline curve;

[0011] An abnormality alarm is determined based on the reference deformation variable.

[0012] In the above implementation process, a reference benchmark curve is generated by combining historical business data to determine the difference between the current business data and the reference benchmark curve. Based on the difference, it can be judged whether the current business data is abnormal. This method can be applied to anomaly identification in different business scenarios and data types, and has higher identification efficiency.

[0013] Optionally, the data time characteristics include at least one of a data sampling period, holidays, activity days, and hot time. This allows the abnormality of business data in different time periods to be evaluated based on the characteristics of different time periods.

[0014] Optionally, if the data time characteristic includes a data sampling period, determining the data time characteristic of the service data includes:

[0015] Converting the service data into frequency domain data using a Fourier transform formula;

[0016] Normalizing the frequency domain data to obtain normalized frequency domain data;

[0017] The sampling period is determined according to the normalized frequency domain data.

[0018] In the above implementation process, the business data is converted into frequency domain data and then normalized. Normalization can eliminate the problem of data dimension, which makes it easier to identify anomalies for business data of different data types.

[0019] Optionally, generating a reference benchmark curve corresponding to historical business data based on the data time characteristic includes:

[0020] Calculate the year-on-year data of the historical business data within the corresponding period according to the data time characteristics;

[0021] The month-on-month data is plotted according to a time series to generate a reference benchmark curve.

[0022] In the above implementation process, by generating a reference benchmark curve from the month-on-month data, the month-on-month data can reflect the changes in the corresponding period, thereby improving the accuracy of identifying abnormal data.

[0023] Optionally, the calculating the reference deformation variable of the business data based on the business data and the reference reference curve includes:

[0024] Normalizing the business data to obtain normalized data;

[0025] A reference deformation variable of the business data is calculated according to the normalized data and the reference reference curve.

[0026] In the above implementation process, after the business data is normalized, it can be easily compared with the reference benchmark curve. Normalization can also reduce the risk of numerical overflow or underflow.

[0027] Optionally, normalizing the business data to obtain normalized data includes:

[0028] The correlation between the business data and the data corresponding to the reference benchmark curve is calculated to achieve normalization processing of the business data, and the correlation is normalized data.

[0029] In the above implementation process, normalizing the business data by calculating the correlation can ensure that the normalized data can reflect the relationship between the two, thereby improving the accuracy of anomaly identification.

[0030] Optionally, calculating the correlation between the business data and data corresponding to the reference benchmark curve includes:

[0031] The correlation between the business data and the data corresponding to the reference benchmark curve is calculated using the cosine similarity calculation formula. This allows for a more accurate calculation of the correlation between the two.

[0032] Optionally, the baseline deformation variable of the business data is calculated using the following calculation formula:

[0033] Baseline deformation variable = (1 - the normalized data) * |the service data - the corresponding data in the reference baseline curve|. Using this formula, the baseline deformation variable accurately reflects the difference between the service data and the reference baseline curve.

[0034] Optionally, the abnormal change alarm includes multiple alarms of different levels, and determining the abnormal change alarm based on the baseline deformation variable includes:

[0035] If the reference deformation variable is greater than the first alarm threshold, determining the level of the abnormal movement alarm to be the most risky level;

[0036] If the reference deformation variable is greater than the second alarm threshold and less than or equal to the first alarm threshold, the level of the abnormal change alarm is determined to be a concern level.

[0037] In the above implementation process, by setting multiple alarms of different levels, the degree of abnormality of business data can be quickly known.

[0038] Optionally, the method further includes:

[0039] The first alarm threshold and / or the second alarm threshold are modified based on the result of the abnormal alarm, so that the alarm threshold can be adjusted based on the actual alarm result to improve the accuracy of subsequent alarms.

[0040] In a second aspect, an embodiment of the present application provides a method for detecting data anomalies in a deformable model, the method comprising:

[0041] Obtain business data and determine the data time characteristics of the business data;

[0042] Determining a reference benchmark curve for the business data based on the data time characteristics and historical business data;

[0043] Calculating a reference deformation variable of the business data based on the business data and the reference reference curve;

[0044] An abnormality alarm is determined based on the reference deformation variable.

[0045] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for detecting data anomalies based on a deformation model described in any one of the first aspect or the second aspect of the present application is implemented.

[0046] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory storing a computer program; a processor communicatively connected to the memory, and for executing the data anomaly detection method based on deformation model described in any one of the first aspect or the second aspect of the present application when calling the computer program.

[0047] The data anomaly detection method based on a deformation model provided in the embodiment of the present application can convert business data anomalies into mathematical and logical anomalies, so that data anomalies with time series characteristics can be quickly identified, thereby improving the accuracy of identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] FIG1 is a schematic diagram of an application scenario of a data anomaly detection method based on a deformation model provided by an embodiment of the present application;

[0050] FIG2 is a flow chart of a method for detecting data anomalies based on a deformation model according to an embodiment of the present application;

[0051] FIG3 is an original waveform diagram of service data provided by an embodiment of the present application;

[0052] FIG4 is a waveform diagram of a Fourier transform of service data provided by an embodiment of the present application;

[0053] FIG5 is a flow chart of obtaining a reference deformation variable according to an embodiment of the present application;

[0054] FIG6 is a data curve diagram of a time period history provided by an embodiment of the present application;

[0055] FIG7 is a data curve diagram after normalization processing of business data provided by an embodiment of the present application;

[0056] FIG8 is a graph showing an acquired deformation variable and an alarm threshold value according to an embodiment of the present application;

[0057] FIG9 is a schematic diagram of a specific implementation process of a data change detection method based on a deformation model provided in an embodiment of the present application;

[0058] FIG10 is a flow chart of another method for detecting data changes based on a deformation model according to an embodiment of the present application;

[0059] FIG11 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0061] An embodiment of the present application provides a data anomaly detection method based on a deformation model. By combining historical business data, a reference baseline curve is generated to determine the difference between the current business data and the reference baseline curve. Based on the difference, it can be judged whether the current business data is abnormal. It can be applied to anomaly identification in different business scenarios and data types, and has higher identification efficiency.

[0062] FIG1 shows a schematic diagram of an application scenario of a data anomaly detection method based on a deformation model according to an embodiment of the present application. As shown in FIG1 , the data anomaly detection method based on a deformation model in this embodiment first determines the data time characteristics of the business data of the scenario, and determines the reference baseline curve based on the performance of the historical business data. The baseline deformation variable of the business data can then be calculated to characterize the difference between the business data and the reference baseline curve. In the process of obtaining the baseline deformation variable, in the specific embodiment of the present application, the business data can be transformed twice. The first is normalized cross-correlation, which normalizes the entire time series data into an interval of [0, 1], removes the peak waves caused by time, and removes the influence of the shape. The baseline deformation variable is then calculated based on the cosine correlation, and the magnitude degradation is achieved by restoring the data magnitude, that is, the magnitude is finally obtained by restoring the magnitude by difference, and the magnitude influence is removed. Finally, an anomaly alarm is issued based on the baseline deformation variable. This application innovatively applies correlation determination to data anomaly identification, converting data anomalies into mathematical and logical anomalies. This can quickly identify data anomalies with time series characteristics, improve the accuracy of identification, and combine the results of introducing two data deformations, coupled with automatic adjustment capabilities, to improve the efficiency of anomaly identification.

[0063] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with other drawings in the embodiments of the present application.

[0064] Please refer to FIG. 2 , which is a flow chart of a method for detecting data changes based on a deformation model according to an embodiment of the present application. The method includes the following steps:

[0065] Step S110: Acquire business data and determine the data time characteristics of the business data.

[0066] In some embodiments, business data may refer to transaction data (such as a user's sales, purchase, payment, and other financial transaction information), or it may also refer to customer data (such as customer purchase information), operational data (such as various aspects of a company's daily operations, such as production, inventory, logistics, etc.), market data (such as market promotion data, etc.), interaction data (such as a company's interaction records through websites, social media, customer service and other channels), behavioral data (such as user behavior trajectories on a website or application, such as click streams, page visit duration, etc.), etc.

[0067] It is understandable that in different business scenarios, the specific information referred to by business data may be different.

[0068] The business data obtained here may refer to the business data that currently needs to be identified as anomalies, and may be obtained from an external device or from an internal memory.

[0069] Business data can refer to data within a period of time, which is regular time series data. The data time characteristics of business data can be understood as the characteristics of the acquired business data over time.

[0070] In some embodiments, the data time characteristics may include at least one of a data sampling period, holidays, activity days, and hot time. These data time characteristics may be determined by analyzing the timestamps of each piece of business data.

[0071] Step S120: Generate a reference benchmark curve corresponding to the historical business data based on the time characteristics of the data.

[0072] The reference benchmark curve is used to characterize the performance of historical business data based on its temporal characteristics. For example, if the temporal characteristics of business data include a sampling period, historical business data within that sampling period can be obtained and then a reference benchmark curve corresponding to that historical business data can be generated.

[0073] Historical business data refers to data of the same type as business data, and is used to compare with current business data. For example, if the sampling period of the current business data is one day, then the historical business data of the previous corresponding day can be obtained. Here, the correspondence can be corresponding to time. For example, if the sampling period of the current business data is Wednesday, then the historical business data of the previous Wednesday can be obtained. For example, if the current sampling period is one week, then the historical business data of the previous week can be obtained, or the historical business data of the same week corresponding to last year can be obtained. Alternatively, if the data time characteristics include activity days, the historical business data obtained may refer to business data generated on historical activity days. Alternatively, if the data time characteristics include holidays (such as the Dragon Boat Festival), the historical business data obtained may refer to business data generated on the same previous holidays (such as the Dragon Boat Festival). Alternatively, if the data time characteristics include hotspot time, the historical business data obtained may refer to business data generated on the same hotspot time.

[0074] Understandably, historical business data can be considered the month-over-month data corresponding to the temporal characteristics of current business data. By acquiring this month-over-month data, we can reflect the month-over-month changes in business data, enabling rapid identification of anomalies. By generating a reference baseline curve for historical business data, we can accurately reflect the historical business data at the corresponding time.

[0075] After acquiring historical business data, a corresponding reference benchmark curve can be generated based on the historical business data. In some embodiments, the reference benchmark curve can be generated by plotting the historical business data in a time series. Of course, if the historical business data includes data from multiple time periods, for example, if the sampling period for the current business data is Wednesday, and if the historical business data acquired is from Wednesdays of the previous three weeks, then three corresponding reference benchmark curves can be generated, each of which can be plotted on a single graph. Of course, multiple reference benchmark curves can also be plotted on a single graph.

[0076] Step S130: Calculating a reference deformation variable of the business data based on the business data and the reference reference curve.

[0077] The baseline deformation variable is used to characterize the difference between business data and the reference baseline curve. Since the reference baseline curve is generated based on historical business data, it represents the data conditions of historical business data. Therefore, calculating the baseline deformation variable can effectively reflect the degree of difference between current business data and historical business data.

[0078] In some embodiments, the reference baseline curve can be generated by fitting data from multiple time points, so that the data values ​​at each time point can be obtained through the reference baseline curve. When calculating the baseline deformation variable, the average value of the reference baseline curve and the average value of the business data can be obtained first, and then the difference between the two average values ​​can be used as the baseline deformation variable.

[0079] In some other implementations, the difference between the maximum value of the business data and the maximum value of the reference baseline curve may be used as the reference deformation variable, or the difference between the minimum value of the business data and the minimum value of the reference baseline curve may be used as the reference deformation variable.

[0080] Step S140: determining an abnormality alarm based on the baseline deformation variable.

[0081] After obtaining the baseline deformation variable, the abnormality of the current business data can be determined based on the baseline deformation variable. In some embodiments, for example, if the baseline deformation variable is greater than a set alarm threshold, the current business data is determined to be abnormal and an alarm is generated. Otherwise, the current business data is considered normal and no alarm is generated.

[0082] In the above implementation process, a reference benchmark curve is generated by combining historical business data to determine the difference between the current business data and the reference benchmark curve. Based on the difference, it can be judged whether the current business data is abnormal. This method can be applied to anomaly identification in different business scenarios and data types, and has higher identification efficiency.

[0083] Based on the above embodiment, if the data time characteristic is the sampling period of the poly data, when determining the sampling period of the business data, the business data can be first converted into frequency domain data through the Fourier transform formula, and then the frequency domain data can be normalized to obtain normalized frequency domain data, and the sampling period can be determined based on the normalized frequency domain data.

[0084] As you can understand, business data is time-series data. The Fourier transform can transform time-domain data into frequency-domain data, which is used to analyze the data's periodicity and frequency composition. Therefore, the Fourier transform formula can be used to convert business data in the time domain into business data in the frequency domain, which is called frequency-domain data.

[0085] After obtaining the frequency domain data, the frequency domain data can be normalized. There are many ways to normalize the data, such as minimum-maximum normalization, Z-score normalization, maximum absolute value normalization, etc.

[0086] For normalized frequency domain data, the main frequency peaks can be identified. The periodicity corresponding to the main frequency peaks can help determine the sampling period of the business data. For example, if the business data is daily data, the main frequency peaks may correspond to daily periods. For example, the sampling period may be equal to the total number of data points N divided by the value of the frequency peak.

[0087] In practical applications, relevant tools in MATLAB can be used to determine the sampling period from normalized frequency domain data.

[0088] Figure 3 shows the original waveform of the business data (time on the horizontal axis, amplitude on the vertical axis). Figure 4 shows the waveform of the frequency domain data obtained after Fourier transforming the business data (frequency on the horizontal axis, amplitude on the vertical axis). Figure 3 shows the actual business data trend obtained. Figure 4 performs a Fourier transform on the business data, where the amplitude peak is at a frequency of 10. Compared to the curve in Figure 3, which fluctuates continuously, the frequency graph in Figure 4 has only one fluctuation point at a frequency of 10, while the rest are all zero, making it relatively static. Therefore, at a frequency of 10, based on a total of 1000 input points, the sampling period of the business data is 100, meaning that 100 sampling points constitute a complete cycle.

[0089] In the above implementation process, the business data is converted into frequency domain data and then normalized. Normalization can eliminate the problem of data dimension, which makes it easier to identify anomalies for business data of different data types.

[0090] Based on the above embodiment, when generating a reference benchmark curve, the year-on-year data of historical business data within the corresponding time period can be calculated according to the time characteristics of the data, and then the year-on-year data can be plotted according to the time series to generate a reference benchmark curve.

[0091] Month-over-month data can refer to daily or weekly data within the same period. This data is then plotted in a time series to form a final reference benchmark curve. Month-over-month data reflects the changes in business data and historical business data over the same period, so generating a reference benchmark curve from this data allows for intuitive visualization of these changes.

[0092] In the above implementation process, by generating a reference benchmark curve from the month-on-month data, the month-on-month data can reflect the changes in the corresponding period, thereby improving the accuracy of identifying abnormal data.

[0093] On the basis of the above embodiment, in the above method of calculating the benchmark deformation variable, the business data can also be normalized to obtain normalized data, and then the benchmark deformation variable of the business data is calculated based on the normalized data and the reference benchmark curve. The specific process can be shown in Figure 5.

[0094] There are many ways of normalization, such as minimum-maximum normalization, Z-score normalization, and maximum absolute value normalization.

[0095] In some embodiments, when calculating the baseline deformation variable, the difference between the average value of the normalized data and the average value of the reference baseline curve can be used as the baseline deformation variable, or the difference between the maximum value of the normalized data and the maximum value of the reference baseline curve can be used as the baseline deformation variable, or the difference between the minimum value of the normalized data and the minimum value of the reference baseline curve can be used as the baseline deformation variable, etc.

[0096] In the above implementation process, after the business data is normalized, it can be easily compared with the reference benchmark curve. Normalization can also reduce the risk of numerical overflow or underflow.

[0097] On the basis of the above embodiment, in order to remove the peak waves of business data caused by time and remove the influence of shape, the business data is normalized into an interval of [0,1]. The correlation between the business data and the corresponding data of the reference benchmark curve can be calculated to achieve normalization processing of the business data. The correlation is the normalized data.

[0098] There are many ways to calculate the correlation between business data and the corresponding data of the reference benchmark curve, such as cosine similarity, Pearson correlation coefficient, and distance correlation. The calculated correlation value can be used as normalized data.

[0099] In the above implementation process, normalizing the business data by calculating the correlation can ensure that the normalized data can reflect the relationship between the two, thereby improving the accuracy of anomaly identification.

[0100] On the basis of the above embodiment, in order to obtain a better normalization effect, the cosine similarity calculation formula can be used to calculate the correlation between the business data and the corresponding data of the reference benchmark curve. This can be achieved by restoring the magnitude and downgrading the magnitude to remove the influence of the magnitude and improve accuracy.

[0101] Among them, the cosine similarity calculation formula can be expressed as follows:

[0102] Where x(n) represents the business data, y(n) represents the corresponding data of the reference benchmark curve, and n represents the number of data points.

[0103] As shown in Figure 6, the data curve diagram for a period of time in history (the horizontal axis is time, the vertical axis is data value), that is, the reference benchmark curve, Figure 7 is the data curve diagram after normalization of the business data (the horizontal axis is time, the vertical axis is data value). From the comparison between Figure 6 and Figure 7, it can be seen that by normalizing the business data by cross-correlating it, the peaks and troughs caused by time can be removed.

[0104] Based on the above embodiment, the baseline deformation variable of the business data can be calculated by the following calculation formula:

[0105] Baseline deformation variable = (1-normalized data)*|business data-corresponding data in the reference base curve|.

[0106] Among them, normalized data can refer to the numerical value of the above-mentioned correlation. In the above calculation formula, the absolute value of the difference between the business data and the corresponding data in the reference benchmark curve is multiplied by a coefficient, which is 1-normalized data. Normalized data refers to correlation and reflects the time series characteristics of the data. Using 1-normalized data as a coefficient can further adjust the sensitivity of the benchmark deformation variable to ensure that when the normalized data is low, the penalty for the difference is greater, which is equivalent to when the correlation between the business data and the reference benchmark curve is not large, it can be more clearly reflected through the benchmark deformation variable.

[0107] In the above implementation process, through calculation using this formula, the difference between the business data and the reference benchmark curve can be accurately reflected through the benchmark deformation variable.

[0108] On the basis of the above embodiment, when performing abnormal change alarm, multi-level alarm can be performed, and the abnormal change alarm can include multiple alarms of different levels. In the method of determining the abnormal change alarm based on the baseline deformation variable, if the baseline deformation variable is greater than the first alarm threshold, the level of the abnormal change alarm is determined to be the most risky level; if the baseline deformation variable is greater than the second alarm threshold and less than or equal to the first alarm threshold, the level of the abnormal change alarm is determined to be the attention level.

[0109] As shown in Figure 8, the obtained benchmark deformation variable and alarm threshold curve diagram (the horizontal axis is time, the vertical axis is data value), where the highest peak can be understood as the benchmark deformation variable, the first alarm threshold is P0, and the second alarm threshold is P1. It can be seen that the benchmark deformation variable is finally obtained by restoring the magnitude through difference, which can effectively remove the influence of the magnitude and improve the accuracy of the benchmark deformation variable.

[0110] In Figure 8, if the baseline deformation variable is greater than the first alarm threshold P0, the anomaly alarm is assigned the highest risk level, indicating a high degree of abnormality in the business data. If the baseline deformation variable is between the second and first alarm thresholds, the anomaly alarm is assigned the highest concern level, indicating a moderate degree of abnormality in the business data and warrants special attention.

[0111] It can be understood that if the baseline deformation variable is less than or equal to the second alarm threshold, the level of the abnormal change alarm can be determined to be a no-risk level, and at this time it can be considered that the business data has no abnormality.

[0112] In some implementations, the level of anomaly alerts can be output to personnel, allowing them to quickly identify the degree of abnormality in business data based on the level and, in turn, take different action based on the severity of the abnormality. Different alert thresholds can be set for corresponding levels of anomaly alerts. Specifically, when the baseline deformation variable reaches a certain level of anomaly threshold, an anomaly alert of the corresponding level is generated.

[0113] In some implementations, the abnormality alarm may be divided into more than the two levels exemplified above, and the definitions of different levels may also be different. In actual applications, the abnormality alarm may be further divided into more levels.

[0114] In the above implementation process, by setting multiple alarms of different levels, the degree of abnormality of business data can be quickly known.

[0115] Based on the above embodiments, the above alarm thresholds (including the first alarm threshold and the second alarm threshold) may be set manually based on experience. In order to better adapt to actual conditions, the first alarm threshold and / or the second alarm threshold may also be modified based on the results of the abnormal alarm.

[0116] In some embodiments, if the abnormal alarm results greater than the first alarm threshold are found to be mostly invalid alarms after manual verification, the first alarm threshold can be appropriately adjusted, such as increasing the value of the first alarm threshold, to optimize the alarm.

[0117] In some other implementations, historical alarm results may be statistically analyzed to calculate the ratio of false positives to missed negatives, and then the alarm threshold may be adjusted based on these statistical data to reduce missed positives or false positives.

[0118] In some other implementations, a machine learning algorithm, such as regression analysis, decision tree, or neural network model, may be used to predict the optimal alarm threshold based on the characteristics of historical alarm data and business data.

[0119] In some other implementations, an adaptive algorithm may be designed to dynamically adjust the alarm threshold based on real-time data and alarm results. For example, if multiple consecutive alarms of the same level are confirmed as invalid alarms, the alarm threshold of that level is automatically increased.

[0120] In the above implementation process, the alarm threshold is adjusted based on the actual alarm result to improve the accuracy of subsequent alarms.

[0121] FIG9 is a schematic diagram of the specific implementation process of the deformation model-based data anomaly detection method in an embodiment of the present application. As shown in FIG9 , in this embodiment, the data time characteristics of the business data of the scenario are first determined, such as the sampling period, statutory holidays, normal working days, etc. Then, based on the historical data performance, the shape of the data curve is predicted. By calculating the daily year-on-year and weekly year-on-year growth rates of the same time period within the historical time period, a curve is drawn according to the time series to determine the final baseline, i.e., the reference baseline curve. The business data is then normalized, i.e., the cosine correlation is calculated, and the entire time series data is normalized to a [0, 1] interval. The baseline deformation variable of the business data is calculated based on the normalized data and the reference baseline curve. Anomaly identification, classification, and processing are performed based on the baseline deformation variable. Finally, the alarm threshold is corrected based on the results of the anomaly alarm. That is, different levels of alarm thresholds are automatically set based on the baseline deformation variable, and automatically corrected based on specific feedback.

[0122] Please refer to FIG10 , which is a flowchart of another data change detection method based on a deformation model provided by an embodiment of the present application. The method specifically includes the following steps:

[0123] Step S210: Acquire business data and determine the data time characteristics of the business data.

[0124] Step S220: Determine a reference benchmark curve for the business data based on the data time characteristics and historical business data.

[0125] Step S230: Calculating a reference deformation variable of the business data based on the business data and the reference reference curve.

[0126] Step S240: determining an abnormality alarm based on the baseline deformation variable.

[0127] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific implementation process of this embodiment can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0128] The protection scope of the data anomaly detection method based on deformation model described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the existing technology based on the principles of the present application are included in the protection scope of the present application.

[0129] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for detecting data anomalies based on a deformation model provided in any embodiment of the present application is implemented.

[0130] In the embodiment of the present application, any combination of one or more storage media can be used. The storage medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by an instruction execution system, device or device or used in combination with it.

[0131] An embodiment of the present application also provides an electronic device. Figure 11 shows a structural schematic diagram of an electronic device 100 provided in an embodiment of the present application. In some embodiments, the electronic device can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) and other terminal devices. In addition, the data anomaly detection method based on deformation model provided in the present application can also be applied to databases, servers and service response systems based on terminal artificial intelligence.

[0132] The embodiments of the present application do not impose any restrictions on the specific application scenarios of the data change detection method based on the deformation model.

[0133] As shown in FIG11 , the electronic device 100 provided in an embodiment of the present application includes a memory 101 and a processor 102 .

[0134] The memory 101 is used to store computer programs. Preferably, the memory 101 includes various media capable of storing program codes, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.

[0135] Specifically, the memory 101 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The electronic device 100 may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 101 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0136] The processor 102 is connected to the memory 101 and is used to execute the computer program stored in the memory 101 so that the electronic device 100 executes the data change detection method based on deformation model provided in any embodiment of the present application.

[0137] Optionally, the processor 102 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0138] Optionally, the electronic device 100 in this embodiment may further include a display 103. The display 103 is communicatively connected to the memory 101 and the processor 102, and is configured to display a GUI interface related to the deformation model-based data change detection method.

[0139] In summary, the data anomaly detection method based on the deformation model provided by the embodiment of the present application can effectively convert data anomalies into mathematical and logical anomalies. This method can quickly identify anomalies from time series data, thereby significantly improving the accuracy of anomaly identification. By applying two data conversion processes, this method not only enhances the ability to identify data anomalies, but also improves the efficiency and accuracy of the identification process. In addition, the reusability of this method has also been enhanced, allowing it to maintain efficient and accurate performance in different data scenarios.

[0140] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.

Claims

1. A data change detection method based on a deformation model, characterized in that: include: Acquire business data, and determine data time characteristics of the business data; Based on the data time characteristic, generating a reference benchmark curve corresponding to the historical business data, wherein the reference benchmark curve is used to characterize the performance of the historical business data on the data time characteristic; Based on the business data and the reference benchmark curve, calculating a benchmark deformation variable of the business data, wherein the benchmark deformation variable is used to characterize the difference between the business data and the reference benchmark curve; An abnormal movement alarm is determined based on the reference deformation variable.

2. The method for detecting data changes based on a deformation model according to claim 1, characterized in that: The data time characteristics include at least one of the data sampling period, holidays, activity days, and hot spots.

3. The method for detecting data changes based on a deformation model according to claim 1 or 2, characterized in that: If the data time characteristic includes a sampling period of data, determining the data time characteristic of the service data includes: Converting the service data into frequency domain data by Fourier transform formula; Normalizing the frequency domain data to obtain normalized frequency domain data; The sampling period is determined according to the normalized frequency domain data.

4. The method for detecting data changes based on a deformation model according to claim 2, characterized in that: The step of generating a reference benchmark curve corresponding to the historical business data based on the data time characteristic includes: Calculate the year-on-year data of the historical business data within the corresponding period according to the data time characteristics; The month-on-month data is plotted according to a time series to generate a reference benchmark curve.

5. The method for detecting data changes based on a deformation model according to claim 1, characterized in that: The calculating, based on the business data and the reference benchmark curve, a benchmark deformation variable of the business data comprises: Normalizing the business data to obtain normalized data; A reference deformation variable of the business data is calculated according to the normalized data and the reference reference curve.

6. The method for detecting data changes based on a deformation model according to claim 5, characterized in that: The normalizing the business data to obtain normalized data includes: The correlation between the business data and the data corresponding to the reference benchmark curve is calculated to achieve normalization processing of the business data, and the correlation is normalized data.

7. The method for detecting data changes based on a deformation model according to claim 6, characterized in that: The calculating the correlation between the business data and the data corresponding to the reference benchmark curve includes: The correlation between the business data and the data corresponding to the reference benchmark curve is calculated using the cosine similarity calculation formula.

8. The method for detecting data changes based on a deformation model according to claim 5, characterized in that: The base deformation variable of the business data is calculated by the following calculation formula: Reference deformation variable=(1-the normalized data)*|the business data-the corresponding data in the reference reference curve|.

9. The method for detecting data changes based on a deformation model according to any one of claims 1 to 8, characterized in that: The abnormal change alarm includes multiple alarms of different levels, and the abnormal change alarm is determined based on the reference deformation variable, including: If the reference deformation variable is greater than the first alarm threshold, determining the level of the abnormal movement alarm to be the most risky level; If the reference deformation variable is greater than the second alarm threshold and less than or equal to the first alarm threshold, the level of the abnormal change alarm is determined to be a concern level.

10. The method for detecting data changes based on a deformation model according to claim 9, characterized in that: The method further comprises: The first alarm threshold and / or the second alarm threshold are modified based on the result of the abnormal alarm.

11. A method for detecting data changes based on a deformation model, characterized in that: The method comprises: Obtain business data and determine the data time characteristics of the business data; Determining a reference benchmark curve for the business data based on the data time characteristics and historical business data; Calculating a reference deformation variable of the business data based on the business data and the reference reference curve; An abnormal movement alarm is determined based on the reference deformation variable.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting data changes based on a deformation model according to any one of claims 1 to 11 is implemented.

13. An electronic device, characterized in that: The electronic device comprises: a memory storing a computer program; and a processor communicating with the memory and executing the deformation model-based data change detection method according to any one of claims 1 to 11 when calling the computer program.

Images