Abnormal business data processing method, device and storage medium

Abstract

The present application provides a method, device and storage medium for processing abnormal business data, wherein the method comprises: obtaining multiple business data of a business, the cycle length T of each business data and the collection time of each business data; based on the time sequence position of the collection time of each business data in the data cycle of the business data, grouping the business data with the same time sequence position into an extended set; for each business data y i , the business data y i The acquisition time t i Before and after each W d Each business data in the extended set to which the business data corresponding to the collection time belongs, business data y i Each business data in the extended set and the collection time t i Before and after each W s The business data corresponding to the collection time constitutes the business data y i The extended neighborhood set N i ; Based on the extended neighborhood set corresponding to each business data, abnormal business data is determined from multiple business data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device and storage medium for processing abnormal business data. Background Art

[0002] Business problems are inevitable during the application process of a business (such as a website application). The troubleshooting process mainly involves identifying abnormal data from the business data generated during the business processing, analyzing the identified abnormal data, and then determining and resolving the business problem.

[0003] Currently, the method for determining abnormal data from all business data is usually the traditional K-Sigma outlier detection method. Specifically, Figure 1 As shown, the business data collection device 11 collects business data (such as the number of visits to a website) and sends multiple business data collected within a period of time (such as a week) and the collection time corresponding to each business data to the abnormal data identification device 12. The abnormality detection unit 121 in the abnormal data identification device 12 uses the following K-Sigma outlier detection method to determine the business data y j Is it abnormal data: Make sure it satisfies the formula |y j -μ j |>K·σ j Business data j is abnormal data. N j is the business data y j Neighborhood set of |N j | is the set N j The cardinality or composition set N j The number of business data v, K is a preset value (such as K = 3). j Neighborhood set N j To: j with y j The corresponding acquisition time t j Before and after each W s A data set consisting of business data corresponding to each collection time. s is the preset window value, and j is a natural number greater than zero.

[0004] When using existing methods to identify abnormal data, there is a problem of inaccurate abnormal data. Inaccurate abnormal data will result in the inability to identify business problems based on the abnormal data or to effectively solve business problems. Summary of the Invention

[0005] The present application provides a method, device and storage medium for processing abnormal business data to solve the problem of inaccurate abnormal data obtained when using existing methods to determine abnormal data.

[0006] In a first aspect, the present application provides a method for processing abnormal business data, comprising:

[0007] Acquire multiple business data of a business, the cycle length T of each business data, and the collection time of each business data; the cycle length is the number of business data included in the data cycle corresponding to the business data;

[0008] Based on the time sequence position of the collection time of each business data in the data cycle of the business data, the business data with the same time sequence position are grouped into an extended set;

[0009] For each business data y i , the business data y i The acquisition time t i Before and after each W d Each business data in the extended set to which the business data corresponding to the collection time belongs, business data y i Each business data in the extended set and the collection time t i Before and after each W s The business data corresponding to the collection time constitutes the business data y i The extended neighborhood set N i ;

[0010] Determine abnormal business data from multiple business data based on the extended neighborhood set corresponding to each business data;

[0011] Where i = 1, 2, 3, ..., n; 2 ≤ n and n is a natural number; W d and W s are all preset values, and W d <W s ; 2≤T and T is a natural number.

[0012] Optionally, the step of grouping the business data with the same time sequence position into an extended set based on the time sequence position of the collection time of each business data in the data cycle of the business data includes:

[0013] Each business data in the multiple business data arranged in time series is sequentially classified into the sequentially arranged extended set Z h In , we get T extended sets;

[0014] The collection time of each service data in each extended set has the same time sequence position in the data period to which the service data belongs; h=1, 2, 3, . . . , T.

[0015] Optionally, the step of grouping the business data with the same time sequence position into an extended set based on the time sequence position of the collection time of each business data in the data cycle of the business data includes:

[0016] Marking each piece of business data with a position identifier, where the position identifier is an identifier of the temporal position of the acquisition time of the business data in the data cycle of the business data;

[0017] The business data with the same location identifier are grouped into an extended set.

[0018] Optionally, determining abnormal business data from multiple business data based on the extended neighborhood set corresponding to each business data includes:

[0019] Based on the extended neighborhood set corresponding to each business data, abnormal business data is determined from the multiple business data in the following manner:

[0020] Make sure to satisfy the formula |y i -μ i |>K·σ i Business data i is abnormal data; among them, N i is the business data y i The extended neighborhood set of |N i | is the set N i The cardinality or composition set N i The number of business data v, K is a preset value.

[0021] Optionally, the cycle length T for obtaining each service data includes:

[0022] Based on the plurality of service data and the collection time of each service data, the cycle length T of each service data is determined in the following manner:

[0023] Arrange n business data in the order of their collection time to form a business data sequence X;

[0024] Calculate the Pearson correlation coefficient c between the sequence A consisting of the 1st business data to the nkth business data in the business data sequence X and the sequence B consisting of the k+1th business data to the nth business data in the business data sequence X. k , obtain n Pearson correlation coefficients c k ;

[0025] The Pearson correlation coefficients are calculated by the Pearson correlation coefficient c kThe numerical values ​​of k are arranged in sequence to form an autocorrelation sequence C; the numerical sequence is from large to small, or from small to large;

[0026] Based on the preset spike threshold C th , determine multiple peak values ​​from the autocorrelation sequence C as follows: Based on the Pearson correlation coefficient c k and c k Before and after each W p A peak set consisting of a Pearson correlation coefficient is determined to have the largest value in the peak set and no less than C th The Pearson correlation coefficient is a sharp peak;

[0027] Sort the position numbers q of the peak values ​​in the autocorrelation sequence C in ascending order according to their numerical values ​​to form a number set, and subtract two adjacent position numbers in the number set to obtain the absolute value of the difference between the two adjacent position numbers;

[0028] The absolute values ​​of the differences are combined into a difference sequence, and the median of the difference sequence is calculated to obtain the median of the difference sequence, where the median represents the period length T of each business data;

[0029] Among them, W p are preset values; i = 1, 2, 3, …, n; k = 0, 1, 2, 3, …, (n-1); q = 1, 2, 3, …, n; 2 ≤ n and n is a natural number.

[0030] In a second aspect, the present application provides an abnormal data processing device, comprising: a period determination unit and a data identification unit;

[0031] The cycle determination unit is used to obtain multiple business data of a business, the cycle length T of each business data and the collection time of each business data; the cycle length is the number of business data included in the data cycle corresponding to the business data;

[0032] The data identification unit is used to classify the business data with the same time sequence position into an extended set based on the time sequence position of the collection time of each business data in the data cycle of the business data; i , the business data y i The acquisition time t i Before and after each W d Each business data in the extended set to which the business data corresponding to the collection time belongs, business data y i Each business data in the extended set and the collection time t i Before and after each W s The business data corresponding to the collection time constitutes the business data yi The extended neighborhood set N i ; and based on the extended neighborhood set corresponding to each business data, determine abnormal business data from multiple business data;

[0033] Where i = 1, 2, 3, ..., n; 2 ≤ n and n is a natural number; W d and W s are all preset values, and W d <W s ; 2≤T and T is a natural number.

[0034] Optionally, the period determination unit includes a processing module, an autocorrelation calculation module, a peak search module, and a period determination module, and the data identification unit includes a data grouping module, a neighborhood determination module, and an anomaly detection module;

[0035] The processing module is used to obtain multiple business data of a business, the cycle length T of each business data and the collection time of each business data; it is also used to obtain multiple business data of a business and the collection time of each business data;

[0036] The autocorrelation calculation module is used to arrange n business data in the order of the collection time of each business data to form a business data sequence X, and then calculate the Pearson correlation coefficient between the sequence A consisting of the 1st business data to the nth business data in the business data sequence X and the sequence B consisting of the k+1th business data to the nth business data in the business data sequence, to obtain n Pearson correlation coefficients c k ; and the Pearson correlation coefficients are calculated according to the Pearson correlation coefficient c k The numerical values ​​of k are arranged in sequence to form an autocorrelation sequence C; the numerical sequence is from large to small, or from small to large;

[0037] The peak search module is used to search for peaks based on a preset peak threshold C th , determine multiple peak values ​​from the autocorrelation sequence C as follows: Based on the Pearson correlation coefficient c k and c k Before and after each W p A peak set consisting of a Pearson correlation coefficient is determined to have the largest value in the peak set and no less than C th The Pearson correlation coefficient is a sharp peak;

[0038] The period determination module is configured to sort the position numbers q of the peak values ​​in the autocorrelation sequence C in ascending order according to numerical value to form a sequence number set, subtract two adjacent position numbers in the sequence number set to obtain the absolute value of the difference between the two adjacent position numbers; and form a difference sequence based on the absolute values ​​of the differences, perform median calculation on the difference sequence to obtain the median of the difference sequence, where the median represents the period length T of each service data;

[0039] The data grouping module is configured to group the service data having the same time sequence position into an extended set based on the time sequence position of the collection time of each service data in the data cycle of the service data;

[0040] The neighborhood determination module is used to determine each service data y i , the business data y i The acquisition time t i Before and after each W d Each business data in the extended set to which the business data corresponding to the collection time belongs, business data y i Each business data in the extended set and the collection time t i Before and after each W s The business data corresponding to the collection time constitutes the business data y i The extended neighborhood set N i ;

[0041] The anomaly detection module is configured to determine abnormal business data from a plurality of business data based on an extended neighborhood set corresponding to each business data;

[0042] Among them, W p are preset values; i = 1, 2, 3, …, n; k = 0, 1, 2, 3, …, (n-1); q = 1, 2, 3, …, n; 2 ≤ n and n is a natural number.

[0043] In a third aspect, the present application provides an abnormal data processing device, comprising:

[0044] processor and memory;

[0045] The memory stores executable instructions executable by the processor;

[0046] The processor executes the executable instructions stored in the memory, so that the processor performs the method described above.

[0047] In a fourth aspect, the present application provides a storage medium, wherein the storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the method described above when executed by a processor.

[0048] In a fifth aspect, the present application provides a program product, comprising a computer program, which implements the method described above when executed by a processor.

[0049] The method, device, and storage medium for processing abnormal business data provided by this application periodically expand the neighborhood set of each business data determined by the prior art method to obtain an expanded neighborhood set of each business data, and then determine accurate abnormal business data from multiple business data based on the expanded neighborhood set of each business data. In the process of determining abnormal business data, the method provided by this application filters out periodic mutation data from periodic business data, ensuring the accuracy of the determined abnormal business data and solving the problem of inaccurate abnormal data determined by the prior art method. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] Figure 1 Determine the system architecture diagram for the anomaly data of the existing technology;

[0052] Figure 2 A schematic diagram of website visits to a website within a week provided in an embodiment of the present application;

[0053] Figure 3 This is a diagram of the system architecture for processing abnormal business data provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a method for processing abnormal business data provided in an embodiment of the present application;

[0055] Figure 5 A structural diagram of an abnormal data processing device provided in an embodiment of the present application;

[0056] Figure 6 This is a structural diagram of the abnormal data processing device provided in an embodiment of the present application.

[0057] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0058] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0059] Figure 1 Determine the system architecture diagram for the abnormal data of the existing technology. Figure 1 As shown, business data collection device 11 collects business data and transmits multiple business data collected over a period of time, along with the corresponding collection time of each business data, to abnormal data identification device 12. Anomaly detection unit 121 in abnormal data identification device 12 uses the K-Sigma outlier detection method to identify abnormal data from the multiple business data. Anomaly detection unit 121 transmits the identified abnormal data to processing device 10, which processes the abnormal data to address business issues related to the business or business system.

[0060] For example, when a website encounters a website access problem during application, abnormal data is usually determined from business data (such as website visits) for a time period (such as a week) including the time when the website problem occurred. Then, the business problem is determined and solved based on the abnormal data. Figure 2 This is a diagram of website visits to a website within a week provided by the embodiment of this application. Figure 2 As shown, the website visit data is periodic data with one day as a cycle, and there will be sudden changes in the data at certain fixed times of each day (such as Figure 2 The periodic mutation data is generated when the website is maintained every day. It is normal business data and does not indicate that there is a business problem with the corresponding business (ie, the website application).

[0061] However, if Figure 1 As shown, the existing method for determining abnormal data is based on business data y j Neighborhood set N j , the K-Sigma outlier detection method is used to determine the business data y j Whether it is abnormal data, the neighborhood set N j The periodicity of business data is not taken into account.

[0062] When using the existing technology method, determine Figure 2 When abnormal data is found in the periodic business data, the following is usually Figure 2The periodic mutation data shown in FIG is determined to be abnormal data. That is, the method of the prior art is used to determine the abnormal data from the periodic mutation data shown in FIG. Figure 2 The abnormal data is determined from the business data shown in the figure. The abnormal data is obtained by Figure 2 The data shown in Figure 1 is composed of multiple periodic mutation data and a real abnormal data. Among the abnormal data determined, only Figure 2 The real abnormal data shown is data that is associated with or can represent business problems, while the periodic mutation data is redundant data that is not associated with business problems. Therefore, when the prior art method is used to determine abnormal data for periodic business data, the abnormal data obtained is not accurate. The inaccuracy of the abnormal data will result in the inability to determine the business problem or the inability to efficiently solve the business problem based on the abnormal data. In addition, if technical personnel are required to analyze the abnormal data determined above to determine the business problem and solve the business problem of the business or the business system, the inaccuracy of the abnormal data will cause great trouble to the technical personnel's analysis and determination of the solution to the business problem, and greatly reduce the efficiency and accuracy of the technical personnel in analyzing and solving the business problem.

[0063] In this regard, the present application proposes a method for processing abnormal business data. By periodically expanding the neighborhood set of each business data, an extended neighborhood set of each business data is obtained. Abnormal business data is determined based on the extended neighborhood set of each business data. The determined abnormal business data does not contain periodic mutation data (i.e., the periodic mutation data has been filtered out), thereby ensuring that the obtained abnormal business data is accurate data associated with the business problem. When abnormal business data is used to analyze and determine the solution to business problems, it will greatly improve the efficiency and accuracy of business problem analysis and solution work, and reduce the amount of data used for business problem analysis and solution.

[0064] The following describes the method for processing abnormal business data provided by this application in conjunction with some embodiments.

[0065] Figure 3 This is a diagram of the system architecture for processing abnormal business data provided by the embodiment of this application. Figure 3 As shown, the system includes: a processing device 10 , a business data collection device 11 and an abnormal data processing device 13 connected thereto, and the abnormal data processing device 13 includes a period determination unit 131 and a data identification unit 132 .

[0066] For example, the business data collection device 11 collects business data for a business and sends multiple business data items, the cycle length T of each business data item, and the collection time of each business data item to the cycle determination unit 131 in the abnormal data processing device 13. The cycle length T is the number of business data items included in the data cycle corresponding to the business data item. The cycle determination unit 131 then sends the obtained multiple business data items, the cycle length T of each business data item, and the collection time of each business data item to the data identification unit 132.

[0067] The data identification unit 132 classifies each business data with the same time sequence position into an extended set based on the time sequence position of the collection time of each business data in the data cycle of the business data. i , the business data y i The acquisition time t i Before and after each W d Each business data in the extended set to which the business data corresponding to the collection time belongs, business data y i Each business data in the extended set and the collection time t i Before and after each W s The business data corresponding to the collection time constitutes the business data y i The extended neighborhood set N i Then, the data identification unit 132 determines abnormal business data from the multiple business data based on the extended neighborhood set corresponding to each business data. The data identification unit 132 sends the abnormal business data to the processing device 10. The processing device 10 processes the business problem of the business or business system based on the obtained abnormal business data.

[0068] Where i = 1, 2, 3, ..., n; 2 ≤ n and n is a natural number; W d and W s are all preset values, and W d <W s ; 2≤T and T is a natural number.

[0069] Optionally, after the business data collection device 11 collects business data for a business, the business data collection device 11 may only send the multiple business data of the business and the collection time of each business data to the cycle determination unit 131 in the abnormal data processing device 13. The cycle determination unit 131 determines the cycle length T of each business data based on the obtained multiple business data and the collection time of each business data. The cycle determination unit 131 sends the obtained multiple business data and the collection time of each business data, as well as the determined cycle length T of each business data, to the data identification unit 132.

[0070] The method for processing abnormal business data provided by the embodiment of the present application classifies each business data with the same time sequence position into an extended set, and based on the extended set to which each business data belongs, determines the extended neighborhood set of each business data, and then determines the accurate abnormal business data from multiple business data based on the extended neighborhood set corresponding to each business data. When the method for processing abnormal business data provided by the present application is used to determine the abnormal business data from multiple business data, the periodic mutation data is filtered out by extending the neighborhood set, thereby avoiding the periodic mutation data from being determined as abnormal data and ensuring the accuracy of the determined abnormal business data. The method for processing abnormal business data provided by the embodiment of the present application solves the problem of inaccurate abnormal data obtained when the method of the prior art is used to determine abnormal data.

[0071] The following combination Figure 4 The method for processing abnormal business data provided in the embodiment of the present application is described. Figure 4 Schematic diagram of the method for processing abnormal business data provided by the embodiment of this application. The execution subject of the embodiment of this application is Figure 3 The abnormal data processing device 13 in the embodiment shown. Figure 4 As shown, the method includes:

[0072] S401 , obtaining multiple business data of a business, a cycle length T of each business data, and a collection time of each business data; the cycle length is the number of business data included in the data cycle corresponding to the business data.

[0073] Specifically, the cycle determination unit 131 in the abnormal data processing device 13 obtains multiple business data of a business, the cycle length T of each business data, and the collection time of each business data from the business data collection device 11. The cycle length is the number of business data included in the data cycle corresponding to the business data; 2≤T, and T is a natural number.

[0074] Optionally, the cycle determination unit 131 may also obtain multiple business data of a business and the collection time of each business data from the business data collection device 11, and determine the cycle length T of each business data based on the obtained multiple business data and the collection time of each business data.

[0075] Next, the cycle determining unit 131 sends the obtained multiple service data, the cycle length T of each service data, and the collection time of each service data to the data identifying unit 132 .

[0076] The cycle determining unit 131 determines the cycle length T of each service data based on the obtained multiple service data and the collection time of each service data. For example, the cycle determining unit 131 determines the cycle length T of each service data as follows:

[0077] The cycle determination unit 131 determines the cycle length T of each piece of service data based on the plurality of service data and the collection time of each piece of service data in the following manner. Assuming that the plurality of service data is specifically n pieces of service data, the cycle determination unit 131 determines the cycle length T of each piece of service data based on the n pieces of service data and the collection time of each piece of service data in steps S4011-4016.

[0078] S4011. Arrange n pieces of business data in the order of their collection time to form a business data sequence X.

[0079] Exemplarily, the period determination unit 131 arranges n pieces of service data in the order of their collection time to form a service data sequence X as shown in Table 1.

[0080] Table 1 Business data sequence X

[0081] The sorting number i of the business data in X 1 2 3 4 … n <![CDATA[Business data y i > <![CDATA[y1]]> <![CDATA[y2]]> <![CDATA[y3]]> <![CDATA[y4]]> … <![CDATA[y n ]]>

[0082] S4012. Calculate the Pearson correlation coefficient c between the sequence A consisting of the 1st to the nth business data in the business data sequence X and the sequence B consisting of the k+1th to the nth business data in the business data sequence X. k , obtain n Pearson correlation coefficients c k Where, i = 1, 2, 3, ..., n; k = 0, 1, 2, 3, ..., (n-1); 2 ≤ n and n is a natural number.

[0083] For example, the period determination unit 131 calculates the Pearson correlation coefficient c between sequence A and sequence B according to the following formulas (1) and (2): k :

[0084]

[0085]

[0086] Where m is the number of business data in sequence A or sequence B; a f is the business data in sequence A, b f is the business data in sequence B; f=1,2,3,…,m; 1≤m and m is a natural number.

[0087] S4013, each Pearson correlation coefficient is calculated according to the Pearson correlation coefficient c k The numerical values ​​of k in are arranged in sequence to form the autocorrelation sequence C.

[0088] For example, the period determination unit 131 converts each Pearson correlation coefficient into a Pearson correlation coefficient c. kThe numerical values ​​of k are arranged in sequence to form an autocorrelation sequence C. An example of the autocorrelation sequence C is shown in Table 2. The numerical sequence is the order from large to small, or the order from small to large.

[0089] Table 2 Autocorrelation sequence C

[0090]

[0091] Wherein, q=1, 2, 3,…, n; 2≤n and n is a natural number.

[0092] S4014, based on the preset peak threshold C th , determine multiple peak values ​​from the autocorrelation sequence C as follows: Based on the Pearson correlation coefficient c k and c k Before and after each W p The peak set composed of Pearson correlation coefficients is determined to have the largest value in the peak set and no less than C th The Pearson correlation coefficient is a sharp peak. p is the default value.

[0093] For example, the period determination unit 131 determines whether each Pearson correlation coefficient in the autocorrelation sequence C is a peak value in the following manner: based on the Pearson correlation coefficient c k and c k Before and after each W p The peak set composed of Pearson correlation coefficients is determined to have the largest value in the peak set and no less than C th The Pearson correlation coefficient is a sharp peak.

[0094] For example, suppose W p =1, the period determination unit 131 determines whether the Pearson correlation coefficient shown in Table 2 is a peak value in the following manner:

[0095] C0 and the Pearson correlation coefficients before and after c0 (i.e., c0, c1) are combined into a peak set [c0, c1], and the peak set [c0, c1] with the largest value that is not less than C is determined. th The Pearson correlation coefficient is a sharp peak;

[0096] C1 and the Pearson correlation coefficients before and after c1 (i.e., c0, c1, c2) are combined into a peak set [c0, c1, c2], and the peak set [c0, c1, c2] with the largest value that is not less than C is determined. th The Pearson correlation coefficient is a sharp peak;

[0097] C2 and the Pearson correlation coefficients before and after c2 (i.e., c1, c2, c3) are combined into a peak set [c1, c2, c3], and the peak set [c1, c2, c3] with the largest value that is not less than C is determined. th The Pearson correlation coefficient is a sharp peak;

[0098] …;

[0099] Similarly, the process continues until it is determined whether each Pearson correlation coefficient in the autocorrelation sequence C is a peak value.

[0100] S4015 , sorting the position numbers q of the peak values ​​in the autocorrelation sequence C in ascending order according to their numerical values ​​to form a number set, and subtracting two adjacent position numbers in the number set to obtain the absolute value of the difference between the two adjacent position numbers.

[0101] For example, if the period determination unit 131 determines in step S4014 that the peak values ​​in the autocorrelation sequence C shown in Table 2 are c0, c4, c8, c 10 The peak values ​​c0, c4, c8, c 10 The position numbers 1, 5, 9, and 11 in the autocorrelation sequence C are sorted in ascending order according to their numerical values ​​to form the sequence number set [1, 5, 9, 11]. The two adjacent position numbers in the sequence number set [1, 5, 9, 11] are subtracted to obtain the absolute value of the difference between the two adjacent position numbers, which is 4, 4, and 2.

[0102] Optionally, the value of the position number q can be the Pearson correlation coefficient c of the autocorrelation sequence C. k The same value of k in .

[0103] S4016. The absolute values ​​of the differences are combined into a difference sequence, and the median of the difference sequence is calculated to obtain the median of the difference sequence. The median represents the period length T of each business data.

[0104] Exemplarily, the period determination unit 131 forms a difference sequence [4, 4, 2] with the absolute values ​​of the difference between the sequence numbers of two adjacent positions determined in step S4015, 4, 4, and 2, and performs median calculation on the difference sequence [4, 4, 2] to obtain a median of 4 for the difference sequence [4, 4, 2]. The median represents the period length T of each business data, that is, T=4.

[0105] S402 : Based on the time sequence position of the collection time of each business data in the data cycle of the business data, group the business data with the same time sequence position into an extended set.

[0106] Exemplarily, after the data identification unit 132 receives multiple business data, the cycle length T of each business data, and the collection time of each business data from the cycle determination unit 131, the data identification unit 132 groups the business data with the same time sequence position into an extended set based on the time sequence position of the collection time of each business data in the data cycle of the business data.

[0107] Optionally, the data identification unit 132 may sequentially classify each of the plurality of service data arranged in time sequence into an extended set Z arranged in sequence. h In this example, T extended sets are obtained. The acquisition time of each service data in each extended set is the same in the time sequence position in the data cycle to which the service data belongs; h = 1, 2, 3, ..., T.

[0108] For example, the data identification unit 132 obtains multiple service data and arranges the service data in the order of the collection time of each amateur data to obtain the service data sequence X shown in Table 1. Assuming that the cycle length of each service data is T=4 and n=12, the data identification unit 132 sequentially classifies each service data in the service data sequence X into the sequentially arranged extended set Z h In the above example, we get T extended sets as shown in Table 3.

[0109] Table 3 Extended set Z h

[0110] <![CDATA[Extended set Z1]]> <![CDATA[[y1,y5,y9]]]> <![CDATA[Extended set Z2]]> <![CDATA[[y2,y6,y 10 ]]]> <![CDATA[Extended set Z3]]> <![CDATA[[y3,y7,y 11 ]]]> <![CDATA[Extended set Z4]]> <![CDATA[[y4,y8,y 12 ]]]>

[0111] Optionally, the data identification unit 132 may further mark each service data with a location identifier, wherein the location identifier is an identifier of the temporal position of the collection time of the service data in the data cycle of the service data. The data identification unit 132 then groups the service data with the same location identifier into an extended set.

[0112] For example, the data identification unit 132 obtains multiple service data in the service data sequence X shown in Table 1. Assuming that the service data in the service data sequence X shown in Table 1 are arranged in time sequence, and the period length of each service data is T=4, and n=12, the data identification unit 132 marks each service data with a location identifier as shown in Table 4, and groups the service data with the same location identifier into an extended set, thereby obtaining the extended set shown in Table 3.

[0113] Table 4 Data cycle and location identifier of each business data

[0114]

[0115] The length of the data cycle of each service data is as shown in Table 4. The lengths of data cycle 1, data cycle 2, and data cycle 3 are all the same.

[0116] S403, for each business data y i , the business data y i The acquisition time t i Before and after each W d Each business data in the extended set to which the business data corresponding to the collection time belongs, business data y i Each business data in the extended set and the collection time t i Before and after each W s The business data corresponding to the collection time constitutes the business data y i The extended neighborhood set N i .

[0117] Specifically, the data identification unit 132 identifies each business data y i , the business data y i The acquisition time t i Before and after each W d Each business data in the extended set to which the business data corresponding to the collection time belongs, business data y i Each business data in the extended set and the collection time t i Before and after each W s The business data corresponding to the collection time constitutes the business data y i The extended neighborhood set N i Where i = 1, 2, 3, ..., n; 2 ≤ n and n is a natural number; W d and W s are all preset values, and W d <W s .

[0118] For example, it is assumed that the multiple business data obtained by the data identification unit 132 are as shown in Table 4, T=4, W d =0,W s =1, and the extended set corresponding to each service data is shown in Table 3. Taking the extended neighborhood set N2 of service data y2 as an example, the data identification unit 132 identifies each service data in the extended set corresponding to the service data of 0 acquisition times before and after the acquisition time t2 of service data y2, and each service data in the extended set Z2 to which service data y2 belongs (i.e., y2, y6, y 10 ) and the business data corresponding to the acquisition time before and after the acquisition time t2 (i.e., y1 and y3), constitute the extended neighborhood set N2 of the business data y2: [y2, y6, y 10, y1, y3].

[0119] In contrast, if the existing technology is used, with the same business data, T=4, W s = 1 and other parameter conditions are the same, the neighborhood set N of the business data y2 is determined 2c is [y2, y1, y3].

[0120] S404: Determine abnormal business data from the plurality of business data based on the extended neighborhood set corresponding to each business data.

[0121] Exemplarily, the data identification unit 132 determines abnormal business data from a plurality of business data based on the extended neighborhood set corresponding to each business data.

[0122] For example, the data identification unit 132 determines abnormal business data from the plurality of business data based on the extended neighborhood set corresponding to each business data in the following manner:

[0123] The data identification unit 132 determines the business data y that satisfies formula (3) i For abnormal data,

[0124] |y i -μ i |>K·σ i (3);

[0125] in, N i is the business data y i The extended neighborhood set of |N i | is the set N i The cardinality or composition set N i The number of business data v, K is a preset value.

[0126] Furthermore, the data identification unit 132 sends the determined abnormal business data to the processing device 10. The processing device 10 processes the business problem of the business or business system based on the obtained abnormal business data.

[0127] As shown in step S403, the extended neighborhood set N2 of the service data y2 determined by the method of the present application is [y2, y6, y 10 , y1, y3]. The neighborhood set N of the business data y2 determined by the existing technology method 2c =[y2, y1, y3]. Assume that the specific values ​​of the business data shown in Table 4 are as shown in Table 5 below.

[0128] Table 5 Business data

[0129]

[0130] As can be seen from Table 5, y2 is periodic mutation data, and the business data y1 and y3 before and after y2 are normal data. When the processing method of abnormal business data provided by the present application is adopted, based on the extended neighborhood set N2 of y2, it can be determined that y2 is not abnormal business data; however, when the method of the prior art is used to determine abnormal data, based on the neighborhood set N 2c , it is determined that y2 is abnormal data.

[0131] The following examples provide a detailed comparison of the technical effects achieved by the method for processing abnormal business data provided by this application and the prior art methods. Assume that the time-series business data and related information generated by business D in time period L are shown in Table 6.

[0132] Table 6 Business data and related information generated by business D in time period L

[0133]

[0134] When the abnormal business data processing method provided by the present application and the method of the prior art are used to determine the abnormal business data from the business data shown in Table 6, the results shown in Table 7 are obtained.

[0135] Table 7 Abnormal business data determined by different processing methods

[0136]

[0137] As can be seen from Tables 6 and 7, the business data "-3" in Table 6 is abnormal business data associated with the business problem encountered by business D, while the other business data in Table 6 is normal business data not associated with the business problem encountered by business D. Therefore, from the abnormal business data results determined in Table 7, it can be seen that the abnormal business data processing method provided by this application can more accurately determine abnormal business data associated with business problems.

[0138] Therefore, the method for processing abnormal business data provided by this application periodically expands the neighborhood set of each business data to obtain the expanded neighborhood set of each business data. Therefore, when using the method for processing abnormal business data provided by this application, periodic mutation data can be filtered out when determining abnormal business data. The method for processing abnormal business data provided by this application avoids mistaking periodic mutation data for abnormal data associated with business problems, thereby interfering with the identification and resolution of business problems.

[0139] The method for processing abnormal business data provided by the present application, by periodically expanding the neighborhood set of each business data, obtaining the extended neighborhood set of each business data, determining the abnormal business data based on the extended neighborhood set of each business data, and obtaining abnormal business data that is associated with the business problem and accurate. The method for processing abnormal business data provided by the present application greatly improves the efficiency and accuracy of business problem analysis and solution. In addition, in the case of periodic information such as data period and period length of unknown business data, the method for processing abnormal business data provided by the present application can also be used to determine the periodic information of the business data.

[0140] The present application also provides an abnormal data processing device. Figure 3 and Figure 5 The abnormal data processing device provided in the embodiment of the present application is described. Figure 5 This is a structural diagram of the abnormal data processing device provided in the embodiment of this application. Figure 3 As shown, the abnormal data processing device 13 includes: a period determination unit 131 and a data identification unit 132;

[0141] The cycle determination unit 131 is used to obtain multiple business data of a business, the cycle length T of each business data and the collection time of each business data; wherein the cycle length is the number of business data included in the data cycle corresponding to the business data.

[0142] The data identification unit 132 is used to group the business data with the same time sequence position into an extended set based on the time sequence position of the collection time of each business data in the data cycle of the business data; i , the business data y i The acquisition time t i Before and after each W d Each business data in the extended set to which the business data corresponding to the collection time belongs, business data y i Each business data in the extended set and the collection time t i Before and after each W s The business data corresponding to the collection time constitutes the business data y i The extended neighborhood set N i ; and based on the extended neighborhood set corresponding to each business data, determine the abnormal business data from the multiple business data; where i = 1, 2, 3, ..., n; 2 ≤ n and n is a natural number; W d and W s are all preset values, and W d <W s ; 2≤T and T is a natural number.

[0143] Alternatively, as Figure 5 As shown, the period determination unit 131 includes a processing module 1311 , an autocorrelation calculation module 1312 , a peak search module 1313 , and a period determination module 1314 .

[0144] The data identification unit 132 includes a data grouping module 1321 , a neighborhood determination module 1322 , and an anomaly detection module 1323 .

[0145] The processing module 1311 is used to obtain multiple business data of a business, the cycle length T of each business data and the collection time of each business data; and is also used to obtain multiple business data of a business and the collection time of each business data.

[0146] The autocorrelation calculation module 1312 is used to arrange n business data in the order of their collection time to form a business data sequence X, and then calculate the Pearson correlation coefficient between the sequence A consisting of the 1st business data to the nth business data in the business data sequence X and the sequence B consisting of the k+1th business data to the nth business data in the business data sequence, to obtain n Pearson correlation coefficients c k ; and the Pearson correlation coefficients are calculated according to the Pearson correlation coefficient c k The numerical values ​​of k are arranged in sequence to form an autocorrelation sequence C. The numerical sequence is the order from large to small, or the order from small to large.

[0147] The peak search module 1313 is used to search for peaks based on a preset peak threshold C th , determine multiple peak values ​​from the autocorrelation sequence C as follows: Based on the Pearson correlation coefficient c k and c k Before and after each W p The peak set composed of Pearson correlation coefficients is determined to have the largest value in the peak set and no less than C th The Pearson correlation coefficient is a sharp peak.

[0148] Period determination module 1314 is configured to sort the position numbers q of each peak value in the autocorrelation sequence C in ascending order to form a set of numbers, subtract two adjacent position numbers in the set to obtain the absolute value of the difference between the two adjacent position numbers, and form a difference sequence based on the absolute values ​​of the differences. The median of the difference sequence is then calculated to obtain the median of the difference sequence. The median represents the period length T of each service data.

[0149] The data grouping module 1321 is configured to group the service data having the same time sequence position into an extended set based on the time sequence position of the collection time of each service data in the data cycle of the service data.

[0150] Neighborhood determination module 1322, for each business data y i , the business data y i The acquisition time t i Before and after each W d Each business data in the extended set to which the business data corresponding to the collection time belongs, business data y i Each business data in the extended set and the collection time t i Before and after each W s The business data corresponding to the collection time constitutes the business data y i The extended neighborhood set N i .

[0151] The anomaly detection module 1323 is configured to determine abnormal business data from a plurality of business data based on an extended neighborhood set corresponding to each business data.

[0152] Among them, W p are preset values; i = 1, 2, 3, …, n; k = 0, 1, 2, 3, …, (n-1); q = 1, 2, 3, …, n; 2 ≤ n and n is a natural number.

[0153] The specific implementation principle and technical effect of the abnormal data processing device provided in the embodiment of the present application are similar to Figure 4 The specific implementation principles and technical effects of the illustrated embodiment are similar and will not be described in detail herein.

[0154] An embodiment of the present application also provides an abnormal data processing device. Figure 6 This is a structural diagram of the abnormal data processing device provided in the embodiment of the present application. Figure 6As shown, the device includes a processor 61 and a memory 62. The memory 62 stores instructions executable by the processor 61, enabling the processor 61 to execute the technical solution of the above-mentioned method embodiment. The implementation principles and technical effects are similar and will not be described in detail in this embodiment. It should be understood that the processor 61 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention can be directly implemented by a hardware processor or by a combination of hardware and software modules in the processor. The memory 62 may include high-speed random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. It can also be a USB flash drive, a mobile hard drive, a read-only memory, a magnetic disk, or an optical disk.

[0155] The present application embodiment also provides a storage medium, in which computer execution instructions are stored, and when these computer execution instructions are executed by a processor, the above-mentioned method for processing abnormal business data is implemented. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (English: Static Random-Access Memory, referred to as: SRAM), electrically erasable programmable read-only memory (English: Electrically-Erasable Programmable Read-Only Memory, referred to as: EEPROM), erasable programmable read-only memory (English: Erasable Programmable Read-Only Memory, referred to as: EPROM), programmable read-only memory (English: Programmable Read-Only Memory, referred to as: PROM), read-only memory (English: Read-Only Memory, referred to as: ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0156] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0157] An embodiment of the present application also provides a program product, such as a computer program, which, when executed by a processor, implements the method for processing abnormal business data covered by the present application.

[0158] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing abnormal business data, characterized in that: include: Obtain multiple business data of a business, the cycle length T of each business data, and the collection time of each business data; The cycle length is the number of business data contained in the data cycle corresponding to the business data; Based on the time sequence position of the collection time of each business data in the data cycle of the business data, the business data with the same time sequence position are grouped into an extended set; For each business data i , business data Collection time Before and after Each business data in the extended set to which the business data corresponding to the collection time belongs, business data Business data in the extended set and collection time Before and after The business data corresponding to the collection time constitutes the business data The extended neighborhood set of ; Based on the extended neighborhood set corresponding to each business data, periodic mutation data is filtered to determine abnormal business data from multiple business data; in, ; and is a natural number; and are preset values, and ; and is a natural number; The cycle length for obtaining each service data ,include: Based on the multiple business data and the collection time of each business data, the cycle length of each business data is determined in the following way : Will The business data are arranged in the order of their collection time to form a business data sequence ; Calculate the business data sequence The first business data to the A sequence of business data , and the business data sequence Middle Business data to The Pearson correlation coefficient of the sequence B composed of business data ,get Pearson correlation coefficient c k ; The Pearson correlation coefficients are calculated according to the Pearson correlation coefficients. middle The numerical order of the autocorrelation sequence is arranged ; The numerical order is the order from large to small, or the order from small to large; Based on preset spike threshold , determine multiple peak values ​​from the autocorrelation sequence C as follows: Based on the Pearson correlation coefficient as well as Before and after A peak set consisting of a Pearson correlation coefficient is determined to have the largest value and no less than The Pearson correlation coefficient is a sharp peak; Sort the position numbers q of the peak values ​​in the autocorrelation sequence C in ascending order according to their numerical values ​​to form a number set, and subtract two adjacent position numbers in the number set to obtain the absolute value of the difference between the two adjacent position numbers; The absolute values ​​of the differences are combined into a difference sequence, and the median of the difference sequence is calculated to obtain the median of the difference sequence, where the median represents the period length T of each business data; in, is the default value; ; , ; ; And n is a natural number; The method of grouping the business data having the same time sequence position as the data cycle of the business data based on the time sequence position of the collection time of the business data into an extended set includes: Each business data in the multiple business data arranged in time series is sequentially classified into the sequentially arranged extended set Z h In , we get T extended sets; The collection time of each service data in each extended set is the same as the time sequence position in the data period to which the service data belongs. ; Based on the extended neighborhood set corresponding to each business data, abnormal business data is determined from multiple business data, including: Based on the extended neighborhood set corresponding to each business data, abnormal business data is determined from the multiple business data in the following manner: Determine if the formula is satisfied Business data i is abnormal data; among them, , , is the business data y i The extended neighborhood set of For collection The cardinality or composition of Business data The number of K is the preset value.

2. An abnormal data processing device, characterized in that: include: a period determination unit and a data identification unit; The cycle determination unit is used to obtain multiple business data of a business, the cycle length T of each business data and the collection time of each business data; The cycle length is the number of business data contained in the data cycle corresponding to the business data; The data identification unit is used to classify the business data with the same time sequence position into an extended set based on the time sequence position of the collection time of each business data in the data cycle of the business data; , business data Collection time Before and after Each business data in the extended set to which the business data corresponding to the collection time belongs, business data Business data in the extended set and collection time Before and after The business data corresponding to the collection time constitutes the business data The extended neighborhood set of ; Based on the extended neighborhood set corresponding to each business data, filter the periodic mutation data and determine the abnormal business data from multiple business data; in, ; And n is a natural number; and are preset values, and ; And T is a natural number; The period determination unit includes a processing module, an autocorrelation calculation module, a peak search module, and a period determination module, and the data identification unit includes a data grouping module, a neighborhood determination module, and an anomaly detection module; The processing module is used to obtain multiple business data of a business, the cycle length T of each business data and the collection time of each business data; it is also used to obtain multiple business data of a business and the collection time of each business data; The autocorrelation calculation module is used to arrange n business data in the order of the collection time of each business data to form a business data sequence X, and then calculate the autocorrelation between the first business data and the second business data in the business data sequence X. A sequence A consisting of business data, and the first The Pearson correlation coefficient of the sequence B composed of the business data to the nth business data is obtained, and n Pearson correlation coefficients c are obtained. k ; and the Pearson correlation coefficients are calculated according to the Pearson correlation coefficient c k The numerical values ​​of k are arranged in sequence to form an autocorrelation sequence C; the numerical sequence is from large to small, or from small to large; The peak search module is used to search for peaks based on a preset peak threshold. , determine multiple peak values ​​from the autocorrelation sequence C as follows: Based on the Pearson correlation coefficient c k and c k Before and after A peak set consisting of a Pearson correlation coefficient is determined to have the largest value and no less than The Pearson correlation coefficient is a sharp peak; The period determination module is configured to sort the position numbers q of the peak values ​​in the autocorrelation sequence C in ascending order according to numerical value to form a sequence number set, subtract two adjacent position numbers in the sequence number set to obtain the absolute value of the difference between the two adjacent position numbers; and form a difference sequence based on the absolute values ​​of the differences, perform median calculation on the difference sequence to obtain the median of the difference sequence, where the median represents the period length T of each service data; The data identification unit includes a data grouping module, a neighborhood determination module and an anomaly detection module; The data grouping module is configured to group the service data having the same time sequence position into an extended set based on the time sequence position of the collection time of each service data in the data cycle of the service data; The anomaly detection module is configured to determine abnormal business data from a plurality of business data based on the extended neighborhood set corresponding to each business data in the following manner: Determine if the formula is satisfied Business data i is abnormal data; among them, , , is the business data y i The extended neighborhood set of For collection The cardinality or composition of Business data The number of K is the preset value.

3. An abnormal data processing device, characterized in that: include: processor and memory; The memory stores executable instructions executable by the processor; The processor executes the executable instructions stored in the memory, so that the processor performs the method according to claim 1.

4. A storage medium, characterized in that The storage medium stores computer-executable instructions, which are used to implement the method according to claim 1 when executed by a processor.

5. A program product comprising a computer program, which implements the method according to claim 1 when executed by a processor.

Images