Data anomaly detection method and device, equipment and storage medium

Abstract

The invention discloses a data anomaly detection method and device, equipment and a storage medium, and relates to the technical field of big data. The method comprises the steps of preprocessing initial business data extracted from a banking business system according to banking business parameters to obtain to-be-detected business data; according to the initial centroid, performing clustering analysis on the to-be-detected service data to obtain an initial clustering analysis result; under the condition that the initial clustering analysis result meets a clustering number adjustment condition, determining a clustering number adjustment frequency; adjusting the clustering number to obtain an adjusted clustering number; and determining a candidate centroid according to the adjusted clustering number, and performing clustering analysis on the to-be-detected service data again until the candidate centroid meets a centroid change rate condition, and obtaining a target clustering analysis result, thereby determining an anomaly detection result. According to the scheme, the clustering analysis is carried out by adopting a self-adaptive clustering number adjustment mode, so that the abnormal data is determined, and the data anomaly detection efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology and can be applied to the field of financial technology, especially to the field of big data technology. Specifically, it relates to a data anomaly detection method, apparatus, device, and storage medium. Background Technology

[0002] In the field of big data processing, data contains a wealth of information, but it also contains anomalous data that may affect the normal operation of the system. Therefore, rapid anomaly detection has become particularly important in the data field.

[0003] Existing anomaly detection technologies are increasingly revealing their limitations when faced with increasingly complex systems. On the one hand, traditional rule-based and feature-based detection systems struggle to cover all anomalies; on the other hand, massive amounts of data place higher demands on the processing capabilities of detection systems, and traditional technologies are inefficient when handling large-scale data, easily leading to missed detections and false alarms. Summary of the Invention

[0004] This application provides a data anomaly detection method, apparatus, device, and storage medium to improve the efficiency of data anomaly detection.

[0005] According to one aspect of this application, a data anomaly detection method is provided, the method comprising:

[0006] Obtain banking business parameters, and preprocess the initial business data extracted from the banking business system based on the banking business parameters to obtain the business data to be tested; wherein, the banking business parameters include basic banking parameters and bank-specific business parameters;

[0007] The initial centroids are determined based on the initial number of clusters, and cluster analysis is performed on the business data to be detected based on the initial centroids to obtain the initial cluster analysis results.

[0008] Determine whether the initial cluster analysis results meet the conditions for adjusting the number of clusters;

[0009] Determine the number of times the number of clusters should be adjusted, provided that the conditions for adjusting the number of clusters are met.

[0010] The number of clusters is adjusted according to the number of times the cluster number is adjusted, resulting in the adjusted number of clusters.

[0011] Based on the adjusted number of clusters, candidate centroids are determined, and based on the candidate centroids, the cluster analysis of the business data to be detected is performed again until the candidate centroids meet the centroid change rate condition, and the target cluster analysis result is obtained.

[0012] Based on the target clustering analysis results, the anomaly detection results are determined.

[0013] According to another aspect of this application, a data anomaly detection device is provided, the device comprising:

[0014] The business data to be detected module is used to acquire banking business parameters and preprocess the initial business data extracted from the banking business system according to the banking business parameters to obtain the business data to be detected; wherein, the banking business parameters include basic banking parameters and specific banking business parameters;

[0015] The initial clustering analysis result determination module is used to determine the initial centroids based on the initial number of clusters, and to perform clustering analysis on the business data to be detected based on the initial centroids to obtain the initial clustering analysis results;

[0016] The clustering analysis result judgment module is used to determine whether the initial clustering analysis result meets the condition for adjusting the number of clusters;

[0017] The cluster number adjustment number determination module is used to determine the number of cluster number adjustments when the conditions for adjusting the cluster number are met.

[0018] The cluster number determination module is used to adjust the cluster number according to the number of times the cluster number is adjusted, so as to obtain the adjusted cluster number.

[0019] The target clustering analysis result determination module is used to determine candidate centroids based on the adjusted number of clusters, and to re-perform clustering analysis on the business data to be detected based on the candidate centroids until the candidate centroids meet the centroid change rate condition, thereby obtaining the target clustering analysis result;

[0020] The anomaly detection result determination module is used to determine the anomaly detection result based on the target clustering analysis result.

[0021] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0022] One or more processors;

[0023] Memory, used to store one or more programs;

[0024] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the data anomaly detection methods provided in the embodiments of this application.

[0025] According to another aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements any of the data anomaly detection methods provided in the embodiments of this application.

[0026] According to another aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the data anomaly detection methods provided in the embodiments of this application.

[0027] This application obtains banking business parameters and preprocesses initial business data extracted from the banking business system based on these parameters to obtain business data to be detected. The banking business parameters include basic banking parameters and specific banking business parameters. Initial centroids are determined based on the initial cluster number, and cluster analysis is performed on the business data to be detected based on these initial centroids to obtain initial cluster analysis results. It is then determined whether the initial cluster analysis results meet the conditions for adjusting the cluster number. If the conditions for adjusting the cluster number are met, the number of cluster number adjustments is determined. Based on the number of cluster number adjustments, the cluster number is adjusted to obtain the adjusted cluster number. Based on the adjusted cluster number, candidate centroids are determined, and cluster analysis is performed again on the business data to be detected based on these candidate centroids until the candidate centroids meet the centroid change rate condition, thus obtaining the target cluster analysis result. Finally, based on the target cluster analysis result, the anomaly detection result is determined. The above scheme dynamically determines the optimal number of clusters by analyzing data characteristics, avoiding the subjectivity and blindness of manually setting the number of clusters, making the clustering results more consistent with the actual distribution of the data. It adopts an adaptive cluster number adjustment method for cluster analysis, thereby identifying abnormal data and improving the efficiency of data anomaly detection. Attached Figure Description

[0028] Figure 1 This is a flowchart of a data anomaly detection method provided according to Embodiment 1 of this application;

[0029] Figure 2 This is a flowchart of a data anomaly detection method according to Embodiment 2 of this application;

[0030] Figure 3 This is a schematic diagram of a data anomaly detection device according to Embodiment 3 of this application;

[0031] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the data anomaly detection method of Embodiment 4 of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] Furthermore, it should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of banking business parameters and initial business data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0035] Example 1

[0036] Figure 1 This is a flowchart of a data anomaly detection method according to Embodiment 1 of this application. This embodiment is applicable to the detection of anomalies in business data within a banking system. The method can be executed by a data anomaly detection device, which can be implemented in hardware and / or software and can be configured in a computer device, such as a server. Figure 1 As shown, the method includes:

[0037] S110. Obtain banking business parameters, and preprocess the initial business data extracted from the banking business system according to the banking business parameters to obtain the business data to be detected; wherein, the banking business parameters include basic banking parameters and specific banking business parameters.

[0038] Among them, banking business parameters can be parameters corresponding to banking business data that require anomaly detection. These parameters can include basic banking parameters and bank-specific business parameters. Basic banking parameters are system-level general parameters, which may include account number, account code, and business type, etc. For example, account numbers may include 1001 (cash account), 1002 (bank deposit), etc.; account codes may include BS001 (balance sheet type), PL002 (profit and loss statement type), etc.; and business types may include C001 (personal deposit account opening), L205 (corporate revolving loan), etc. Bank-specific business parameters are parameters specific to banking business scenarios, which may include interest rate parameters, credit limit parameters, risk parameters, and time-sensitive parameters, etc. For example, interest rate parameters may include benchmark interest rate, floating interest rate, etc.; credit limit parameters may include daily credit card payment limit, annual cross-border remittance facilitation limit, etc.; risk parameters may include provision coverage ratio, collateral discount rate, etc.; and time-sensitive parameters may include large transaction monitoring timeframe, wealth management product subscription deadline, etc. Extracting initial business data from a banking system can involve extracting initial business data from a big data platform or the data warehouse of a banking business management system, and then preprocessing the initial business data to obtain the business data to be tested.

[0039] Optionally, the step of preprocessing the initial business data extracted from the banking system according to the banking business parameters to obtain the business data to be tested includes: performing format unification processing on the initial business data extracted from the banking system according to the basic banking parameters to obtain formatted business data; and performing missing value filling and invalid value removal on the formatted business data according to the specific banking business parameters to obtain the business data to be tested.

[0040] Because the initial business data extracted from the banking system may contain problematic data, it is necessary to filter and validate the initial business data to clean up unusable data. Specifically, based on the bank's basic parameters, the initial business data is formatted to obtain formatted business data. For example, account numbers must be 4 digits, padded with zeros if necessary. Further, based on the bank's specific business parameters, missing values ​​are filled and invalid values ​​are removed from the formatted business data to obtain the business data to be tested. For example, if the benchmark interest rate is missing, it is filled with the parameter's default value; if the floating interest rate exceeds the regulatory range, it is directly marked as invalid and then removed. Preprocessing the initial business data extracted from the banking system, correcting or removing invalid data, and obtaining valid business data to be tested lays the foundation for subsequent detection of abnormal data.

[0041] S120. Determine the initial centroid based on the initial cluster number, and perform cluster analysis on the business data to be detected based on the initial centroid to obtain the initial cluster analysis result.

[0042] The initial number of clusters can be determined based on empirical values ​​or preset values, or it can be determined according to the elbow rule. The core idea of ​​the elbow rule is to find the "inflection point" by analyzing the trend of intra-cluster compactness changes of clustering results under different K values ​​(number of clusters) to determine the optimal number of clusters. This application does not specifically limit this. Preferably, the initial number of clusters can be 3.

[0043] Specifically, unsupervised clustering algorithms (such as mean clustering) can be used to perform cluster analysis on the business data to be detected, obtaining initial cluster analysis results. Data can then be automatically grouped through distance and mean iterations, thereby enabling the detection of abnormal data.

[0044] Optionally, determining the initial centroid based on the initial cluster number includes: randomly selecting a service data point from the service data to be detected as the first initial centroid; calculating the distances from other service data points to the first initial centroid; selecting the service data point furthest from the first initial centroid as the second initial centroid; calculating the sum of the distances from each remaining service data point to the first initial centroid and the second initial centroid; and selecting the service data point with the largest sum of distances as the third initial centroid.

[0045] For example, the initial number of clusters can be 3, that is, 3 initial centroids are determined. First, a data point A can be randomly selected as the first initial centroid. Then, the distances from other data points to A are calculated, and the data point B that is farthest from A is selected as the second initial centroid. Next, the distances from the remaining data points to A and B are calculated. For each data point, the sum of its minimum distances to A and B is calculated, and the data point C with the largest sum of minimum distances is selected as the third initial centroid. This makes the initial centroids sufficiently dispersed.

[0046] Optionally, the step of performing cluster analysis on the service data to be detected based on the initial centroid to obtain the initial cluster analysis result includes: calculating the initial distance between each service data point in the service data to be detected and the initial centroid, and assigning each service data point to the cluster containing the nearest initial centroid based on the initial distance; recalculating the mean of each cluster as the target centroid, calculating the target distance between each service data point in the service data to be detected and the target centroid, and assigning each service data point to the cluster containing the nearest target centroid based on the target distance to obtain the initial cluster analysis result.

[0047] The initial distance can be calculated using Euclidean distance, which is computationally efficient and converges quickly. Other distance metrics, such as Manhattan distance, can also be used; this embodiment does not impose a specific limitation and can be adapted to the actual situation. Further, based on the initial distance, each business data point is assigned to the cluster containing the nearest initial centroid. Through distance metrics, data with similar features are forcibly aggregated to form an interpretable cluster structure. Then, the mean of each cluster is recalculated as the target centroid. The new centroid is the mean of all points within the cluster, representing the data centroid of that cluster and reflecting the distribution characteristics of points within the cluster more accurately than the original centroid. Next, the target distance between each business data point in the business data to be detected and the target centroid is calculated, and each business data point is assigned to the cluster containing the nearest target centroid, obtaining the initial clustering analysis results. The initial clustering analysis results may include cluster labels, inter-cluster separation, the distance from each business data point within a cluster to the centroid, the silhouette coefficient, and the cluster size distribution. By optimizing the centroid position, the intra-cluster distance is minimized and the inter-cluster distance is maximized, and the globally optimal cluster partitioning is approximated through multiple iterations.

[0048] S130. Determine whether the initial cluster analysis results meet the conditions for adjusting the number of clusters.

[0049] The condition for adjusting the number of clusters allows for adjustment of the number of clusters. When this condition is met, the current number of clusters can be adjusted to allow for cluster analysis based on the adjusted number, thereby obtaining the optimal cluster analysis results. The condition for adjusting the number of clusters may include an inter-cluster separation less than a preset separation. The inter-cluster centroid distance characterizes the average distance between the centroids of each cluster; a larger average distance is better. Optionally, the condition for adjusting the number of clusters may also be that the number of iterations exceeds 3 / 4 of the preset maximum number of iterations.

[0050] S140. If the conditions for adjusting the number of clusters are met, determine the number of times the number of clusters can be adjusted.

[0051] Among them, the number of cluster adjustment times is the number of times the number of clusters has been adjusted, which is used to represent how many times the number of clusters has been adjusted. At the beginning, the number of cluster adjustment times is 0. Each time it is adjusted, the number of cluster adjustment times is incremented by one.

[0052] S150. Adjust the number of clusters according to the number of times the number of clusters is adjusted to obtain the adjusted number of clusters.

[0053] Specifically, different adjustment methods can be used to adjust the number of clusters based on whether the number of cluster number adjustments meets the curve plotting conditions, thus obtaining the adjusted number of clusters. The curve plotting conditions can be used to determine whether a curve showing the change in clustering error with the number of clusters needs to be plotted, and then the adjusted number of clusters can be determined through the curve. The curve plotting conditions are preset based on actual conditions or empirical values, and this application embodiment does not specifically limit them. For example, the condition can be that the number of cluster number adjustments is greater than a preset number. For instance, the preset number can be 10 times. When the number of cluster number adjustments is greater than 10 times, a curve showing the change in clustering error with the number of clusters is plotted, and the adjusted number of clusters is determined through the curve. When the number of cluster number adjustments is less than or equal to 10 times, a target cluster number adjustment method is determined from the candidate cluster number adjustment methods, and the cluster number is adjusted using the target cluster number adjustment method to obtain the adjusted number of clusters.

[0054] S160. Based on the adjusted number of clusters, determine the candidate centroids, and based on the candidate centroids, re-perform cluster analysis on the business data to be detected until the candidate centroids meet the centroid change rate condition, and obtain the target cluster analysis result.

[0055] Among them, the centroid change rate condition is the core condition used to characterize whether the algorithm has converged (stopped iteration). For example, the centroid change rate condition can be that the centroid change rate is less than a preset threshold. When the centroid change rate is less than the preset threshold, it means that the centroid no longer moves significantly, and the unsupervised clustering algorithm has converged or is close to convergence. At this time, continuing the iteration may consume more computing resources, and the improvement in cluster quality is negligible. Therefore, the iteration ends, and the target clustering analysis result is obtained.

[0056] S170. Based on the target clustering analysis results, determine the anomaly detection results.

[0057] The results of the target clustering analysis may include cluster labels, inter-cluster separation, number of iterations required for convergence, distance from each business data point within a cluster to the centroid, profile coefficient, and cluster size distribution.

[0058] Optionally, determining the anomaly detection result based on the target clustering analysis result includes: classifying business data points whose distance from the centroid of their cluster is greater than a preset distance as anomalous data.

[0059] Specifically, outliers that are more than a preset threshold away from the cluster centroid are identified as anomalous data. The anomalous points are automatically identified by utilizing the natural distribution of the data and are determined based on an intuitive judgment of distance.

[0060] This application embodiment obtains banking business parameters and preprocesses the initial business data extracted from the banking business system based on these parameters to obtain the business data to be detected. The banking business parameters include basic banking parameters and specific banking business parameters. Initial centroids are determined based on the initial cluster number, and cluster analysis is performed on the business data to be detected based on these initial centroids to obtain initial cluster analysis results. It is then determined whether the initial cluster analysis results meet the conditions for adjusting the cluster number. If the conditions for adjusting the cluster number are met, the number of cluster number adjustments is determined. The cluster number is adjusted based on the number of adjustments to obtain the adjusted cluster number. Candidate centroids are determined based on the adjusted cluster number, and cluster analysis is performed again on the business data to be detected based on these candidate centroids until the candidate centroids meet the centroid change rate condition, resulting in the target cluster analysis result. Finally, the anomaly detection result is determined based on the target cluster analysis result. The above scheme dynamically determines the optimal number of clusters by analyzing data characteristics, avoiding the subjectivity and blindness of manually setting the number of clusters, making the clustering results more consistent with the actual distribution of the data. It adopts an adaptive cluster number adjustment method for cluster analysis, thereby identifying abnormal data and improving the efficiency of data anomaly detection.

[0061] Example 2

[0062] Figure 2 This is a flowchart of a data anomaly detection method according to Embodiment 2 of this application. Based on the technical solutions of the above embodiments, this embodiment refines "adjusting the number of clusters according to the number of cluster number adjustments to obtain the adjusted number of clusters" into "when the number of cluster number adjustments meets the curve plotting conditions, calculating the clustering error corresponding to each historical cluster number in the historical cluster number set based on the historical cluster number set, obtaining a clustering error set; plotting a curve of clustering error changing with the number of clusters based on the historical cluster number set and the clustering error set, and adjusting the number of clusters according to the number of clusters at the inflection point of the curve to obtain the adjusted number of clusters; when the number of cluster number adjustments does not meet the curve plotting conditions, determining the target cluster number adjustment method from the candidate cluster number adjustment methods based on the initial clustering analysis results, and adjusting the number of clusters using the target cluster number adjustment method to obtain the adjusted number of clusters." It should be noted that parts not detailed in this embodiment can be found in the relevant descriptions of other embodiments. Figure 2 As shown, the method includes:

[0063] S210. Obtain banking business parameters, and preprocess the initial business data extracted from the banking business system according to the banking business parameters to obtain the business data to be detected; wherein, the banking business parameters include basic banking parameters and specific banking business parameters.

[0064] S220. Determine the initial centroid based on the initial cluster number, and perform cluster analysis on the business data to be detected based on the initial centroid to obtain the initial cluster analysis result.

[0065] S230. Determine whether the initial cluster analysis results meet the conditions for adjusting the number of clusters.

[0066] S240. If the conditions for adjusting the number of clusters are met, determine the number of times the number of clusters can be adjusted.

[0067] The condition for adjusting the number of clusters is a condition that allows the number of clusters to be adjusted. When this condition is met, the current number of clusters can be adjusted so that cluster analysis can be performed based on the adjusted number of clusters, thereby obtaining the optimal cluster analysis results. The condition for adjusting the number of clusters can include the inter-cluster separation being less than a preset separation. The inter-cluster centroid distance is used to characterize the average distance between the centroids of each cluster; a larger average distance is better. Optionally, the condition for adjusting the number of clusters can also be that the number of iterations exceeds 3 / 4 of the preset maximum number of iterations. The number of times the number of clusters has been adjusted indicates how many times the number of clusters has been adjusted. Initially, the number of times the number of clusters has been adjusted is 0; each time an adjustment is made, the number of times the number of clusters has been adjusted increments by one.

[0068] S250. If the number of times the cluster number is adjusted meets the curve plotting conditions, calculate the clustering error corresponding to each historical cluster number in the historical cluster number set according to the historical cluster number set, and obtain the clustering error set.

[0069] The curve plotting conditions can be used to determine whether a curve showing the change in clustering error with the number of clusters needs to be plotted, and then the adjusted number of clusters can be determined through the curve. The curve plotting conditions are preset based on actual conditions or empirical values, and this embodiment does not specifically limit them. For example, the condition could be that the number of cluster number adjustments is greater than a preset number. The clustering error can be calculated using the elbow rule, based on the current number of clusters, by calculating the sum of squared errors of the distance from each business data point in each cluster to the centroid. The smaller the sum of squared errors, the denser the business data points within the cluster, and the better the clustering effect. Furthermore, the calculation method for clustering error can be found in existing technologies, and will not be repeated in this embodiment. Further, when the number of cluster number adjustments meets the curve plotting conditions, the clustering error corresponding to each historical cluster number is calculated based on multiple historical cluster numbers before adjustment, resulting in a clustering error set.

[0070] S260. Based on the historical cluster number set and the cluster error set, plot the curve of cluster error as a function of cluster number, and adjust the cluster number according to the cluster number at the inflection point of the curve to obtain the adjusted cluster number.

[0071] Specifically, based on the historical cluster count set and cluster error set, a curve is plotted with the cluster count as the x-axis and the cluster error as the y-axis to obtain the curve of cluster error changing with the cluster count. The cluster count at the inflection point of the curve can be determined as the adjusted cluster count. Alternatively, the second derivative at each point on the curve can be calculated, and the adjusted cluster count can be determined based on the second derivative result.

[0072] S270. If the number of cluster number adjustments does not meet the curve plotting conditions, based on the initial cluster analysis results, a target cluster number adjustment method is determined from the candidate cluster number adjustment methods, and the cluster number is adjusted using the target cluster number adjustment method to obtain the adjusted cluster number.

[0073] The curve plotting condition can be used to determine whether a curve showing the clustering error changing with the number of clusters needs to be plotted. For example, this condition could be that the number of cluster number adjustments exceeds a preset number. Candidate cluster number adjustment methods may include fine-tuning, transition adjustment, and business-driven adjustment. Fine-tuning involves adding or removing one cluster at a time, requiring minimal computation and offering stability; transition adjustment involves adjusting in leaps (e.g., K+5), offering high efficiency; and business-driven adjustment forcibly sets a K value based on banking business rules, providing strong interpretability.

[0074] Specifically, if the number of cluster number adjustments does not meet the conditions for curve plotting, the target cluster number adjustment method can be determined from the candidate cluster number adjustment methods based on the cluster size balance, the presence of empty or minimal clusters, and the low silhouette coefficient in the initial cluster analysis results. This allows for the adjustment of the initial cluster number to obtain the adjusted cluster number. For example, if a cluster accounts for 80% of the business data, indicating a small cluster number, the cluster number can be increased according to the adjustment method; conversely, if a cluster has fewer than 5 business data points, indicating a large cluster number, the cluster number can be decreased according to the adjustment method.

[0075] S280. Based on the adjusted number of clusters, determine the candidate centroids, and based on the candidate centroids, re-perform cluster analysis on the business data to be detected until the candidate centroids meet the centroid change rate condition, and obtain the target cluster analysis result.

[0076] S290. Based on the target clustering analysis results, determine the anomaly detection results.

[0077] This application embodiment obtains banking business parameters and preprocesses the initial business data extracted from the banking business system based on these parameters to obtain the business data to be detected. The banking business parameters include basic banking parameters and specific banking business parameters. Initial centroids are determined based on the initial cluster number, and cluster analysis is performed on the business data to be detected based on these initial centroids to obtain initial cluster analysis results. It is then determined whether the initial cluster analysis results meet the conditions for adjusting the cluster number. If the conditions for adjusting the cluster number are met, the number of cluster number adjustments is determined. The cluster number is adjusted based on the number of adjustments to obtain the adjusted cluster number. Candidate centroids are determined based on the adjusted cluster number, and cluster analysis is performed again on the business data to be detected based on these candidate centroids until the candidate centroids meet the centroid change rate condition, resulting in the target cluster analysis result. Finally, the anomaly detection result is determined based on the target cluster analysis result. The above scheme dynamically determines the optimal number of clusters by analyzing data characteristics, avoiding the subjectivity and blindness of manually setting the number of clusters, making the clustering results more consistent with the actual distribution of the data. It adopts an adaptive cluster number adjustment method for cluster analysis, thereby identifying abnormal data and improving the efficiency of data anomaly detection.

[0078] Example 3

[0079] Figure 3 This is a schematic diagram of a data anomaly detection device according to Embodiment 3 of this application. It is applicable to detecting anomalies in business data within a banking system. The data anomaly detection device can be implemented in hardware and / or software and can be configured in a computer device, such as a server. Figure 3 As shown, the device includes:

[0080] The business data to be detected module 310 is used to acquire banking business parameters and preprocess the initial business data extracted from the banking business system according to the banking business parameters to obtain the business data to be detected; wherein, the banking business parameters include basic banking parameters and specific banking business parameters;

[0081] The initial clustering analysis result determination module 320 is used to determine the initial centroid based on the initial cluster number, and to perform clustering analysis on the business data to be detected based on the initial centroid to obtain the initial clustering analysis result;

[0082] The clustering analysis result judgment module 330 is used to determine whether the initial clustering analysis result meets the condition for adjusting the number of clusters;

[0083] The cluster number adjustment number determination module 340 is used to determine the number of cluster number adjustments when the conditions for adjusting the cluster number are met.

[0084] The cluster number determination module 350 is used to adjust the cluster number according to the number of times the cluster number is adjusted, so as to obtain the adjusted cluster number.

[0085] The target clustering analysis result determination module 360 ​​is used to determine candidate centroids based on the adjusted number of clusters, and to re-perform clustering analysis on the business data to be detected based on the candidate centroids until the candidate centroids meet the centroid change rate condition, thereby obtaining the target clustering analysis result;

[0086] The anomaly detection result determination module 370 is used to determine the anomaly detection result based on the target clustering analysis result.

[0087] Optionally, the cluster number determination module 350 is adjusted, specifically for:

[0088] If the number of cluster adjustments meets the curve plotting conditions, the clustering error corresponding to each historical cluster number in the historical clustering set is calculated based on the historical clustering set to obtain the clustering error set.

[0089] Based on the historical cluster count set and the cluster error set, a curve showing the change of cluster error with the number of clusters is plotted, and the number of clusters is adjusted according to the number of clusters at the inflection point of the curve to obtain the adjusted number of clusters.

[0090] If the number of cluster number adjustments does not meet the curve plotting conditions, based on the initial cluster analysis results, a target cluster number adjustment method is determined from the candidate cluster number adjustment methods, and the cluster number is adjusted using the target cluster number adjustment method to obtain the adjusted cluster number.

[0091] Optionally, the initial cluster analysis result determination module 320 includes:

[0092] The first initial centroid determination unit is used to randomly select a service data point as the first initial centroid in the service data to be detected;

[0093] The distance determination unit is used to calculate the distance from other business data points to the first initial centroid;

[0094] The second initial centroid determination unit is used to take the service data point farthest from the first initial centroid as the second initial centroid.

[0095] The distance sum determination unit is used to calculate the sum of the distances from each remaining business data point to the first initial centroid and the second initial centroid;

[0096] The third initial centroid determination unit is used to select the business data point with the largest sum of distances as the third initial centroid.

[0097] Optionally, the initial cluster analysis result determination module 320 also includes:

[0098] A service data point allocation unit is used to calculate the initial distance between each service data point in the service data to be detected and the initial centroid, and allocate each service data point to the cluster containing the nearest initial centroid according to the initial distance;

[0099] The initial clustering analysis result determination unit is used to recalculate the mean of each cluster as the target centroid, calculate the target distance between each business data point in the business data to be detected and the target centroid, and assign each business data point to the cluster containing the nearest target centroid according to the target distance, thereby obtaining the initial clustering analysis result.

[0100] Optionally, the anomaly detection result determination module 370 is specifically used for:

[0101] Business data points whose distance from the centroid of their cluster is greater than a preset distance are considered abnormal data.

[0102] Optionally, the business data determination module 310 includes:

[0103] The formatted business data determination unit is used to perform format unification processing on the initial business data extracted from the banking business system according to the bank's basic parameters to obtain formatted business data.

[0104] The business data determination unit is used to fill in missing values ​​and remove invalid values ​​from the formatted business data according to the specific business parameters of the bank to obtain the business data to be tested.

[0105] This application embodiment obtains banking business parameters and preprocesses the initial business data extracted from the banking business system based on these parameters to obtain the business data to be detected. The banking business parameters include basic banking parameters and specific banking business parameters. Initial centroids are determined based on the initial cluster number, and cluster analysis is performed on the business data to be detected based on these initial centroids to obtain initial cluster analysis results. It is then determined whether the initial cluster analysis results meet the conditions for adjusting the cluster number. If the conditions for adjusting the cluster number are met, the number of cluster number adjustments is determined. The cluster number is adjusted based on the number of adjustments to obtain the adjusted cluster number. Candidate centroids are determined based on the adjusted cluster number, and cluster analysis is performed again on the business data to be detected based on these candidate centroids until the candidate centroids meet the centroid change rate condition, resulting in the target cluster analysis result. Finally, the anomaly detection result is determined based on the target cluster analysis result. The above scheme dynamically determines the optimal number of clusters by analyzing data characteristics, avoiding the subjectivity and blindness of manually setting the number of clusters, making the clustering results more consistent with the actual distribution of the data. It adopts an adaptive cluster number adjustment method for cluster analysis, thereby identifying abnormal data and improving the efficiency of data anomaly detection.

[0106] The data anomaly detection device provided in this application embodiment can execute the data anomaly detection method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing each data anomaly detection method.

[0107] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0108] Example 4

[0109] Figure 4 This is a schematic diagram of the structure of an electronic device 410 implementing the data anomaly detection method of this application embodiment. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0110] like Figure 4As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory 412 or a random access memory 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 412 or loaded from storage unit 418 into the random access memory 413. The random access memory 413 can also store various programs and data required for the operation of the electronic device 410. The processor 411, read-only memory 412, and random access memory 413 are interconnected via a bus 414. An input / output interface 415 is also connected to the bus 414.

[0111] Multiple components in electronic device 410 are connected to input / output interface 415, including: input unit 416, such as keyboard, mouse, etc.; output unit 417, such as various types of monitors, speakers, etc.; storage unit 418, such as disk, optical disk, etc.; and communication unit 419, such as network card, modem, wireless transceiver, etc. Communication unit 419 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0112] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as data anomaly detection methods.

[0113] In some embodiments, the data anomaly detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 410 via read-only memory 412 and / or communication unit 419. When the computer program is loaded into random access memory 413 and executed by processor 411, one or more steps of the data anomaly detection method described above may be performed. Alternatively, in other embodiments, processor 411 may be configured as the data anomaly detection method by any other suitable means (e.g., by means of firmware).

[0114] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), payload programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data anomaly detection device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube or liquid crystal display monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0118] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0119] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system to address the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0120] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for detecting data anomalies, characterized in that, include: Obtain banking business parameters, and preprocess the initial business data extracted from the banking business system based on the banking business parameters to obtain the business data to be tested; wherein, the banking business parameters include basic banking parameters and bank-specific business parameters; The initial centroids are determined based on the initial number of clusters, and cluster analysis is performed on the business data to be detected based on the initial centroids to obtain the initial cluster analysis results. Determine whether the initial cluster analysis results meet the conditions for adjusting the number of clusters; Determine the number of times the number of clusters should be adjusted, provided that the conditions for adjusting the number of clusters are met. The number of clusters is adjusted according to the number of times the cluster number is adjusted, resulting in the adjusted number of clusters. Based on the adjusted number of clusters, candidate centroids are determined, and based on the candidate centroids, the cluster analysis of the business data to be detected is performed again until the candidate centroids meet the centroid change rate condition, and the target cluster analysis result is obtained. Based on the target clustering analysis results, the anomaly detection results are determined.

2. The method according to claim 1, characterized in that, The step of adjusting the cluster number based on the number of cluster number adjustments to obtain the adjusted cluster number includes: If the number of cluster adjustments meets the curve plotting conditions, the clustering error corresponding to each historical cluster number in the historical clustering set is calculated based on the historical clustering set to obtain the clustering error set. Based on the historical cluster count set and the cluster error set, a curve showing the change of cluster error with the number of clusters is plotted, and the number of clusters is adjusted according to the number of clusters at the inflection point of the curve to obtain the adjusted number of clusters. If the number of cluster number adjustments does not meet the curve plotting conditions, based on the initial cluster analysis results, a target cluster number adjustment method is determined from the candidate cluster number adjustment methods, and the cluster number is adjusted using the target cluster number adjustment method to obtain the adjusted cluster number.

3. The method according to claim 1, characterized in that, The step of determining the initial centroid based on the initial cluster number includes: In the service data to be detected, a service data point is randomly selected as the first initial centroid; Calculate the distance from other business data points to the first initial centroid; The business data point farthest from the first initial centroid is used as the second initial centroid; Calculate the sum of the distances from each remaining business data point to the first initial centroid and the second initial centroid; The business data point with the largest sum of distances is used as the third initial centroid.

4. The method according to claim 1, characterized in that, The step of performing cluster analysis on the business data to be detected based on the initial centroid to obtain the initial cluster analysis results includes: Calculate the initial distance between each service data point in the service data to be detected and the initial centroid, and assign each service data point to the cluster containing the nearest initial centroid based on the initial distance; The mean of each cluster is recalculated as the target centroid. The target distance between each business data point in the business data to be detected and the target centroid is calculated. Based on the target distance, each business data point is assigned to the cluster containing the nearest target centroid to obtain the initial clustering analysis results.

5. The method according to claim 1, characterized in that, The step of determining the anomaly detection result based on the target clustering analysis result includes: Business data points whose distance from the centroid of their cluster is greater than a preset distance are considered abnormal data.

6. The method according to claim 1, characterized in that, The step of preprocessing the initial business data extracted from the banking business system according to the banking business parameters to obtain the business data to be detected includes: Based on the aforementioned basic bank parameters, the initial business data extracted from the banking business system is processed to unify the format, resulting in formatted business data. Based on the bank's specific business parameters, the formatted business data is filled with missing values ​​and invalid values ​​are removed to obtain the business data to be tested.

7. A data anomaly detection device, characterized in that, include: The business data to be detected module is used to acquire banking business parameters and preprocess the initial business data extracted from the banking business system according to the banking business parameters to obtain the business data to be detected; wherein, the banking business parameters include basic banking parameters and specific banking business parameters; The initial clustering analysis result determination module is used to determine the initial centroids based on the initial number of clusters, and to perform clustering analysis on the business data to be detected based on the initial centroids to obtain the initial clustering analysis results; The clustering analysis result judgment module is used to determine whether the initial clustering analysis result meets the condition for adjusting the number of clusters; The cluster number adjustment number determination module is used to determine the number of cluster number adjustments when the conditions for adjusting the cluster number are met. The cluster number determination module is used to adjust the cluster number according to the number of times the cluster number is adjusted, so as to obtain the adjusted cluster number. The target clustering analysis result determination module is used to determine candidate centroids based on the adjusted number of clusters, and to re-perform clustering analysis on the business data to be detected based on the candidate centroids until the candidate centroids meet the centroid change rate condition, thereby obtaining the target clustering analysis result; The anomaly detection result determination module is used to determine the anomaly detection result based on the target clustering analysis result.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data anomaly detection method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data anomaly detection method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the data anomaly detection method according to any one of claims 1-6.

Images