Data processing method, electronic equipment and storage medium

Abstract

The invention discloses a data processing method, electronic equipment and a storage medium. Comprising the following steps: acquiring to-be-identified data, performing dimension reduction processing on the to-be-identified data, and performing clustering processing on the dimension-reduced data through a pre-trained clustering model to obtain a first clustering result; determining a verification result corresponding to the first clustering result, and updating the first clustering result based on the verification result to obtain a second clustering result; determining a global anomaly judgment result corresponding to the second clustering result, and updating the second clustering result based on the global anomaly judgment result to obtain a third clustering result; determining a local anomaly evaluation value of each data record of the cluster in the third clustering result through a local anomaly evaluation algorithm, and determining local anomaly data based on the local anomaly evaluation value of each data record in each cluster; and performing exception marking processing on the local exception data and global exception data in the global exception judgment result. And the accuracy and efficiency of abnormal data identification are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method, an electronic device and a storage medium. BACKGROUND

[0002] With the gradual deepening of digital transformation, major banks have established their own data warehouse, data lake or lake-warehouse integration. As the amount of data increases, data quality management will become increasingly important.

[0003] Downstream data service systems consume data in data lake-warehouse, which can perform simple null value recognition, data volume recognition, record missing, illegal data value, and other quality management. It can also evaluate and analyze data quality based on statistical methods and indicators to find problems and provide basis for data quality improvement. However, this method requires that the data conforms to a specific distribution, such as normal distribution, and then identifies abnormalities based on mean, variance, quantile, and other statistical rules. For large amounts of data with many dimensions and uneven distribution, accurate and efficient abnormal data identification is not possible, which can easily lead to abnormal data flowing into downstream data service systems, thereby affecting the accuracy and efficiency of subsequent business processing. SUMMARY

[0004] The present application provides a data processing method, an electronic device and a storage medium to solve the problem of inaccurate and inefficient abnormal data identification.

[0005] According to an aspect of the present application, a data processing method is provided, comprising:

[0006] Obtaining the data to be identified, performing dimensionality reduction processing on the data to be identified to obtain reduced data, wherein the data to be identified includes multi-dimensional data;

[0007] Performing clustering processing on the reduced data through a pre-trained clustering model to obtain a first clustering result;

[0008] Performing verification processing on the first clustering result to obtain a verification result corresponding to the first clustering result, performing update operation on the first clustering result based on the verification result to obtain a second clustering result;

[0009] Performing global anomaly determination processing on the second clustering result to obtain a global anomaly determination result, performing update processing on the second clustering result based on the global anomaly determination result to obtain a third clustering result;

[0010] For each clustering cluster in the third clustering result, determining a local anomaly evaluation value of each data record in the clustering cluster through a local anomaly evaluation algorithm, and determining a local anomaly data based on the local anomaly evaluation value of each data record in each clustering cluster;

[0011] The local abnormal data is marked as an exception, and if the global abnormal data exists in the global abnormality determination result, the global abnormal data is marked as an exception.

[0012] Optionally, the to-be-identified data is subjected to dimension reduction processing to obtain reduced dimension data, including: determining a comprehensive weight of each data dimension in the to-be-identified data, and performing weighting processing on the to-be-identified data based on the comprehensive weight of each data dimension to obtain weighted to-be-identified data; and performing dimension reduction processing on the weighted to-be-identified data by using a preset dimension reduction processing algorithm to obtain reduced dimension data, wherein the preset dimension reduction processing algorithm includes UMAP dimension reduction, PacMAP dimension reduction and TriMAP dimension reduction.

[0013] Optionally, the comprehensive weight of each data dimension in the to-be-identified data is determined by: obtaining a business scenario data set, wherein the business scenario data set includes a plurality of business scenarios and first weight data corresponding to each business scenario; for each business scenario, obtaining second weight data of each data dimension in the to-be-identified data under the business scenario; and determining the comprehensive weight of each data dimension based on the second weight data of each data dimension under each business scenario and the first weight data corresponding to each business scenario.

[0014] Optionally, the reduced dimension data is subjected to clustering processing by using a pre-trained clustering model to obtain a first clustering result, including: performing data format conversion processing on the reduced dimension data to obtain data meeting input requirements of the pre-trained clustering model; and inputting the data meeting the input requirements of the pre-trained clustering model into the pre-trained clustering model to obtain the first clustering result, wherein the pre-trained clustering model includes at least two clustering algorithms, and the at least two clustering algorithms are switched according to a data distribution form of input data of the pre-trained clustering model.

[0015] Optionally, the first clustering result is verified to obtain a verification result corresponding to the first clustering result, the first clustering result is updated based on the verification result to obtain a second clustering result, including: determining clustering index data of the first clustering result based on a preset clustering index calculation method, wherein the clustering index data includes at least two of a silhouette coefficient, Davies-Bouldin index, and variance ratio criterion; if at least two of the silhouette coefficient, Davies-Bouldin index, and variance ratio criterion satisfy corresponding preset index thresholds, it is determined that the verification result corresponding to the first clustering result is verification success, the first clustering result is not updated, and the first clustering result is determined as the second clustering result; if at least two of the silhouette coefficient, Davies-Bouldin index, and variance ratio criterion do not satisfy corresponding preset index thresholds, it is determined that the verification result corresponding to the first clustering result is verification failure, the clustering model is controlled to be optimized, and the dimension-reduced data is continuously clustered based on the optimized clustering model until the second clustering result is obtained, wherein at least two of the silhouette coefficient, Davies-Bouldin index, and variance ratio criterion corresponding to the second clustering result satisfy corresponding preset index thresholds.

[0016] Optionally, the second clustering result is subjected to global anomaly determination processing to obtain a global anomaly determination result, the second clustering result is updated based on the global anomaly determination result to obtain a third clustering result, including: performing global anomaly data filtering processing in the second clustering result based on a preset global anomaly data filtering condition to obtain a global anomaly filtering result; if the global anomaly filtering result is that there is global anomaly data, the global anomaly data is removed from the corresponding clustering cluster to obtain the third clustering result, and if the global anomaly filtering result is that there is no global anomaly data, the second clustering result is determined as the third clustering result.

[0017] Optionally, a local anomaly evaluation value of each data record in the clustering cluster is determined by a local anomaly evaluation algorithm, including: each data record in the clustering cluster is processed by the local anomaly evaluation algorithm to determine a local anomaly evaluation value corresponding to each data record, wherein the local anomaly evaluation value includes one or more of a local outlier factor value, a connected outlier factor value, and an impact outlier factor value.

[0018] Optionally, a local anomaly data is determined based on the local anomaly evaluation value of each data record in each clustering cluster, including: for any clustering cluster, the local anomaly evaluation value of each data record in the clustering cluster is obtained by traversing the clustering cluster; for any data record, if the local anomaly evaluation value of the data record satisfies a local anomaly filtering condition, the data record is determined as local anomaly data; and if the local anomaly evaluation value of the data record does not satisfy the local anomaly filtering condition, the data record is determined as non-local anomaly data.

[0019] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected with the at least one processor; wherein

[0022] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the data processing method of any embodiment of the present application.

[0023] According to another aspect of the present application, a computer readable storage medium is provided, the computer readable storage medium stores computer instructions for enabling a processor to implement the data processing method of any embodiment of the present application when executed by the processor.

[0024] The technical scheme of the embodiments of the present application, by obtaining to-be-identified data, performing dimension reduction processing on the to-be-identified data to obtain reduced dimension data, wherein the to-be-identified data comprises multi-dimensional data; performing clustering processing on the reduced dimension data by a pre-trained clustering model to obtain a first clustering result; performing verification processing on the first clustering result to obtain a verification result corresponding to the first clustering result, performing updating operation on the first clustering result based on the verification result to obtain a second clustering result; performing global anomaly judgment processing on the second clustering result to obtain a global anomaly judgment result, performing updating processing on the second clustering result based on the global anomaly judgment result to obtain a third clustering result; for each clustering cluster in the third clustering result, determining a local anomaly evaluation value of each data record in the clustering cluster by a local anomaly evaluation algorithm, determining local anomaly data based on the local anomaly evaluation value of each data record in each clustering cluster; performing anomaly marking processing on the local anomaly data, and if there is global anomaly data in the global anomaly judgment result, performing anomaly marking processing on the global anomaly data. By performing dimension reduction processing on the multi-dimensional to-be-identified data, data redundancy and computational complexity can be effectively reduced, laying a high-efficiency foundation for subsequent clustering; after obtaining the first clustering result by means of the pre-trained clustering model, the second clustering result is obtained by updating through verification processing, which can correct the deviation of the initial clustering and improve the clustering accuracy; the third clustering result is obtained by updating the second clustering result through global anomaly judgment, which can first eliminate the global anomaly that obviously deviates from the overall distribution, optimizing the overall quality of the clustering cluster; then, the local anomaly evaluation and screening marking are performed on each cluster in the third clustering result, which can accurately identify the hidden local anomaly in the cluster, further purifying the data of each cluster; at the same time, the global and local anomaly data are marked respectively, ensuring the comprehensiveness of anomaly identification, solving the problem of inaccurate and low-efficiency anomaly data identification, providing a high-quality and high-identification data foundation for subsequent data application, and the overall process takes into account the efficiency and accuracy, and the anomaly processing is hierarchical and targeted.

[0025] It is to be understood that the details set forth herein do not limit the scope of the embodiments of the application to the specific embodiments described. Rather, the scope of the embodiments of the application is to be defined by the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0027] Figure 1 is a flow chart of a data processing method provided by the first embodiment of the present application;

[0028] Figure 2 is a flow chart of a data processing method provided by the second embodiment of the present application;

[0029] Figure 3 is a structural schematic diagram of a data processing device provided by the third embodiment of the present application;

[0030] Figure 4 is a structural schematic diagram of an electronic device for implementing the data processing method of the present application. DETAILED DESCRIPTION

[0031] In order to make the technical personnel in the art better understand the present application, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should be within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment One

[0034] Figure 1 is a flowchart of a data processing method provided by Embodiment One of the present application. The present embodiment can be applied to the case of performing data anomaly identification processing. The method can be executed by a data processing device, which can be implemented in the form of hardware and / or software, and can be configured in a computer, a server or other electronic equipment.

[0035] It should be noted that in the field of banking business, a large number of user business demands are processed through various business systems on a daily basis. A large amount of data will be generated during the operation of various business systems. In order to achieve global data integration, break down data silos, reduce technical and operation and maintenance costs, and improve data governance and compliance capabilities, data lake and warehouse are used as the hub of bank digital transformation to store data generated from different business systems. As the amount of data increases, the quality management of data becomes more and more important. The data lake and warehouse platform belongs to the data service field of cloud-native data lake and is a data application level platform, which includes batch data preprocessing, data warehouse, real-time data warehouse and other modules, and provides data lake entry, consumption and other services for the data application system of the whole bank. As a downstream data service system, there are many systems that provide tag and index services to the internal lines of the head office and branch office, support bank internal business personnel to timely and accurately perform customer group analysis, accurate marketing, risk control, business decision-making and special analysis, and these application scenarios are all based on customers and institutions. The downstream data service system often uses user behavior, transaction characteristics and other similar customer data as the basis for service input. User behavior includes but is not limited to access clicks and transaction frequency, and transaction characteristics include but are not limited to age and transaction amount. Therefore, the accuracy of data consumption is increasingly important, and data anomalies in the source system will have an important impact on data processing, marking, business handling and regulatory reporting of the downstream data service system. Therefore, in the case of transmitting data in the data lake and warehouse to the downstream service system for consumption data, there is an urgent need for an efficient data processing method to identify anomalies in the multi-dimensional data in the data lake and warehouse, which can timely inspect data and efficiently identify whether there are anomalies in the data in the data lake and warehouse, so as to avoid the impact of abnormal data on the normal operation of the downstream service system.

[0036] As shown in Figure 1 , the method comprises:

[0037] S110, acquiring to-be-identified data, performing dimension reduction processing on the to-be-identified data to obtain reduced data, wherein the to-be-identified data comprises multi-dimensional data.

[0038] The to-be-identified data can be understood as data that needs to be identified for abnormal data, and the to-be-identified data includes multi-dimensional data. These dimensions can be different attributes, indicators or parameters. For example, the multi-dimensional data can include a customer number, a plurality of label identifiers and a plurality of indicator values. The to-be-identified data can be read from a preset storage space. The preset storage space can be a data warehouse, a data lake or a data lake warehouse. In this embodiment, in order to solve the problem that the distance metric can be invalid due to too large data dimensions, thereby affecting the accuracy of subsequent data processing, a preset dimension reduction algorithm can be called to perform dimension reduction processing on the multi-dimensional data in the to-be-identified data. On the premise of trying to retain key information, the number of dimensions of the data is reduced to obtain reduced data corresponding to the to-be-identified data. The preset dimension reduction algorithm includes but is not limited to UMAP dimension reduction, PacMAP dimension reduction and TriMAP dimension reduction. A suitable dimension reduction algorithm is selected according to the characteristics of the to-be-identified data. This is not limited here.

[0039] Specifically, the to-be-identified data can be obtained from a specified data storage space or data source. For example, the to-be-identified data can be read from a pre-constructed data lake warehouse. It should be noted that the amount of data in the data lake warehouse is extremely large. Therefore, a daily data consumption task can be set. The daily data consumption task can be triggered and executed at a regular time. The daily consumption data is read from the data lake warehouse, and the daily consumption data is used as the to-be-identified data. After obtaining the to-be-identified data, a data dimension reduction processing algorithm is called to perform dimension reduction processing on the to-be-identified data. The high-dimensional space is mapped to a low-dimensional space, and the reduced data is obtained on the premise of trying to reduce information loss.

[0040] In this embodiment, by performing dimension reduction processing on the to-be-identified data, the data dimensions can be greatly reduced, the calculation cost and storage requirement of subsequent data processing can be reduced, the problem that the distance metric can be invalid due to too large data dimensions can be avoided, and the accuracy and efficiency of subsequent data processing can be improved.

[0041] S120, performing clustering processing on the reduced data by using the pre-trained clustering model to obtain a first clustering result.

[0042] The pre-trained clustering model refers to a clustering algorithm model that is trained in advance through a large number of labeled or unlabeled sample data, for example, a pre-trained K-means, hierarchical clustering model, and the like. The pre-trained clustering model has learned general rules of data clustering or distribution patterns in specific scenarios, can directly receive new data and automatically perform a clustering task without retraining, and can quickly adapt to clustering requirements in similar scenarios. The first clustering result can be understood as a clustering result obtained by performing clustering processing on the data by using the clustering model. The clustering model can perform clustering processing on the input data according to similarity features between the data, and output the clustering result. The similarity features between the data include, but are not limited to, distance features and density features. The first clustering result includes, but is not limited to, a clustering cluster label to which a data point belongs, a distribution range of each clustering cluster, and a cluster center point, and is a preliminary division of a data group structure, providing a basis for subsequent secondary verification, subdivision analysis, or decision-making.

[0043] Specifically, a clustering model that has been trained through a large number of sample data is called first, and low-dimensional data obtained by performing dimensionality reduction processing on the data is input into the clustering model. The clustering model automatically classifies data points with similar features into the same category according to the similarity between the data, and classifies data points with large feature differences into different categories. Finally, the first clustering result including a clustering cluster label, a cluster center point of each clustering cluster, and data in each clustering cluster is output.

[0044] In this embodiment, the pre-trained clustering model is used to perform clustering processing on the data obtained by performing dimensionality reduction on the to-be-recognized data, to obtain the first clustering result. The new dimensionally reduced data can be quickly clustered, and the processing efficiency is improved. Meanwhile, the dimensionally reduced data reduces the interference of redundant information, so that the clustering model is more likely to capture the core distribution pattern of the data, and the accuracy and stability of the clustering result are improved. In addition, the automatic clustering process does not require manual labeling of categories, reduces the dependence on professional knowledge, and facilitates quick mining of hidden group structures in the data.

[0045] Optionally, the pre-trained clustering model is used to perform clustering processing on the dimensionally reduced data to obtain the first clustering result, including: performing data format conversion processing on the dimensionally reduced data to obtain data meeting input requirements of the pre-trained clustering model; and inputting the data meeting the input requirements of the pre-trained clustering model into the pre-trained clustering model to obtain the first clustering result. The pre-trained clustering model includes at least two clustering algorithms, and the at least two clustering algorithms are switched according to a data distribution form of input data of the pre-trained clustering model.

[0046] Specifically, after obtaining the dimension-reduced data, the low-dimensional data after dimension reduction can be first converted in format, such as uniform data type, adjusting the arrangement of feature dimensions, standardizing the numerical range, etc., so as to conform to the input specification of the pre-trained clustering model; then the data after format conversion processing is input into the clustering model for processing to obtain a first clustering result, wherein the clustering model internally contains at least two clustering algorithms, and the at least two clustering algorithms include but are not limited to K-means and DBSCAN. After the data after format conversion processing is input into the clustering model, the data distribution form of the input data is first analyzed, for example, whether the input data is in convex set distribution, whether the density is uniform, and according to the identified data distribution form, the adaptive algorithm is automatically switched to cluster, for example, convex set distribution is adapted to K-means, and non-convex and non-uniform density distribution is adapted to DBSCAN. Finally, the first clustering result containing each clustering cluster label, the cluster center point of each clustering cluster and the data in each clustering cluster is output.

[0047] In the embodiment, the data format conversion ensures the compatibility of the model input, avoids the clustering deviation caused by the format problem; the multi-algorithm adaptive switching mechanism in the clustering model can select the optimal clustering strategy according to the actual distribution of the data, and compared with a single algorithm, it can better adapt to complex data scenarios and improve the accuracy and robustness of the clustering result; at the same time, the reuse of the pre-trained clustering model reduces the repeated training cost, and combined with the characteristics of the dimension-reduced data, the inherent group structure of the data can be efficiently mined.

[0048] S130, verifying the first clustering result to obtain a verification result corresponding to the first clustering result, updating the first clustering result based on the verification result to obtain a second clustering result.

[0049] The check result can be understood as a check state representing the first clustering result, and can include a check success state and a check failure state. The check state can be presented by a preset representation form, which includes but is not limited to a string and a number, and is not limited herein. Optionally, a corresponding check algorithm can be set according to the accuracy and effectiveness requirements of the clustering result. In the case of checking the first clustering result, the corresponding check algorithm can be directly called to obtain the corresponding check result. For example, a corresponding quantitative index can be set according to the check requirement, which includes but is not limited to a silhouette coefficient, a Davies-Bouldin index, and a variance ratio criterion. Then, the clustering result is processed according to the calculation method of the quantitative index to obtain quantitative index data. The check result is determined according to the quantitative index data and a preset check rule. Other check algorithms can also be set, which are not limited herein. The second clustering result specifically represents an optimized result obtained by modifying the first clustering result based on the check result. The optimized clustering result can be obtained by optimizing a clustering model, adjusting a clustering parameter, reallocating abnormal data, or optimizing cluster division logic, to solve the unreasonable problems in the first clustering result, ensure the consistency of data in the clustering cluster is stronger, the distinction between clusters is clearer, and is closer to the real distribution rule of data, which can provide more reliable clustering basis for subsequent analysis or application.

[0050] Specifically, first, for the first clustering result, a preset check algorithm is called to process the first clustering result to obtain a check result. Then, the first clustering result is updated according to the check result, and the updated clustering result is determined as the second clustering result. If the check result is a check success, the first clustering result is not processed, and the first clustering result is directly determined as the second clustering result. If the check result is a check failure, the first clustering result is re-clustered to obtain the second clustering result. It should be noted that in the process of re-clustering, a clustering model optimization method can be called to optimize the clustering model, and the clustering is reprocessed by the optimized clustering model. The second clustering result can also be obtained by splitting clusters with large internal differences, merging highly similar adjacent clusters, adjusting the category of boundary data, and the like.

[0051] In some specific embodiments, the first clustering result can be systematically checked by preset checking rules, the preset checking rules include but are not limited to similarity checking rules of data in the clustering cluster and attribution rationality checking rules of clustering boundary data, checking rules of whether the clustering logic conforms to the business scenario by combining prior knowledge, and then the result obtained by checking the first clustering result based on the preset checking rules is statistically analyzed to obtain the checking result of the first clustering result. At least two of the preset checking rules can be set as checking success, and then it is determined that the checking result of the first clustering result is checking success, otherwise, it is determined that the checking result of the first clustering result is failure. For the similarity checking rules of data in the clustering cluster, if the similarity between the data in the clustering cluster meets the preset similarity threshold, it is determined that the similarity checking is successful. For the attribution rationality checking rules of clustering boundary data, if the attribution of the clustering boundary data is within a reasonable range, it is determined that the attribution rationality checking is successful. For the checking rules of whether the clustering logic conforms to the business scenario by combining prior knowledge, if the actual values of each dimension in the clustering cluster meet the set value range of the corresponding dimension in the corresponding business scenario, it is determined that the business scenario checking is successful. It should be noted that other checking rules can also be set according to actual needs, which are not limited here.

[0052] In the embodiment, the checking link effectively identifies unreasonable structures in the first clustering result, avoiding the direct application of incorrect results. The updating operation based on the checking specifically improves the accuracy and rationality of the clustering result, making the final result more consistent with the real distribution of data, and providing a more reliable basis for subsequent analysis and decision-making.

[0053] Optionally, the first clustering result is checked to obtain a checking result corresponding to the first clustering result, and an updating operation is performed on the first clustering result based on the checking result to obtain a second clustering result, including: determining clustering index data of the first clustering result based on a preset clustering index calculation method, wherein the clustering index data includes silhouette coefficient, Davies-Bouldin index and variance ratio standard; if at least two of the silhouette coefficient, Davies-Bouldin index and variance ratio standard meet the corresponding preset index threshold, it is determined that the checking result corresponding to the first clustering result is checking success, and the first clustering result is not updated. The first clustering result is determined as the second clustering result; if at least two of the silhouette coefficient, Davies-Bouldin index and variance ratio standard do not meet the corresponding preset index threshold, it is determined that the checking result corresponding to the first clustering result is checking failure, and the clustering model is controlled to be optimized. The dimensionality-reduced data is continuously clustered based on the optimized clustering model until the second clustering result is obtained, wherein at least two of the silhouette coefficient, Davies-Bouldin index and variance ratio standard corresponding to the second clustering result meet the corresponding preset index threshold.

[0054] Specifically, first, a preset clustering index calculation method is called to process the first clustering result, and the silhouette coefficient, Davies-Bouldin index and variance ratio standard clustering index data corresponding to the first clustering result are calculated. The silhouette coefficient is used to measure the similarity within the cluster and the difference between the clusters. The Davies-Bouldin index is used to evaluate the ratio of the distance between clusters and the distance within the cluster. The variance ratio standard is used to reflect the ability of the clustering interpretation data variance. Then, the indexes are compared with the corresponding preset threshold. If at least two indexes meet the threshold, the verification result is successful, and the first clustering result is directly determined as the second clustering result. If at least two indexes do not meet the threshold, the verification result is failed. At this time, the clustering model is optimized by, for example, adjusting the cluster number and distance calculation method, and the dimensionality reduction data is clustered again by using the optimized model. The verification and optimization process is repeated until the second clustering result that meets the threshold of at least two indexes is obtained.

[0055] In some embodiments, the clustering algorithm in the clustering model takes K-means as an example, and the clustering model is optimized as follows:

[0056] (1) Initialize the center point: store the preprocessed dimensionality reduction data set X in the distributed file system HDFS. Select k initial cluster center points from the data set to ensure that the k points are sufficiently dispersed to reduce the number of iterations required for convergence as much as possible. Write the center points to a file.

[0057] (2) Distributed computing map stage: perform sharding processing on the data set X according to a preset time window, read a data shard and the current center point. Each center point corresponds to a cluster, and each cluster has its cluster ID. Calculate the Euclidean distance from each data point to the center point selected in the previous step, and assign each point to the cluster corresponding to the nearest cluster center point. Output the vector sum and count of the data points (which may have duplicate values) and the cluster ID.

[0058] (3) Reduce phase of distributed computing: perform iteration model optimization processing. In each iteration process, read the data point vector and data volume of the same cluster, and calculate the average distance between the data points in each cluster after iteration as the density estimation value. The larger the average value, the smaller the density value, where D i represents the sum of distances of all points within the cluster, represents the average density, and i represents the density estimation value, n i represents the number of data points. Thus, the cluster with a significantly lower density than other clustering centers is shifted to a high density, improving the clustering effect, where i represents the cluster sequence number.

[0059] ​(4) Repeat steps (2), (3), and update the center point of the cluster ID each time until the center point changes less than the preset change threshold, and the iteration can be stopped. Thus, the optimized clustering model is obtained.

[0060] In the embodiment, the clustering quality is objectively evaluated by multi-dimensional quantitative index verification, and subjective judgment deviation is avoided. The model optimization mechanism when the verification fails forms a closed loop iteration, and the reliability of the clustering result is improved. The second clustering result obtained after strict index verification has better intra-cluster consistency and inter-cluster distinguishability, and can provide more accurate structured data support for subsequent analysis.

[0061] S140, performing global anomaly determination processing on the second clustering result to obtain a global anomaly determination result, updating the second clustering result based on the global anomaly determination result to obtain a third clustering result.

[0062] The global anomaly determination result can be understood as a determination result representing whether there is global anomaly data in the clustering result. The determination information about the anomaly data obtained after global analysis of the second clustering result includes the specific identification of the identified anomaly data, the deviation degree of each anomaly data from the global data distribution, and other contents. The third clustering result is obtained by updating the second clustering result based on the global anomaly determination result. The anomaly data is separated from the original cluster or divided into an anomaly cluster, so that the consistency of the normal data in the cluster is stronger, and the clustering structure more accurately reflects the distribution of the normal data, providing a more reliable basis for subsequent analysis and decision-making.

[0063] Specifically, a global anomaly determination algorithm can be called to perform global anomaly determination processing on the second clustering result. By calculating the deviation of each data point from the cluster center and analyzing the isolation degree of the data point in the global data distribution, the anomaly data that does not conform to any cluster characteristics is identified, and the global anomaly determination result containing at least the anomaly data identifier and the cluster identifier is generated. Then, the second clustering result is updated according to the global anomaly determination result. If global anomaly data is identified, the data determined as abnormal is separated from the original cluster or divided into an anomaly cluster, and the third clustering result is finally obtained. For example, if the anomaly data identifier of a data point is true, the data point is determined as global anomaly data. If the anomaly data identifier of a data point is false, the data point is determined as non-global anomaly data. It should be noted that the representation form of the anomaly data identifier can be set according to the requirements, which is not limited herein.

[0064] In the embodiment, the global anomaly determination can effectively filter out interference data and avoid the destruction of the cluster structure by the abnormal values; the purity of the data in the cluster is higher by updating the obtained third clustering result, the clustering structure is more consistent with the real distribution of the normal data, the practicability and reliability of the clustering result are improved, and a more accurate data basis is provided for subsequent analysis or decision based on the clustering result.

[0065] Optionally, the second clustering result is subjected to a global anomaly determination processing to obtain a global anomaly determination result, and the second clustering result is subjected to an updating processing based on the global anomaly determination result to obtain a third clustering result, including: performing a global anomaly data screening processing in the second clustering result based on a preset global anomaly data screening condition to obtain a global anomaly screening result; if the global anomaly screening result is that there is global anomaly data, the global anomaly data is removed from the corresponding cluster to obtain the third clustering result; if the global anomaly screening result is that there is no global anomaly data, the second clustering result is determined as the third clustering result.

[0066] The preset global anomaly data screening condition can be understood as a condition set for screening global anomaly data. In the embodiment, it can be a series of rules or standards set in advance for identifying global anomaly data from the second clustering result, which are formulated based on the distribution characteristics, distance metrics and other characteristics of the data, such as the distance between the data point and the cluster center exceeding a certain threshold, the feature value of the data point exceeding the reasonable fluctuation range of the global data, the isolation degree of the data point in the global dimension reaching a preset level, etc., which provides clear and quantifiable basis for the determination of abnormal data. The global anomaly screening result can be understood as the result obtained by screening the second clustering result according to the preset global anomaly data screening condition, mainly including the determination result of whether there is global anomaly data, and if there is global anomaly data, the specific identifier or position of the global anomaly data is also specified, which directly reflects the data situation in the second clustering result that does not conform to the normal cluster characteristics.

[0067] Specifically, all data points in the second clustering result can be screened according to the preset global anomaly data screening condition to obtain the global anomaly screening result of whether there is global anomaly data; if the screening result shows that there is global anomaly data, these global anomaly data are removed from the cluster to which they belong to form a new clustering structure as the third clustering result; if the screening result shows that there is no global anomaly data, the second clustering result is directly determined as the third clustering result.

[0068] In the embodiment, the global exception determination is performed through the explicit global exception screening condition, so as to ensure the objectivity and consistency of the global exception data identification; the third clustering result obtained after the global exception data is removed reduces the interference of the abnormal value on the clustering structure, so that the homogeneity of the data in the cluster is stronger, the clustering boundary is clearer, the purity and reliability of the clustering result are improved, and a higher-quality data basis is provided for subsequent clustering-based analysis or application.

[0069] In S150, for each clustering cluster in the third clustering result, a local exception evaluation value of each data record in the clustering cluster is determined through a local exception evaluation algorithm, and local exception data is determined based on the local exception evaluation value of each data record in each clustering cluster.

[0070] The local exception evaluation value can be understood as a value for representing the local exception degree of the data record in the clustering result. The local exception evaluation value of each data record in each clustering cluster in the third clustering result can be calculated through a local exception evaluation algorithm. The local exception evaluation algorithm includes but is not limited to a local outlier factor calculation method, a connected outlier factor calculation method, and an influence outlier factor calculation method. The local exception data can be understood as a data record that is obviously deviated from the distribution characteristics of most data in the local range of the cluster and is screened out from each clustering cluster based on the local exception evaluation value. Such data may not be determined as an exception at the global level, but shows obvious inconsistency in the local structure of the cluster to which it belongs. Whether the data record is local exception data can be determined according to the local exception evaluation value of the data record according to a preset local exception determination rule. For example, the preset local exception determination rule can be set as: if the exception evaluation value of the data record is greater than or equal to a preset exception evaluation threshold, the data record is determined as local exception data; if the exception evaluation value of the data record is less than the preset exception evaluation threshold, the data record is determined as non-local exception data. Optionally, in the case where a plurality of exception evaluation values corresponding to each data record are obtained, the preset local exception determination rule can be set as: the plurality of exception evaluation values can be normalized and averaged, and then the obtained exception evaluation average value is compared with a corresponding preset exception evaluation average threshold. If the exception evaluation average value is greater than the preset exception evaluation average threshold, the data record is determined as local exception data. Alternatively, the plurality of exception evaluation values corresponding to each data record can be compared with the corresponding preset exception evaluation threshold. If at least two exception evaluation values are greater than the corresponding preset exception evaluation threshold, the data record is determined as local exception data. The specific setting is determined according to the actual exception data identification requirement, which is not listed here. It should be noted that the data record and the data point represent the same content.

[0071] Specifically, for each cluster in the third clustering result, a local anomaly evaluation algorithm is applied respectively to calculate the deviation of each data record in the cluster relative to its adjacent data, for example, the LOF value reflects the difference between the local density of the data point and the surrounding points, so as to obtain the local anomaly evaluation value of each record; then according to the local anomaly evaluation value of each record and the corresponding preset anomaly threshold, the local anomaly of each data record is determined according to the preset local anomaly determination rule, and the local anomaly data is filtered out from each cluster according to the determination result.

[0072] In the embodiment, by focusing on the data relationship inside each cluster, the hidden local anomaly in the cluster that may be ignored by global analysis can be accurately identified, such as the individual point obviously deviating from the local structure in a cluster with overall dense distribution; the clustering evaluation improves the pertinence and accuracy of anomaly identification, and further purifies the clustering result, providing a more pure data basis for subsequent fine analysis of each cluster.

[0073] Optionally, the local anomaly evaluation value of each data record in the cluster is determined by the local anomaly evaluation algorithm, including: each data record in the cluster is processed by the local anomaly evaluation algorithm to determine the corresponding local anomaly evaluation value of each data record, wherein the local anomaly evaluation value includes one or more of a local outlier factor value, a connected outlier factor value and an influence outlier factor value.

[0074] The local outlier factor value is used to reflect the density deviation of the data point relative to the neighborhood points, which can be calculated by a local outlier factor calculation method; the connected outlier factor value is used to reflect the abnormal degree of the connection between the data point and the neighborhood points, which can be calculated by a connected outlier factor calculation method; and the influence outlier factor value is used to measure the abnormal influence degree of the data point on the neighborhood data distribution, which can be calculated by an influence outlier factor calculation method.

[0075] Specifically, for all data records in each cluster in the third clustering result, a local anomaly evaluation algorithm is applied to process each data record one by one to determine the anomaly evaluation value of each data record. In the processing process, the density difference, connection relationship strength and influence degree of the data record on the surrounding data distribution are calculated to determine the local anomaly evaluation value corresponding to each data record, which can include one or more of a local outlier factor value, a connected outlier factor value and an influence outlier factor value.

[0076] In this embodiment, the multiple types of outlier factor values from different dimensions depict the local abnormal characteristics of the data records, can more comprehensively and accurately reflect the abnormal state of the data in the cluster, and avoid misjudgment caused by a single indicator; meanwhile, the algorithm is targeted for each data record, ensuring the delicacy and accuracy of the evaluation results, and providing a rich and reliable basis for subsequent judgment of local abnormal data.

[0077] Optionally, determining the local abnormal data based on the local abnormal evaluation value of each data record in each cluster includes: for any cluster, traversing the cluster to obtain the local abnormal evaluation value of each data record in the cluster; for any data record, if the local abnormal evaluation value of the data record meets the local abnormal screening condition, the data record is determined as the local abnormal data; if the local abnormal evaluation value of the data record does not meet the local abnormal screening condition, the data record is determined as the non-local abnormal data.

[0078] The local abnormal screening condition can be understood as a set of rules set for identifying the local abnormal data records from the data cluster, and the core logic is to determine the abnormality by comparing the feature difference between the data record and the local environment. By setting multi-dimensional quantitative or qualitative thresholds, the local abnormal screening condition can accurately capture the data records that may not be obvious in the global perspective but show significant abnormal characteristics in the local environment, thereby effectively identifying the local small-range abnormality, which can be set based on different types of local abnormal evaluation values and thresholds of each type of local abnormal evaluation value.

[0079] Specifically, for each cluster in the third clustering result, all data records in the cluster are traversed one by one, and the local abnormal evaluation value of each data record obtained by the local abnormal evaluation algorithm is extracted; then the local abnormal evaluation value of each data record is compared with the preset local abnormal screening condition, if the local abnormal screening condition is met, the data record is determined as the local abnormal data, otherwise the data record is determined as the non-local abnormal data.

[0080] In this embodiment, by means of cluster-by-cluster traversal and record-by-record verification, the comprehensiveness of local abnormal data identification is ensured, and hidden abnormalities within the cluster are avoided; based on the explicit screening condition for determination, the objectivity and consistency of abnormal identification are ensured, the data conforming to the local distribution characteristics and the data deviating from the local structure within the cluster are effectively distinguished, and the purity of each cluster is further improved, providing accurate basis for subsequent in-depth analysis of normal data within the cluster.

[0081] S160, performing abnormal marking processing on the local abnormal data, and if the global abnormality determination result exists the global abnormal data, performing abnormal marking processing on the global abnormal data.

[0082] Specifically, for the identified local abnormal data, a preset marking rule can be used for separate marking, such as adding a specific label, marking the abnormal type, etc., to distinguish the normal data within the cluster. At the same time, if the previous global abnormality determination result shows that there is global abnormal data, the same abnormal marking processing is performed on these global abnormal data, and a unified or differentiated marking method is used to clearly indicate the abnormal properties.

[0083] Optionally, after the abnormal marking processing of the local abnormal data and / or the abnormal marking processing of the global abnormal data, the pre-warning prompt information can be generated according to the data after the abnormal marking processing, and the pre-warning prompt information is transmitted to the upstream data management system and / or the downstream service system for pre-warning and providing data modification basis for the upstream data management system and / or the downstream service system.

[0084] In this embodiment, through clear abnormal marking, normal data and various types of abnormal data such as local abnormal data and global abnormal data can be intuitively distinguished, which facilitates subsequent data screening, analysis or model training to quickly identify and process abnormal information. At the same time, the classification marking makes it easier to trace the source and nature of the abnormal data, providing clear clues for in-depth exploration of the causes of abnormality and optimization of data quality, and improving the pertinence and efficiency of data processing.

[0085] The technical scheme of the embodiment is that, by acquiring the to-be-recognized data, the to-be-recognized data is subjected to dimension reduction processing to obtain reduced dimension data, wherein the to-be-recognized data comprises multi-dimensional data; the reduced dimension data is subjected to clustering processing by a pre-trained clustering model to obtain a first clustering result; the first clustering result is subjected to verification processing to obtain a verification result corresponding to the first clustering result, the first clustering result is subjected to updating operation based on the verification result to obtain a second clustering result; the second clustering result is subjected to global anomaly determination processing to obtain a global anomaly determination result, the second clustering result is subjected to updating processing based on the global anomaly determination result to obtain a third clustering result; for each clustering cluster in the third clustering result, a local anomaly evaluation value of each data record in the clustering cluster is determined by a local anomaly evaluation algorithm, and local anomaly data is determined based on the local anomaly evaluation value of each data record in each clustering cluster; the local anomaly data is subjected to anomaly marking processing, and if the global anomaly determination result exists global anomaly data, the global anomaly data is subjected to anomaly marking processing. The scheme can effectively reduce data redundancy and computational complexity by first performing dimension reduction processing on multi-dimensional to-be-recognized data, laying an efficient foundation for subsequent clustering; after obtaining the first clustering result by means of the pre-trained clustering model, the second clustering result is obtained by updating through verification processing, which can correct the deviation of the initial clustering and improve the clustering accuracy; the third clustering result is obtained by updating the second clustering result through global anomaly determination, which can first eliminate the global anomaly that deviates from the overall distribution, and optimize the overall quality of the clustering cluster; then, the local anomaly evaluation and screening marking are performed on each cluster in the third clustering result, which can accurately identify the hidden local anomaly in the cluster and further purify the cluster data; at the same time, the global and local anomaly data are marked respectively, which ensures the comprehensiveness of anomaly recognition, solves the problems of inaccurate and low-efficiency anomaly data recognition, and provides a high-quality and high-identification data basis for subsequent data application. The overall process takes into account the efficiency and accuracy, and the anomaly processing is hierarchical and targeted.

[0086] Embodiment two

[0087] Figure 2 is a flowchart of a data processing method provided by the embodiment two of the application, the method of the embodiment is a further optimization of the method of the above-mentioned embodiment. Optionally, the comprehensive weight of each data dimension in the to-be-recognized data is determined, the to-be-recognized data is subjected to weighting processing based on the comprehensive weight of each data dimension to obtain the to-be-recognized data subjected to weighting processing, and the to-be-recognized data subjected to weighting processing is subjected to dimension reduction processing by a preset dimension reduction processing algorithm to obtain reduced dimension data, wherein the preset dimension reduction processing algorithm comprises UMAP dimension reduction, PacMAP dimension reduction and TriMAP dimension reduction. As shown in Figure 2 , the method comprises:

[0088] S210, acquire the to-be-recognized data, determine the comprehensive weight of each data dimension in the to-be-recognized data, perform weighted processing on the to-be-recognized data based on the comprehensive weight of each data dimension, and obtain the to-be-recognized data after weighted processing.

[0089] The comprehensive weight is a weight value determined by comprehensively considering the importance of each dimension in the multi-dimensional data in combination with the data characteristics, the business scenario, and the preset evaluation standard. The comprehensive weight is not derived from a single method (such as only subjective experience or only objective data distribution), but is finally formed by combining subjective weighting methods (such as the analytic hierarchy process and the Delphi method) and objective weighting methods (such as the entropy weight method and the coefficient of variation method), or by fusing the weight results of different evaluation systems, so as to comprehensively reflect the actual contribution of each dimension in the overall analysis, decision-making, or evaluation. For example, in user behavior analysis, the comprehensive weight of “browsing time” and “transaction frequency” not only refers to the dispersion degree of the data itself, but also combines the transaction behavior in the business, which can better reflect the user value, so as to more reasonably balance the influence of each dimension and provide a basis more suitable for actual business needs for subsequent data processing. In this embodiment, a comprehensive weight determination method can be called in advance to determine the comprehensive weight of each data dimension, and the comprehensive weight of each data dimension is stored in a preset storage space. In the case of weighted processing of the to-be-recognized data, the comprehensive weight of each data dimension can be directly called. It should be noted that a timing task can be set to trigger the update processing of the comprehensive weight of each data dimension at regular intervals, so as to ensure the real-time and accuracy of data processing.

[0090] Specifically, the to-be-recognized data containing multi-dimensional features is collected from a data source or a specified data storage space, and the comprehensive weight of each data dimension is called. The original data of each data dimension is weighted according to the comprehensive weight of each data dimension, such as multiplying each dimension value by the corresponding comprehensive weight and then summarizing, to obtain the to-be-recognized data after weighted processing.

[0091] In this embodiment, the comprehensive weight reflects the importance difference of different dimensions, avoiding the dilution of key information caused by equal treatment of each dimension. The data after weighted processing is more suitable for actual analysis needs, highlights the influence of core features, improves the pertinence and accuracy of subsequent data processing, and enhances the interpretability of data.

[0092] Optionally, the comprehensive weight of each data dimension in the to-be-recognized data is determined, including: acquiring a business scenario data set, wherein the business scenario data set includes a plurality of business scenarios and first weight data corresponding to each business scenario; for each business scenario, acquiring second weight data of each data dimension in the to-be-recognized data under the business scenario; and determining the comprehensive weight of each data dimension based on the second weight data of each data dimension under each business scenario and the first weight data corresponding to each business scenario.

[0093] The first weight data specifically refers to a weight value set for each business scenario in the business scenario data set, which is used to measure the importance or priority of different business scenarios in the overall analysis. For example, in the e-commerce scenario, the first weight of the "promotion period" scenario may be higher than that of the "daily sales period", reflecting its greater impact on data evaluation. The second weight data specifically refers to the weight value of each data dimension in the to-be-identified data for a specific business scenario, reflecting the relative importance of the data dimension in the business scenario. For example, in the "promotion period" scenario, the second weight of the "discount sensitivity" dimension may be higher than that of the "historical transaction frequency". The comprehensive weight specifically refers to the final weight of each data dimension calculated by combining the first weight data and the second weight data. It is obtained by weighting and summing the second weight of the dimension in the scenario with the first weight of the business scenario as the coefficient, which reflects not only the priority difference of different scenarios, but also the importance of the dimension in each scenario, and can fully reflect the actual contribution of the dimension in the overall business analysis.

[0094] Specifically, first, a business scenario data set covering multiple business scenarios is collected, which includes first weight data corresponding to each scenario, reflecting the importance of different business scenarios. Then, for each business scenario, the second weight data of each dimension in the to-be-identified data is determined through business rule analysis, historical data training or expert evaluation, reflecting the relative importance of the dimension in the specific scenario. Finally, the second weight in the corresponding scenario is weighted and summed with the first weight of each business scenario as the coefficient, for example, the comprehensive weight of a certain dimension = scene A first weight x scene A dimension second weight + scene B first weight x scene B dimension second weight + …, to obtain the comprehensive weight of each data dimension.

[0095] In this embodiment, by determining the comprehensive weight, the to-be-identified data is weighted and processed, which not only considers the priority difference of different business scenarios, but also takes into account the adaptability of the dimension in each scenario, and can more comprehensively fit the actual business logic. Compared with the weight calculation of a single scenario or a single dimension, the comprehensive weight obtained by this method is more universal and targeted, which can make the subsequent data weighting processing more accurately reflect the core business needs and improve the effectiveness of data application.

[0096] S220, performing dimension reduction processing on the weighted to-be-identified data by a preset dimension reduction processing algorithm to obtain reduced data.

[0097] The preset dimension reduction processing algorithm includes UMAP dimension reduction, PacMAP dimension reduction and TriMAP dimension reduction.

[0098] Specifically, a preset dimension reduction processing algorithm is used to perform dimension reduction operation on the weighted processed to-be-recognized data, and the preset dimension reduction processing algorithm includes but is not limited to UMAP dimension reduction, PacMAP dimension reduction and TriMAP dimension reduction. The algorithm can be selected according to the feature distribution and complexity of the data. For example, UMAP dimension reduction is suitable for preserving local structure, PacMAP dimension reduction is good at balancing local and global features, and TriMAP dimension reduction is stable in high-dimensional data dimension reduction. Through algorithm calculation, high-dimensional data is mapped to low-dimensional space, such as two-dimensional or three-dimensional, redundant features are removed and core distribution rules are preserved, and finally dimension-reduced data is obtained.

[0099] For example, the UMAP dimension reduction method is used to perform dimension reduction processing on the data. After the downstream data service system consumes the data every day, the data is stored in the database. The customer number, label 1 value, label 2 value, …, label m, index value 1, index value 2, …, index value n, time dimension feature 1, time dimension feature 2, …, time dimension feature k are extracted from the database. The extracted data is used as to-be-recognized data. Optionally, the time dimension (day of the month, day of the week) can be added as a feature to enhance the time correlation. At the same time, according to the business importance analysis data table, important label index values are analyzed to preliminarily reduce the dimension of the business, and normalization processing is required before UMAP dimension reduction. Since there are no obvious outliers in normal data, Min-Max normalization is used, where x is the original data, x min and x max are the maximum and minimum values in the data, and x normal is the normalized data, and the calculation method is as follows:

[0100] x normal =(x-x min ) / (x max -x min );

[0101] The UMAP dimension reduction target is to find a low-dimensional space corresponding to the customer and the label index value, so that the topological structure of the data in the low-dimensional space is as similar as possible to the structure in the high-dimensional space. First, a high-dimensional space graph is constructed. We mainly ensure that each edge of the high-dimensional space is similar to the low-dimensional space, and the importance of each edge is also different, which is mainly represented by weight. The high-dimensional weight w ij is calculated as follows, where x i and x j represent two points in the high-dimensional space, p i is the distance from the data point x i to the nearest neighbor point, and s i is an adaptive parameter to ensure smooth weight distribution:

[0102] w ij= exp(-(d(x i ,x j )-ρ i ) / σ i )。

[0103] Similarly, a graph can be constructed in a low-dimensional space, and a low-dimensional weight q ij can be defined using a probability distribution function, y i , y j represent two points in a low-latitude space, where a and b are both probability distribution parameters, and exemplary, a = 1.929 and b = 0.7915 can be set, and a low-dimensional weight q ij The calculation formula is as follows:

[0104]

[0105] Then the computer optimizes the loss function, minimizes the difference between the high-dimensional weight and the low-dimensional weight, and the cross-entropy loss function is as follows:

[0106] C = ∑w ij log(w ij / q ij )+(1-w ij )log((1-w ij ) / (1-q ij ));

[0107] The coordinates can be updated by calculating the gradient of the loss function, where t represents the current iteration number, η is the learning rate, m is the maximum iteration number set fixed, t ∈ (0, m), and the convergence condition and performance of the UMAP algorithm are optimized after adding the decay factor in the calculation process. The learning rate is calculated in each iteration process as follows:

[0108] η t+1 = (m-t)η t / m;

[0109] The low-dimensional coordinate iteration calculation formula is as follows:

[0110]

[0111] In this embodiment, the selection of multiple dimension reduction algorithms can adapt to different data characteristics and analysis requirements, and flexibly retain the core structure of the data. The reduced data reduces redundant features and computational burden, which not only facilitates efficient execution of subsequent clustering and other processing, but also avoids the interference of dimension disaster on the results, thereby improving the reliability and applicability of the dimension reduction results.

[0112] S230, cluster the reduced data by using the pre-trained clustering model to obtain a first clustering result.

[0113] S240, a verification process is performed on the first clustering result to obtain a verification result corresponding to the first clustering result, an updating operation is performed on the first clustering result based on the verification result, and a second clustering result is obtained.

[0114] S250, a global anomaly determination process is performed on the second clustering result to obtain a global anomaly determination result, an updating process is performed on the second clustering result based on the global anomaly determination result, and a third clustering result is obtained.

[0115] S260, for each clustering cluster in the third clustering result, a local anomaly evaluation value of each data record in the clustering cluster is determined through a local anomaly evaluation algorithm, and local anomaly data is determined based on the local anomaly evaluation value of each data record in each clustering cluster.

[0116] S270, an anomaly marking process is performed on the local anomaly data, and if there is global anomaly data in the global anomaly determination result, an anomaly marking process is performed on the global anomaly data.

[0117] The technical scheme of the embodiment is as follows: the data to be identified is acquired, the comprehensive weight of each data dimension in the data to be identified is determined, the data to be identified is weighted based on the comprehensive weight of each data dimension, and the data to be identified after the weighting is obtained; the data to be identified after the weighting is processed by a preset dimension reduction processing algorithm, and the data after the dimension reduction is obtained, wherein the preset dimension reduction processing algorithm includes UMAP dimension reduction, PacMAP dimension reduction and TriMAP dimension reduction; the data after the dimension reduction is processed by a pre-trained clustering model, and the first clustering result is obtained; the first clustering result is verified, and the verification result corresponding to the first clustering result is obtained; the first clustering result is updated based on the verification result, and the second clustering result is obtained; the second clustering result is subjected to global anomaly determination processing, and the global anomaly determination result is obtained; the second clustering result is updated based on the global anomaly determination result, and the third clustering result is obtained; for each clustering cluster in the third clustering result, the local anomaly evaluation value of each data record in the clustering cluster is determined by a local anomaly evaluation algorithm, and the local anomaly data is determined based on the local anomaly evaluation value of each data record in each clustering cluster; the local anomaly data is subjected to anomaly marking processing, and if the global anomaly determination result exists, the global anomaly data is subjected to anomaly marking processing. The scheme can make the data dimension importance fit the business scenario and the multi-dimensional feature through comprehensive weight processing, avoid weakening of key information, introduce dimension reduction algorithms such as UMAP, PacMAP and TriMAP, retain the core structure while reducing the dimension, improve the subsequent clustering efficiency and accuracy, correct the initial clustering deviation by combining the pre-trained clustering model with the verification update mechanism, enhance the reliability of the first clustering result, and realize the comprehensiveness and accuracy of anomaly detection through hierarchical processing of global anomaly determination and local anomaly evaluation, which can identify the global anomaly deviating from all clustering clusters and deeply mine the local outlier data in each clustering cluster. Finally, the two types of anomaly data are marked respectively, so that the anomaly information is not missed, the overall process considers the business adaptability, data processing efficiency and integrity of anomaly detection, and can provide high-quality anomaly identification results for data monitoring, risk early warning and other scenes.

[0118] Embodiment three

[0119] Figure 3 is a structural schematic diagram of a data processing device provided by the embodiment three of the application. As shown in the figure, Figure 3 the device comprises:

[0120] The data dimension reduction processing module 310 is configured to acquire the data to be identified, perform dimension reduction processing on the data to be identified, and obtain the data after the dimension reduction, wherein the data to be identified comprises multi-dimensional data.

[0121] The first clustering result determination module 320 is configured to perform clustering processing on the reduced data by using the pre-trained clustering model to obtain a first clustering result.

[0122] The second clustering result determination module 330 is configured to perform verification processing on the first clustering result to obtain a verification result corresponding to the first clustering result, perform updating operation on the first clustering result based on the verification result, and obtain a second clustering result.

[0123] The third clustering result determination module 340 is configured to perform global anomaly determination processing on the second clustering result to obtain a global anomaly determination result, perform updating processing on the second clustering result based on the global anomaly determination result, and obtain a third clustering result.

[0124] The local anomaly data determination module 350 is configured to, for each clustering cluster in the third clustering result, determine a local anomaly evaluation value of each data record in the clustering cluster by using a local anomaly evaluation algorithm, and determine local anomaly data based on the local anomaly evaluation value of each data record in each clustering cluster.

[0125] The anomaly marking processing module 360 is configured to perform anomaly marking processing on the local anomaly data, and perform anomaly marking processing on global anomaly data if the global anomaly determination result exists.

[0126] The technical scheme of the embodiment comprises the following steps: a data dimension reduction processing module obtains to-be-recognized data, performs dimension reduction processing on the to-be-recognized data, and obtains data after dimension reduction, wherein the to-be-recognized data comprises multi-dimensional data; a first clustering result determination module performs clustering processing on the data after dimension reduction by using a pre-trained clustering model, and obtains a first clustering result; a second clustering result determination module performs verification processing on the first clustering result, obtains a verification result corresponding to the first clustering result, performs updating operation on the first clustering result based on the verification result, and obtains a second clustering result; a third clustering result determination module performs global anomaly judgment processing on the second clustering result, obtains a global anomaly judgment result, performs updating processing on the second clustering result based on the global anomaly judgment result, and obtains a third clustering result; a local anomaly data determination module determines, for each clustering cluster in the third clustering result, a local anomaly evaluation value of each data record in the clustering cluster by using a local anomaly evaluation algorithm, and determines local anomaly data based on the local anomaly evaluation value of each data record in each clustering cluster; and an anomaly marking processing module performs anomaly marking processing on the local anomaly data, and performs anomaly marking processing on global anomaly data if the global anomaly judgment result exists.

[0127] On the basis of the above embodiment, the data dimension reduction processing module 310 is specifically configured to determine comprehensive weights of each data dimension in the to-be-recognized data, perform weighted processing on the to-be-recognized data based on the comprehensive weights of each data dimension, and obtain to-be-recognized data after weighted processing; perform dimension reduction processing on the to-be-recognized data after weighted processing by using a preset dimension reduction processing algorithm, and obtain data after dimension reduction, wherein the preset dimension reduction processing algorithm comprises UMAP dimension reduction, PacMAP dimension reduction, and TriMAP dimension reduction.

[0128] Optionally, the data dimension reduction processing module 310 is specifically configured to obtain a service scenario data set, wherein the service scenario data set includes a plurality of service scenarios and first weight data corresponding to each service scenario; for each service scenario, obtain second weight data of each data dimension in the to-be-identified data under the service scenario; and determine a comprehensive weight of each data dimension based on the second weight data of each data dimension under each service scenario and the first weight data corresponding to each service scenario.

[0129] Optionally, the first clustering result determination module 320 is specifically configured to perform data format conversion processing on the reduced data to obtain data meeting input requirements of a pre-trained clustering model; input the data meeting the input requirements of the pre-trained clustering model into the pre-trained clustering model to obtain a first clustering result, wherein the pre-trained clustering model includes at least two clustering algorithms, and the at least two clustering algorithms are switched according to a data distribution form of input data of the pre-trained clustering model.

[0130] Optionally, the second clustering result determination module 330 is specifically configured to determine clustering index data of the first clustering result based on a preset clustering index calculation method, wherein the clustering index data includes a silhouette coefficient, a Davies-Bouldin index, and a variance ratio criterion; if at least two of the silhouette coefficient, the Davies-Bouldin index, and the variance ratio criterion meet corresponding preset index thresholds, it is determined that a verification result corresponding to the first clustering result is verification success, and no update operation is performed on the first clustering result, and the first clustering result is determined as a second clustering result; if at least two of the silhouette coefficient, the Davies-Bouldin index, and the variance ratio criterion do not meet the corresponding preset index thresholds, it is determined that the verification result corresponding to the first clustering result is verification failure, and the clustering model is controlled to be optimized, and the reduced data is continuously clustered based on the optimized clustering model until the second clustering result is obtained, wherein at least two of the silhouette coefficient, the Davies-Bouldin index, and the variance ratio criterion corresponding to the second clustering result meet the corresponding preset index thresholds.

[0131] Optionally, the third clustering result determination module 340 is specifically configured to perform global abnormal data screening processing in the second clustering result based on a preset global abnormal data screening condition to obtain a global abnormal screening result; if the global abnormal screening result is that there is global abnormal data, the global abnormal data is excluded from the corresponding clustering cluster to obtain a third clustering result; if the global abnormal screening result is that there is no global abnormal data, the second clustering result is determined as the third clustering result.

[0132] Optionally, the local anomaly data determination module 350 is specifically configured to determine a local anomaly evaluation value corresponding to each data record in the cluster by processing each data record in the cluster through a local anomaly evaluation algorithm, wherein the anomaly evaluation result includes one or more of a local outlier factor value, a connected outlier factor value and an influence outlier factor value.

[0133] Optionally, the local anomaly data determination module 350 is further specifically configured to, for any cluster, traverse the cluster to obtain the local anomaly evaluation value of each data record in the cluster; for any data record, if the local anomaly evaluation value of the data record satisfies a local anomaly screening condition, determine that the data record is local anomaly data; and if the local anomaly evaluation value of the data record does not satisfy the local anomaly screening condition, determine that the data record is non-local anomaly data.

[0134] The data processing apparatus provided in the embodiments of the present application can execute the data processing method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0135] Embodiment Four

[0136] Figure 4 is a structural schematic diagram of an electronic device provided in Embodiment Four of the present application. The electronic device 10 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices (such as headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0137] As shown in Figure 4 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is in communication with the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0138] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0139] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as a data processing method.

[0140] In some embodiments, the data processing method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the data processing method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the data processing method by any other appropriate means, such as by means of firmware.

[0141] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0142] A computer program for implementing the data processing method of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor, implements the functions / operations specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine, or entirely on a remote machine or server.

[0143] Embodiment five

[0144] Embodiment five of the present application also provides a computer readable storage medium, which stores computer instructions for causing a processor to execute a data processing method, the method comprising:

[0145] Obtaining to-be-identified data, performing dimension reduction processing on the to-be-identified data to obtain reduced dimension data, wherein the to-be-identified data comprises multi-dimensional data;

[0146] Performing clustering processing on the reduced dimension data through a pre-trained clustering model to obtain a first clustering result;

[0147] Performing verification processing on the first clustering result to obtain a verification result corresponding to the first clustering result, performing updating operation on the first clustering result based on the verification result to obtain a second clustering result;

[0148] Performing global anomaly determination processing on the second clustering result to obtain a global anomaly determination result, performing updating processing on the second clustering result based on the global anomaly determination result to obtain a third clustering result;

[0149] For each clustering cluster in the third clustering result, determining a local anomaly evaluation value of each data record in the clustering cluster through a local anomaly evaluation algorithm, and determining local anomaly data based on the local anomaly evaluation value of each data record in each clustering cluster;

[0150] Performing anomaly marking processing on the local anomaly data, and if there is global anomaly data in the global anomaly determination result, performing anomaly marking processing on the global anomaly data.

[0151] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0152] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0153] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0154] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0155] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0156] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: acquiring to-be-identified data, performing dimension reduction processing on the to-be-identified data to obtain reduced data, wherein the to-be-identified data comprises multidimensional data; performing clustering processing on the reduced data by using a pre-trained clustering model to obtain a first clustering result; performing verification processing on the first clustering result to obtain a verification result corresponding to the first clustering result, performing updating operation on the first clustering result based on the verification result to obtain a second clustering result; performing global anomaly judgment processing on the second clustering result to obtain a global anomaly judgment result, performing updating processing on the second clustering result based on the global anomaly judgment result to obtain a third clustering result; for each clustering cluster in the third clustering result, determining a local anomaly evaluation value of each data record in the clustering cluster by using a local anomaly evaluation algorithm, and determining local anomaly data based on the local anomaly evaluation value of each data record in each clustering cluster; performing anomaly marking processing on the local anomaly data, and if there is global anomaly data in the global anomaly judgment result, performing anomaly marking processing on the global anomaly data.

2. The method of claim 1, wherein, The method comprises the following steps: determining comprehensive weights of each data dimension in the to-be-identified data, performing weighted processing on the to-be-identified data based on the comprehensive weights of each data dimension to obtain weighted to-be-identified data; performing dimension reduction processing on the weighted to-be-identified data by using a preset dimension reduction processing algorithm to obtain the reduced data, wherein the preset dimension reduction processing algorithm comprises UMAP dimension reduction, PacMAP dimension reduction and TriMAP dimension reduction.

3. The method of claim 2, wherein, The method comprises the following steps: acquiring a business scenario data set, wherein the business scenario data set comprises a plurality of business scenarios and first weight data corresponding to each business scenario; for each business scenario, acquiring second weight data of each data dimension in the to-be-identified data under the business scenario; determining comprehensive weights of each data dimension based on the second weight data of each data dimension under each business scenario and the first weight data corresponding to each business scenario.

4. The method of claim 1, wherein, The method comprises the following steps: performing data format conversion processing on the reduced data to obtain data meeting input requirements of the pre-trained clustering model; inputting the data meeting the input requirements of the pre-trained clustering model into the pre-trained clustering model to obtain the first clustering result, wherein the pre-trained clustering model comprises at least two clustering algorithms, and the at least two clustering algorithms are switched according to a data distribution form of input data of the pre-trained clustering model.

5. The method of claim 1, wherein, The method comprises the following steps: performing verification processing on the first clustering result to obtain a verification result corresponding to the first clustering result, performing updating operation on the first clustering result based on the verification result to obtain a second clustering result, The clustering index data of the first clustering result is determined based on a preset clustering index calculation method, wherein the clustering index data includes a silhouette coefficient, a Davies-Bouldin index, and a variance ratio criterion; If at least two of the silhouette coefficient, the Davies-Bouldin index, and the variance ratio criterion satisfy corresponding preset index thresholds, it is determined that the verification result corresponding to the first clustering result is verification success, and the first clustering result is not updated, and the first clustering result is determined as the second clustering result; If at least two of the silhouette coefficient, the Davies-Bouldin index, and the variance ratio criterion do not satisfy corresponding preset index thresholds, it is determined that the verification result corresponding to the first clustering result is verification failure, and the clustering model is controlled to be optimized, and the dimension-reduced data is continuously clustered based on the optimized clustering model until a second clustering result is obtained, wherein at least two of the silhouette coefficient, the Davies-Bouldin index, and the variance ratio criterion corresponding to the second clustering result satisfy corresponding preset index thresholds.

6. The method of claim 1, wherein, The second clustering result is subjected to global anomaly determination processing to obtain a global anomaly determination result, and the second clustering result is updated based on the global anomaly determination result to obtain a third clustering result, including: Global anomaly data in the second clustering result is filtered based on a preset global anomaly data filtering condition to obtain a global anomaly filtering result; If the global anomaly filtering result is that there is global anomaly data, the global anomaly data is excluded from the corresponding clustering cluster to obtain the third clustering result, and if the global anomaly filtering result is that there is no global anomaly data, the second clustering result is determined as the third clustering result.

7. The method of claim 1, wherein, The local anomaly evaluation value of each data record in the clustering cluster is determined by a local anomaly evaluation algorithm, including: Each data record in the clustering cluster is processed by the local anomaly evaluation algorithm to determine the local anomaly evaluation value corresponding to each data record, wherein the local anomaly evaluation value includes one or more of a local outlier factor value, a connected outlier factor value, and an impact outlier factor value.

8. The method of claim 1, wherein, The local anomaly data is determined based on the local anomaly evaluation value of each data record in each clustering cluster, including: For any clustering cluster, the local anomaly evaluation value of each data record in the clustering cluster is obtained by traversing the clustering cluster; For any data record, if the local anomaly evaluation value of the data record satisfies a local anomaly filtering condition, the data record is determined as local anomaly data; if the local anomaly evaluation value of the data record does not satisfy the local anomaly filtering condition, the data record is determined as non-local anomaly data.

9. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected in communication with the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the data processing method in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to implement the data processing method in any one of claims 1-8 when executed.

Images