Data monitoring method and device and related equipment
By splitting the data categories and applying matching anomaly detection algorithm, the problem of poor data monitoring effect in the existing technology is solved, and more accurate business operation status evaluation and abnormal detection are achieved.
Patent Information
- Application Number
- CN202411217398.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-12-06
AI Technical Summary
The existing data monitoring methods cannot effectively represent the overall situation of the entire business operation process, resulting in poor data monitoring effects.
By splitting the original data set based on the data category, an abnormality detection algorithm with the highest degree of matching with each data category is applied to perform abnormality detection, and the detection results are analyzed to generate target monitoring results.
It improves the accuracy of data monitoring, can more effectively identify abnormal situations in business operations, replaces a single data category analysis method, and provides a more comprehensive business operation status assessment.
Smart Images

Figure CN119088645A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of big data, and in particular to a data monitoring method, apparatus, and related equipment. Background Art
[0002] Currently, relevant technologies will pre-select specific dimensions from multiple dimensions (also known as categories) involved in the business, and determine whether the business operation is normal by collecting and analyzing the data corresponding to the specific dimensions when the business is running.
[0003] It is found in applications that the specific dimensions selected by related technologies are usually unable to effectively represent the overall situation of the entire business operation process. For example, even if the business data corresponding to the selected specific dimensions is normal, business operations may still be abnormal.
[0004] That is to say, the data monitoring method provided by the related art has a poor data monitoring effect. Summary of the Invention
[0005] The purpose of the present disclosure is to provide a data monitoring method, apparatus and related equipment, so as to solve the technical problem of poor data monitoring effect existing in the data monitoring methods provided by related technologies.
[0006] In a first aspect, the present application provides a data monitoring method, the method comprising:
[0007] Splitting an original data set collected by a target business during a target period based on data categories to obtain a plurality of first subsets, each of which includes data corresponding to a data category in the original data set;
[0008] performing anomaly detection on each of the plurality of first subsets according to a first algorithm corresponding to each of the data categories to obtain a plurality of first detection results, wherein the plurality of first detection results correspond to the plurality of first subsets in a one-to-one manner, the first algorithm corresponding to the data category being an anomaly detection algorithm with the highest degree of match with the data category among the plurality of preset anomaly detection algorithms, and the degree of match between the anomaly detection algorithm and the data category being used to represent: a probability that an anomaly detection result obtained by performing anomaly detection on the data belonging to the data category based on the anomaly detection algorithm is a true result;
[0009] The multiple first detection results are analyzed to obtain a target monitoring result, wherein the target monitoring result is used to characterize whether there is data anomaly for the target business in the target time period.
[0010] In a second aspect, the present application further provides a data monitoring device, comprising:
[0011] a data splitting module, configured to split an original data set collected by a target business during a target period based on data categories to obtain a plurality of first subsets, each of which includes data corresponding to a data category in the original data set;
[0012] an anomaly detection module, configured to perform anomaly detection on each of the plurality of first subsets according to a first algorithm corresponding to each of the data categories, to obtain a plurality of first detection results, wherein the plurality of first detection results correspond one-to-one to the plurality of first subsets, the first algorithm corresponding to the data category being the anomaly detection algorithm with the highest degree of match with the data category among the plurality of preset anomaly detection algorithms, and the degree of match between the anomaly detection algorithm and the data category being used to represent: a probability that an anomaly detection result obtained by performing anomaly detection on the data belonging to the data category based on the anomaly detection algorithm is a true result;
[0013] The calculation module is used to analyze the multiple first detection results to obtain a target monitoring result, wherein the target monitoring result is used to characterize whether there is data anomaly for the target business in the target time period.
[0014] In a third aspect, the present application provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described in the first aspect.
[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
[0016] In a fifth aspect, the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect.
[0017] In the present application, the original data set is split based on data categories to obtain multiple first subsets, and anomaly detection is performed on the multiple first subsets according to the anomaly monitoring algorithm corresponding to each data category to obtain multiple first detection results. Finally, the target monitoring result is obtained by analyzing the multiple first detection results. The multi-data category analysis method is used to replace the single data category analysis method adopted by the related technology, so that more accurate monitoring results can be obtained. Among them, compared with the method of setting the same anomaly detection algorithm for multiple first subsets, each first subset is set to use the anomaly detection algorithm with the highest degree of matching with its corresponding data category for anomaly detection, so as to adapt to the data characteristics of the first subsets corresponding to different data categories, so as to obtain more accurate first detection results, so that the final target monitoring result is also more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 This is a flow chart of a data monitoring method provided by an embodiment of the present disclosure;
[0019] Figure 2 This is a schematic diagram of a process of dimension splitting and data preprocessing provided by an embodiment of the present disclosure;
[0020] Figure 3 This is a schematic diagram of a process for performing anomaly detection on various dimensions provided by an embodiment of the present disclosure;
[0021] Figure 4 This is a flow chart of a comprehensive evaluation of anomaly detection results in various dimensions provided by an embodiment of the present disclosure;
[0022] Figure 5 is a structural diagram of a data monitoring device provided by an embodiment of the present disclosure;
[0023] Figure 6 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0024] The present disclosure provides a data monitoring method, such as Figure 1 As shown, the data monitoring method includes:
[0025] Step 101: Split an original data set collected by a target business during a target period based on data categories to obtain multiple first subsets.
[0026] Each of the first subsets includes data corresponding to a data category in the original data set.
[0027] The above-mentioned target business can be understood as the business corresponding to any data-intensive application, such as: the production management business corresponding to the industrial production system, the traffic dispatching business corresponding to the intelligent transportation system, the risk control business corresponding to the financial system, etc.
[0028] The above-mentioned target period can be understood as any period before the current moment. For example, when the current moment is 1:00 on C day, month B, year A, the target period can indicate 0:00 on 2nd day, month B, year A - 1:00 on 2nd day, month B, year A, or 0:00 on 1st day, month B, year A - 24:00 on 1st day, month B, year A.
[0029] Among them, splitting the original data set collected by the target business in the target time period based on data category to obtain multiple first subsets can also be understood as: splitting the original data set collected by the target business in the target time period based on data dimension to obtain multiple first subsets, wherein each of the first subsets includes data corresponding to a data dimension in the original data set.
[0030] Among them, the multiple data categories / data dimensions corresponding to the multiple first subsets can be understood as at least part of the data categories / data dimensions of all data categories / data dimensions involved in the target business. For example: the number of all data categories / data dimensions involved in the target business is 100, then the number of multiple data categories / data dimensions corresponding to the multiple first subsets is greater than 1 and less than or equal to 100.
[0031] For example, when the target business is a production management business corresponding to an industrial production system, the multiple first subsets obtained by splitting the original data set may include: a first subset indicating the manufacturer of the device, a first subset indicating the installation location of the device, a first subset indicating the process corresponding to the device, a first subset indicating the volume of the device, and a first subset indicating the material used for the device;
[0032] When the target service is a traffic dispatch service corresponding to an intelligent transportation system, the multiple first subsets obtained by splitting the original data set may include: a first subset indicating a road number, a first subset indicating a road congestion level, a first subset indicating a road length, a first subset indicating a road width, a first subset indicating a road signal light switching rule, and a first subset indicating a road location;
[0033] When the target business is a risk control business corresponding to a financial system, the multiple first subsets obtained by splitting the original data set may include: a first subset indicating a transaction number, a first subset indicating a number of an exchange-related account, a first subset indicating a transaction amount, a first subset indicating a transaction frequency, and a first subset indicating a transaction type;
[0034] Among them, the above-mentioned equipment manufacturer, equipment size, road congestion level, road traffic light switching rules, transaction amount, transaction type, etc. can be understood as the aforementioned data categories / data dimensions.
[0035] Step 102: Perform anomaly detection on each of the plurality of first subsets according to a first algorithm corresponding to each of the data categories to obtain a plurality of first detection results.
[0036] Among them, the multiple first detection results and the multiple first subsets correspond one-to-one, the first algorithm corresponding to the data category is the anomaly detection algorithm with the highest degree of matching with the data category among the preset multiple anomaly detection algorithms, and the degree of matching between the anomaly detection algorithm and the data category is used to represent: based on the anomaly detection algorithm, the probability that the anomaly detection result obtained by performing anomaly detection on the data belonging to the data category is a true result.
[0037] Specifically, the higher the degree of matching between the anomaly detection algorithm and the data category, the higher the probability that the anomaly detection result obtained by performing anomaly detection on the data belonging to the data category based on the anomaly detection algorithm is a true result.
[0038] In some embodiments, data preprocessing can be performed on multiple first subsets separately, such as missing value filling, numerical normalization and other preprocessing operations, and then anomaly detection can be performed on the multiple first subsets that have undergone data preprocessing according to the first algorithm corresponding to each of the data categories to obtain more accurate multiple first detection results.
[0039] Exemplarily, the multiple anomaly detection algorithms can be Isolation Forest (iForest), One-Class Support Vector Machine (One-Class SVM), K-means Clustering (K-means), Gaussian Mixture Models (GMM), Outlier Factor (Outlier Factor), k-Nearest Neighbors (k-NN), Cluster-based Anomaly Detection (Cluster-based Anomaly Detection), Local Outlier Factor (LOF), Density-Based Clustering of Applications with Noise (DBSCAN), using autoencoders and other algorithms.
[0040] In one example, a corresponding test data set can be configured for each data category, and the multiple anomaly detection algorithms can be used to perform anomaly detection on the test data sets respectively to obtain multiple detection results corresponding one-to-one to the multiple anomaly detection algorithms. The degree of matching between each anomaly detection algorithm and the data category is determined by calculating the difference between each detection result and the true value result of the test data set, wherein the smaller the difference between the corresponding detection result and the true value result of the test data set, the higher the degree of matching between the anomaly detection algorithm and the data category.
[0041] In another example, a corresponding detection data set can be pre-configured for each data category, and anomaly detection results of the detection data set pre-configured for each data category by different anomaly detection algorithms are collected. The difference between the anomaly detection result and the true value result of the corresponding detection data set is compared to form training data (consisting of three parts: the difference, the category identifier of the data category corresponding to the detection data set used, and the algorithm identifier of the anomaly detection algorithm used). The neural network model is iteratively trained based on the training data, and then the category identifier of the data category and the algorithm identifier of the anomaly detection algorithm are combined in pairs to form a model input. The model input is processed using the trained neural network model to obtain a model output result - used to indicate the degree of matching between the data category of the input model and the anomaly detection algorithm of the input model, and based on this, the anomaly detection algorithm with the highest degree of matching with the data category can be determined.
[0042] Step 103: Analyze the multiple first detection results to obtain target monitoring results.
[0043] The target monitoring result is used to indicate whether there is data anomaly in the target business during the target period.
[0044] Exemplarily, data analysis may be performed on the plurality of first detection results based on a pre-trained neural network model, and the target monitoring result may be generated based on the data analysis result output by the neural network model;
[0045] It is also possible to normalize multiple first detection results separately, and count the number of first detection results exceeding the threshold among the normalized multiple first detection results, and generate the aforementioned target monitoring result accordingly. Among them, if the number of first detection results exceeding the threshold among the normalized multiple first detection results is greater than or equal to the preset number threshold, a target monitoring result is generated indicating that the target business has data anomalies in the target time period; if the number of first detection results exceeding the threshold is less than the preset number threshold, a target monitoring result is generated indicating that the target business has no data anomalies in the target time period.
[0046] It should be noted that, in the application, other analysis methods may be used to process the multiple first detection results to obtain the target monitoring result.
[0047] For example, if the target business is set as a membership business, then the various data dimensions in the original data set corresponding to the target business may include: paid member retention rate, new member registration rate, member active period, member geographical distribution, etc.;
[0048] By analyzing the paid member retention rate, new member registration rate, member active period, member geographical distribution, etc., we can know the operating status of the membership business. If the retention rate / new member registration rate increases or remains unchanged, and the member active period remains unchanged or extends, it means that the membership business is operating well. Otherwise, it means that there is an abnormality in the operation of the membership business (such as launching functions or content that conflict with the needs of members).
[0049] In one embodiment, analyzing the plurality of first detection results to obtain a target monitoring result includes:
[0050] A weighted calculation is performed on the multiple first detection results to obtain a target monitoring result.
[0051] In this embodiment, based on the weighted calculation method, the analysis of the multiple first detection results can be completed conveniently, so as to accelerate the generation efficiency of the target monitoring results while ensuring the accuracy of the target monitoring results.
[0052] When the weighted calculation is performed on the multiple first detection results, the weight data corresponding to different first detection results may be the same or different.
[0053] For example, the degree of correlation between each data category and the target indicator that the target business focuses on during the target period (for example: the defective rate / production rate of the production management business, the accident rate of the traffic scheduling business, and the bad debt risk of the risk control business) can be determined first, and then the multiple correlation degrees corresponding to the multiple data categories can be converted into multiple weight data through normalization. The multiple weight data correspond to the multiple data categories one-to-one, and the calculation of the degree of correlation between each data category and the target indicator that the target business focuses on during the target period can be completed by an association rule mining algorithm.
[0054] In some embodiments, when the target monitoring result indicates that the target business has data anomalies in the target time period, the target monitoring result may further indicate the degree of data anomalies of the target business in the target time period (as reflected by the size of the result value of weighted calculation of the multiple first detection results).
[0055] In the present application, the original data set is split based on data categories to obtain multiple first subsets, and anomaly detection is performed on the multiple first subsets according to the anomaly monitoring algorithm corresponding to each data category to obtain multiple first detection results. Finally, the target monitoring result is obtained by analyzing the multiple first detection results. The multi-data category analysis method is used to replace the single data category analysis method adopted by the related technology, so that more accurate monitoring results can be obtained. Among them, compared with the method of setting the same anomaly detection algorithm for multiple first subsets, each first subset is set to use the anomaly detection algorithm with the highest degree of matching with its corresponding data category for anomaly detection, so as to adapt to the data characteristics of the first subsets corresponding to different data categories, so as to obtain more accurate first detection results, so that the final target monitoring result is also more accurate.
[0056] In one embodiment, after splitting the original data set collected by the target business during the target period based on data categories to obtain multiple first subsets, the method further includes:
[0057] determining at least one second subset, each second subset including at least two of the first subsets;
[0058] performing anomaly detection on the at least one second subset according to the second algorithm corresponding to each second subset, to obtain at least one second detection result, wherein the at least one second detection result corresponds to the at least one second subset in a one-to-one manner, and the second algorithm corresponding to the second subset is the anomaly detection algorithm with the highest degree of match with the second subset among the multiple anomaly detection algorithms;
[0059] After performing anomaly detection on each of the plurality of first subsets according to the first algorithm corresponding to each of the data categories to obtain a plurality of first detection results, the method further includes:
[0060] The multiple first detection results and the at least one second detection result are analyzed to obtain a target monitoring result.
[0061] Among them, the at least one second subset can be a data set formed by permuting and combining the multiple first subsets, or it can be a data set formed based on preset rules, wherein the preset rules can be understood as rules used to represent the association relationship between different data categories. The preset rules can be manually set by the user based on the business experience corresponding to the target business, or they can be obtained by causal mining different data categories associated with the target business through a causal mining algorithm.
[0062] Each element in the second subset is composed of at least two first subset elements included in the second subset and associated with each other. Different first subset elements taken from the same data in the original data set are associated first subset elements.
[0063] For example, if a piece of data in the original data set is set to: 010 (transaction number) - 100 (transaction amount) - 001 (transaction type), where the element of the first subset corresponding to the transaction amount in the data is "100", and the element of the first subset corresponding to the transaction type in the data is "001", if the second subset includes the first subset corresponding to the transaction amount and the first subset corresponding to the transaction type, then the element corresponding to the second subset in the data is "100-001".
[0064] The process of selecting the anomaly detection algorithm with the highest degree of matching with the second subset from multiple anomaly detection algorithms can be found in the above description, and will not be described again to avoid repetition.
[0065] In this embodiment, in addition to performing anomaly detection in a single dimension, different dimensions will be combined and then anomaly detection will be performed to identify anomalies that cannot be discovered in a single dimension. That is, the anomaly detection results of a single dimension and the anomaly detection results of the dimension combination are combined to form the final target monitoring result, which can make the determined target monitoring result more accurate.
[0066] For example, for production management business, the hardware of the production equipment is provided by manufacturer C, and the software of the production equipment is provided by manufacturer D. An evaluation of the hardware provided by manufacturer C from the hardware dimension shows no abnormalities. An evaluation of the software provided by manufacturer D from the software dimension also shows no abnormalities. However, there may be incompatibility issues between the hardware provided by manufacturer C and the software provided by manufacturer D in the application. At this time, the above problems cannot be identified through only single-dimensional anomaly detection, but through a comprehensive evaluation of software and hardware (that is, combining the software dimension and the hardware dimension), the above compatibility issues can be effectively identified.
[0067] Alternatively, for traffic dispatch services, an abnormality detection is performed on the brightness of traffic lights, which shows no abnormality. An abnormality detection is performed on the window glass of a specific vehicle model, which also shows no abnormality. However, in reality, the window glass of a specific vehicle model may amplify the brightness of the traffic light, thereby seriously interfering with the driver's driving. Alternatively, the window glass of a specific vehicle model may weaken the light display effect of the traffic light at a specific light brightness, thereby increasing the probability of the driver missing the traffic light. At this time, the above problems cannot be identified through only single-dimensional abnormality detection, but the above-mentioned traffic light display problems can be effectively identified by comprehensively evaluating the light brightness and window glass.
[0068] In one embodiment, determining at least one second subset includes:
[0069] performing association analysis on a plurality of data categories corresponding to the plurality of first subsets, respectively, to determine a first degree of association between any two data categories among the plurality of data categories, wherein the first degree of association is used to represent a degree of correlation between the corresponding two data categories;
[0070] Determining at least one category combination based on a first correlation between any two data categories among the multiple data categories, each of the category combinations including at least two data categories, and a first correlation between any two different data categories in each of the category combinations being greater than or equal to a preset correlation threshold;
[0071] Based on the at least one category combination and the multiple first subsets, at least one second subset is determined, and the at least one second subset corresponds to the at least one category combination one-to-one. The second subset includes at least two first subsets corresponding to the at least two data categories included in the corresponding category combination.
[0072] In this embodiment, association analysis is performed on the multiple data categories to determine the first degree of association between any two data categories in the multiple data categories, and at least one category combination is determined based on this, and then at least one corresponding second subset is determined. The above-mentioned determination of the second subset through automated analysis can, on the one hand, avoid the problem of excessive number of category combinations caused by random permutations and combinations, reduce data processing volume, and improve processing efficiency; on the other hand, it can avoid interference caused by the mixing of redundant category combinations, and ensure the reliability of the target monitoring results obtained.
[0073] Exemplarily, association analysis of the multiple data categories respectively corresponding to the multiple first subsets may be completed by association rule mining algorithms such as Apriori algorithm, FP-growth algorithm, Eclat algorithm, and association rule tree algorithm.
[0074] Among them, the first correlation degree is used to describe the probability of a causal relationship between the corresponding two different data categories. The larger the first correlation degree, the higher the probability of a causal relationship between the corresponding two data categories. That is, in the corresponding two different data categories, if the data of one data category changes, the probability of the data of the other data category also changes is higher.
[0075] The above preset correlation threshold can be adaptively set according to actual needs, for example: 0.6, 0.7, etc.
[0076] In one embodiment, before performing weighted calculation on the multiple first detection results to obtain the target monitoring result, the method further includes:
[0077] Based on the multiple data categories corresponding to the multiple first subsets, the historical data set collected by the target business during the historical period is split to obtain multiple third subsets, each of the third subsets including data corresponding to one of the data categories in the original data set, and the historical period is earlier than the target period;
[0078] performing anomaly detection on each of the plurality of third subsets according to the first algorithm corresponding to each of the data categories to obtain a plurality of third detection results, wherein the plurality of third detection results correspond one-to-one to the plurality of third subsets;
[0079] performing regression analysis on the plurality of third detection results according to abnormal true value information of the target business in the historical period to determine weight data for each of the data categories;
[0080] The weighted calculation of the plurality of first detection results to obtain a target monitoring result includes:
[0081] According to the weight data of each data category, the multiple first detection results are weighted and calculated to obtain the target monitoring result.
[0082] In this embodiment, by splitting the historical data set, multiple third subsets corresponding to multiple data categories are obtained, and according to the first algorithm corresponding to each of the data categories, the multiple third subsets are respectively subjected to anomaly detection to obtain multiple third detection results. The third detection results are subjected to regression analysis in combination with the corresponding anomaly true value information to determine the weight data corresponding to each data category in an automated manner, which not only avoids interference from human factors, but also makes the determined weight data more accurate.
[0083] Among them, the abnormal true value information is used to indicate that there is data anomaly in the target business in the historical period, or to indicate that there is no data anomaly in the target business in the historical period. In application, the abnormal true value information can be formed by collecting the actual business operation information of the target business in the historical period.
[0084] For example, if a historical data set is set to describe information such as the defective rate and production rate of different parts of a vehicle, and the anomaly detection target in this scenario is the defective rate of finished vehicles, then after the parts are produced and assembled into finished vehicles, the defective rate of the finished vehicles in the historical period can be obtained by performing various tests on the finished vehicles, that is, the anomaly true value information.
[0085] For example, the above regression analysis operation can be completed by regression methods such as linear regression, multivariate linear regression, logistic regression, ridge regression, Lasso regression, polynomial regression, exponential regression, and random forest regression.
[0086] In one embodiment, before splitting the original data set collected by the target business during the target period based on data categories to obtain multiple first subsets, the method further includes:
[0087] performing association analysis on the business objective of the target business and multiple candidate categories corresponding to the target business, respectively, to obtain multiple second association degrees, wherein the multiple second association degrees correspond to the multiple candidate categories one-to-one, and the second association degrees are used to represent the degree of relevance between the corresponding candidate categories and the business objective;
[0088] A plurality of data categories corresponding to the plurality of first subsets are determined from the plurality of candidate categories, wherein the data categories are the candidate categories whose corresponding second association degrees are greater than or equal to a preset correlation threshold.
[0089] In this embodiment, association analysis is performed on the business objectives of the target business and multiple candidate categories corresponding to the target business to obtain multiple second association degrees, and based on this, multiple data categories corresponding to the multiple first subsets are determined in the multiple candidate categories by threshold comparison. The multiple data categories are determined in an automated manner, which not only avoids interference from human factors but also makes the determined data categories more accurate.
[0090] The greater the second degree of association, the higher the probability that there is a causal relationship between the corresponding candidate category and the business goal.
[0091] For easier understanding, the following examples are provided:
[0092] First, if Figure 2 As shown, the original data set is read, the dimension of the original data set is split (that is, the data of each dimension is extracted), and the data of each dimension is processed independently through data preprocessing. The data of each dimension after data preprocessing will be stored in an independent data storage; among them, the measure of splitting the dimension can ensure the independence of the data of each dimension, so as to support the subsequent anomaly detection of the data of each dimension one by one, and ensure that more accurate anomaly detection results can be obtained in the end, and the data preprocessing measures can ensure the cleanliness of the data of the split dimensions, and reduce the interference of abnormal data on the final anomaly detection results.
[0093] The step of dimension splitting is specifically as follows: extracting data of each dimension from the original data set. The extracted data of each dimension can be understood as the aforementioned multiple first subsets, and one first subset corresponds to one dimension / category.
[0094] The steps of independently processing the data of each dimension by means of data preprocessing are specifically: performing data preprocessing operations such as missing value filling and normalization on the data of each dimension.
[0095] After dimension splitting and data preprocessing of the original data set, we can Figure 3 As shown, the pre-processed data of each dimension is read from the aforementioned independent data storage, and a corresponding anomaly detection algorithm is selected for the data of each dimension among multiple preset anomaly detection algorithms (such as Isolation Forest (iForest), One-Class Support Vector Machine (One-Class SVM)), etc. (which can be understood as selecting the corresponding first algorithm for each of the data categories), and anomaly detection is performed on the data of each dimension based on the anomaly detection algorithm selected for that dimension, and the anomaly detection result of each dimension (which can be understood as the aforementioned first detection result) is recorded and stored.
[0096] Finally, if Figure 4 As shown, by integrating the anomaly detection results of each dimension (which can be understood as analyzing the multiple first detection results), the abnormal data of the overall data is comprehensively evaluated (which can be understood as obtaining the target monitoring results), that is, whether there are anomalies in the business process represented by the original data set.
[0097] The specific steps for integrating the anomaly detection results of each dimension to comprehensively evaluate the anomaly data of the overall data are as follows:
[0098] First, collect the anomaly detection information (including the aforementioned anomaly true value information and historical data sets) of each dimension in the past (which can be understood as the aforementioned historical period), and extract the anomaly score reflected by the anomaly detection information (which can be achieved by standardizing the anomaly true value information, that is, converting the anomaly true value information into anomaly scores), the correlation between different dimensions (which can be understood as the aforementioned first correlation degree) and other key feature information.
[0099] Then, a machine learning algorithm (such as linear regression, random forest regression or other suitable regression algorithms) is used to perform regression analysis on the collected key feature information (which can be understood as the aforementioned multiple third detection results) to automatically generate weight data for each dimension (that is, the weight data for each of the aforementioned data categories).
[0100] Among them, the method of automatically generating weight data for each dimension can avoid interference from human factors, ensure the accuracy of the weight data for each dimension obtained, and thus make the final anomaly detection results more accurate and reliable.
[0101] And use an association rule mining algorithm (for example, an Apriori algorithm or an FP-Growth algorithm, etc.) to analyze the association relationship between different dimensions, and generate association rules (which can be understood as the aforementioned at least one category combination) based on different dimensions with strong association (the first correlation degree between different dimensions is greater than or equal to a preset correlation threshold). For example: the aforementioned multiple dimensions are arranged and combined to obtain multiple dimension groups, each dimension group is composed of two different dimensions from the aforementioned multiple dimensions, and the dimensions included in different dimension groups are at least partially different. The multiple dimension groups are respectively associated with each other to obtain the dimension correlation value (which can be understood as the aforementioned first correlation degree) between the two different dimensions included in each dimension group in the multiple dimension groups. Then, based on the aforementioned preset correlation threshold, the dimension correlation values between the two different dimensions included in each dimension group in the multiple dimension groups are clustered to obtain at least one target dimension combination (which can be understood as the aforementioned at least one category combination), wherein the target dimension combination is composed of at least two different dimensions, and the dimension correlation value between any two different dimensions in the target dimension combination is greater than or equal to the preset correlation threshold.
[0102] On the one hand, the automatic generation of the above-mentioned association rules can avoid the problem of too many dimension combinations caused by random arrangement and combination of the multiple dimensions, reduce the amount of data processing to be performed for anomaly detection, and improve the efficiency of generating anomaly detection results. On the other hand, it can avoid the interference caused by the mixing of redundant dimension combinations and ensure the reliability of the generated anomaly detection results.
[0103] Finally, based on the aforementioned automatically generated weight data of each dimension and the association rules, the current anomaly detection results of each dimension are comprehensively analyzed, and a final anomaly detection report (which can be understood as the aforementioned target monitoring result) is generated.
[0104] By independently detecting each segmented data dimension of the data set corresponding to the business, this application can conduct a more comprehensive analysis of the business process reflected in the data set, and thus discover anomalies that cannot be identified by a single dimension; and by setting an appropriate anomaly detection algorithm for each segmented data dimension, the anomaly detection accuracy of each segmented data dimension is improved; and by combining measures such as automatically setting weight data and automatically extracting association rules between different segmented data dimensions, a comprehensive evaluation of the anomaly detection results of each segmented data dimension is conducted, thereby improving the overall intelligence level and accuracy of anomaly detection in the data set.
[0105] The application of the above solution can discover more potential anomalies during business operations, while reducing the dependence of the business anomaly monitoring process on manual labor and lowering the operation and maintenance costs of implementing the anomaly monitoring solution.
[0106] See also Figure 5 , Figure 5 This is a data monitoring device provided by an embodiment of the present disclosure, such as Figure 5 As shown, the data monitoring device 500 includes:
[0107] A data splitting module 501 is configured to split an original data set collected by a target business during a target period based on data categories to obtain a plurality of first subsets, each of which includes data corresponding to a data category in the original data set;
[0108] Anomaly detection module 502 is configured to perform anomaly detection on each of the plurality of first subsets based on a first algorithm corresponding to each of the data categories, to obtain a plurality of first detection results, wherein the plurality of first detection results correspond one-to-one to the plurality of first subsets, the first algorithm corresponding to the data category being the anomaly detection algorithm with the highest degree of match with the data category among the plurality of preset anomaly detection algorithms, and the degree of match between the anomaly detection algorithm and the data category being used to represent the probability that an anomaly detection result obtained by performing anomaly detection on the data belonging to the data category based on the anomaly detection algorithm is a true result;
[0109] The calculation module 503 is configured to analyze the multiple first detection results to obtain a target monitoring result, wherein the target monitoring result is used to indicate whether there is data anomaly in the target business during the target time period.
[0110] In one embodiment, the data splitting module 501 is further configured to:
[0111] determining at least one second subset, each second subset including at least two of the first subsets;
[0112] The anomaly detection module 502 is further configured to:
[0113] performing anomaly detection on the at least one second subset according to the second algorithm corresponding to each second subset, to obtain at least one second detection result, wherein the at least one second detection result corresponds to the at least one second subset in a one-to-one manner, and the second algorithm corresponding to the second subset is the anomaly detection algorithm with the highest degree of match with the second subset among the multiple anomaly detection algorithms;
[0114] The data monitoring device 500 further includes a hybrid weighting module, which is configured to:
[0115] The multiple first detection results and the at least one second detection result are analyzed to obtain a target monitoring result.
[0116] In one embodiment, the data splitting module 501 is specifically configured to:
[0117] performing association analysis on a plurality of data categories corresponding to the plurality of first subsets, respectively, to determine a first degree of association between any two data categories among the plurality of data categories, wherein the first degree of association is used to represent a degree of correlation between the corresponding two data categories;
[0118] performing clustering based on a first degree of association between any two data categories among the plurality of data categories to determine at least one category combination, each of the category combinations including at least two of the data categories, and wherein the first degree of association between any two different data categories in each of the category combinations is greater than or equal to a preset association threshold;
[0119] Based on the at least one category combination and the multiple first subsets, at least one second subset is determined, and the at least one second subset corresponds to the at least one category combination one-to-one. The second subset is composed of at least two first subsets corresponding to at least two data categories included in the corresponding category combination.
[0120] In one embodiment, the calculation module 503 includes a single-dimensional weighting unit, and the single-dimensional weighting unit is configured to:
[0121] A weighted calculation is performed on the multiple first detection results to obtain a target monitoring result.
[0122] In one embodiment, the data monitoring device 500 further includes a weight determination module, which is specifically configured to:
[0123] Based on the multiple data categories corresponding to the multiple first subsets, the historical data set collected by the target business during the historical period is split to obtain multiple third subsets, each of the third subsets including data corresponding to one of the data categories in the original data set, and the historical period is earlier than the target period;
[0124] performing anomaly detection on each of the plurality of third subsets according to the first algorithm corresponding to each of the data categories to obtain a plurality of third detection results, wherein the plurality of third detection results correspond one-to-one to the plurality of third subsets;
[0125] performing regression analysis on the plurality of third detection results according to abnormal true value information of the target business in the historical period to determine weight data for each of the data categories;
[0126] The single-dimensional weighting unit is specifically used for:
[0127] According to the weight data of each data category, the multiple first detection results are weighted and calculated to obtain the target monitoring result.
[0128] In one embodiment, the data monitoring device 500 further includes a category determination module, which is specifically configured to:
[0129] performing association analysis on the business objective of the target business and multiple candidate categories corresponding to the target business, respectively, to obtain multiple second association degrees, wherein the multiple second association degrees correspond to the multiple candidate categories one-to-one, and the second association degrees are used to represent the degree of relevance between the corresponding candidate categories and the business objective;
[0130] A plurality of data categories corresponding to the plurality of first subsets are determined from the plurality of candidate categories, wherein the data categories are the candidate categories whose corresponding second association degrees are greater than or equal to a preset correlation threshold.
[0131] The data monitoring device 500 provided in the embodiment of the present disclosure can implement each process in the above-mentioned data monitoring method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.
[0132] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device and a readable storage medium.
[0133] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0134] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0135] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.
[0136] The computing unit 601 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the data monitoring method. For example, in some embodiments, the data monitoring method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the data monitoring method described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the data monitoring method in any other appropriate manner (eg, by means of firmware).
[0137] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0138] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0139] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0140] As used herein, the term "machine-readable medium" refers to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0142] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.
[0143] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.
[0144] The present application also provides a computer program product including computer instructions, which, when executed by a processor, implement the above Figures 1 to 3 The various processes of the method embodiment shown can achieve the same technical effect, and to avoid repetition, they will not be described here.
[0145] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.
[0146] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.
Claims
1. A data monitoring method, characterized in that: The method comprises: Splitting the original data set collected by the target business in the target time period based on the data category to obtain a plurality of first subsets, each of which includes data corresponding to a data category in the original data set; According to the first algorithm corresponding to each of the data categories, the multiple first subsets are respectively subjected to anomaly detection to obtain multiple first detection results, wherein the multiple first detection results correspond to the multiple first subsets one by one, the first algorithm corresponding to the data category is an anomaly detection algorithm with the highest degree of matching with the data category among the preset multiple anomaly detection algorithms, and the degree of matching between the anomaly detection algorithm and the data category is used to represent: based on the anomaly detection algorithm, the probability that the anomaly detection result obtained by performing anomaly detection on the data belonging to the data category is a true result; The multiple first detection results are analyzed to obtain a target monitoring result, wherein the target monitoring result is used to characterize whether there is data anomaly for the target business in the target time period.
2. The method according to claim 1, characterized in that After splitting the original data set collected by the target business in the target time period based on the data category to obtain a plurality of first subsets, the method further includes: determining at least one second subset, each of the second subsets including at least two of the first subsets; According to the second algorithm corresponding to each of the second subsets, perform anomaly detection on the at least one second subset respectively to obtain at least one second detection result, wherein the at least one second detection result corresponds to the at least one second subset in a one-to-one manner, and the second algorithm corresponding to the second subset is the anomaly detection algorithm with the highest matching degree with the second subset among the multiple anomaly detection algorithms; After performing anomaly detection on the plurality of first subsets respectively according to the first algorithm corresponding to each of the data categories to obtain a plurality of first detection results, the method further includes: The multiple first detection results and the at least one second detection result are analyzed to obtain a target monitoring result.
3. The method according to claim 2, characterized in that The determining of at least one second subset comprises: Performing association analysis on a plurality of data categories respectively corresponding to the plurality of first subsets, and determining a first degree of association between any two data categories among the plurality of data categories, wherein the first degree of association is used to characterize a degree of correlation between the corresponding two data categories; Perform clustering based on the first correlation between any two data categories among the multiple data categories to determine at least one category combination, each of the category combinations includes at least two data categories, and the first correlation between any two different data categories in each of the category combinations is greater than or equal to a preset correlation threshold; At least one second subset is determined based on the at least one category combination and the multiple first subsets, the at least one second subset corresponds to the at least one category combination one-to-one, and the second subset is composed of at least two first subsets corresponding to at least two data categories included in the corresponding category combination.
4. The method according to claim 1, characterized in that: The analyzing the plurality of first detection results to obtain a target monitoring result includes: The multiple first detection results are weightedly calculated to obtain a target monitoring result.
5. The method according to claim 1, characterized in that Before performing weighted calculation on the plurality of first detection results to obtain the target monitoring result, the method further includes: Based on the multiple data categories respectively corresponding to the multiple first subsets, the historical data set collected by the target business in the historical period is split to obtain multiple third subsets, each of the third subsets includes data corresponding to one of the data categories in the original data set, and the historical period is earlier than the target period; According to the first algorithm corresponding to each of the data categories, the plurality of third subsets are respectively subjected to anomaly detection to obtain a plurality of third detection results, wherein the plurality of third detection results correspond to the plurality of third subsets in a one-to-one manner; According to the abnormal true value information of the target business in the historical period, performing regression analysis on the plurality of third detection results to determine weight data of each of the data categories; The step of performing weighted calculation on the plurality of first detection results to obtain a target monitoring result includes: According to the weight data of each data category, the multiple first detection results are weighted and calculated to obtain the target monitoring result.
6. The method according to any one of claims 1 to 5, characterized in that Before splitting the original data set collected by the target business in the target time period based on the data category to obtain a plurality of first subsets, the method further includes: Performing association analysis on the business goal of the target business and multiple candidate categories corresponding to the target business respectively to obtain multiple second association degrees, wherein the multiple second association degrees correspond to the multiple candidate categories one by one, and the second association degrees are used to characterize the degree of correlation between the corresponding candidate categories and the business goal; A plurality of data categories corresponding to the plurality of first subsets respectively are determined from the plurality of candidate categories, wherein the data categories are the candidate categories whose corresponding second association degrees are greater than or equal to a preset correlation threshold.
7. A data monitoring device, characterized in that: The device comprises: A data splitting module, used to split the original data set collected by the target business in the target time period based on the data category to obtain multiple first subsets, each of which includes data corresponding to a data category in the original data set; an anomaly detection module, configured to perform anomaly detection on the plurality of first subsets respectively according to a first algorithm corresponding to each of the data categories, and obtain a plurality of first detection results, wherein the plurality of first detection results correspond to the plurality of first subsets one-to-one, the first algorithm corresponding to the data category is an anomaly detection algorithm with the highest degree of matching with the data category among the plurality of preset anomaly detection algorithms, and the degree of matching between the anomaly detection algorithm and the data category is used to represent: a probability that an anomaly detection result obtained by performing anomaly detection on the data belonging to the data category based on the anomaly detection algorithm is a true result; The calculation module is used to analyze the multiple first detection results to obtain a target monitoring result, wherein the target monitoring result is used to characterize whether there is data anomaly for the target business in the target time period.
8. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 6 when executed by the processor.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.
