Data anomaly detection method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of data processing, and discloses a data exception detection method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining to-be-detected target data; sampling the target data for multiple times from multiple dimensions to obtain sampling data under each dimension; performing abnormal reasoning on the multiple sampling data under each dimension to generate an abnormal reasoning result under each dimension; and performing consistency evaluation on the anomaly reasoning result under each dimension, and determining a data anomaly detection result. By implementing the technical scheme of the invention, anomaly detection can be performed on the same target data from different dimensions, the adaptability of data detection is enhanced, the precision of data anomaly detection is improved, and the accuracy of data anomaly detection is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data anomaly detection method, device, equipment, storage medium and program product. Background Art

[0002] In data processing and analysis, data anomaly detection is an important part of ensuring data quality and reliability. When facing large-scale data sets, the current data anomaly detection methods have a significant increase in computational costs as the amount of data increases, and it is difficult to accurately identify anomalies in complex data environments. Therefore, how to efficiently and accurately detect abnormal data in large-scale data has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, the present disclosure provides a data anomaly detection method, apparatus, device, storage medium and program product to solve the problem of poor abnormal data detection effect.

[0004] In a first aspect, the present disclosure provides a data anomaly detection method, including: obtaining target data to be detected; sampling the target data multiple times from multiple dimensions to obtain sampled data in each dimension; performing anomaly reasoning on the multiple sampled data in each dimension to generate anomaly reasoning results in each dimension; performing consistency evaluation on the anomaly reasoning results in each dimension to determine the data anomaly detection result.

[0005] In a second aspect, the present disclosure provides a data anomaly detection device, including: an acquisition module, used to acquire target data to be detected; a sampling module, used to sample the target data multiple times from multiple dimensions to obtain sampled data in each dimension; an abnormal reasoning module, used to perform abnormal reasoning on the multiple sampled data in each dimension to generate abnormal reasoning results in each dimension; an evaluation module, used to perform consistency evaluation on the abnormal reasoning results in each dimension to determine the data anomaly detection result.

[0006] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data anomaly detection method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0007] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data anomaly detection method of the first aspect or any corresponding embodiment thereof.

[0008] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to enable a computer to execute the data anomaly detection method of the first aspect or any corresponding embodiment thereof.

[0009] The data anomaly detection method, device, equipment, storage medium and program product provided by the present disclosure can perform anomaly detection on the same target data from different dimensions by sampling the target data multiple times from multiple dimensions and using the multiple sampled data in each dimension for anomaly reasoning, thereby enhancing the adaptability of data detection without setting different detection methods for different data sets. By evaluating the consistency of the anomaly reasoning results in each dimension, it is convenient to improve the detection reliability through cross-validation, thereby improving the accuracy of data anomaly detection and ensuring the accuracy of data anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 is a flow chart of a data anomaly detection method according to an embodiment of the present disclosure;

[0012] Figure 2 is a flow chart of another data anomaly detection method according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram of a prompt word template according to an embodiment of the present disclosure;

[0014] Figure 4 is a flowchart of another data anomaly detection method according to an embodiment of the present disclosure;

[0015] Figure 5 is a specific example diagram of the data anomaly detection method according to an embodiment of the present disclosure;

[0016] Figure 6 is a structural block diagram of a data anomaly detection device according to an embodiment of the present disclosure;

[0017] Figure 7 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0019] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0020] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0021] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0023] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0024] With the rapid development of big data technology, the amount of data is growing exponentially. Although the Z-score and IQR methods based on statistics, and the K-means and SVM anomaly detection methods based on machine learning perform well on small-scale data sets, they are often unable to cope with large-scale data sets, and have defects such as high computational complexity, low detection accuracy and poor adaptability.

[0025] Based on this, the technical solution disclosed in the present invention performs anomaly detection on the same data from different dimensions, which is convenient for improving detection reliability through cross-validation. By performing consistency evaluation on the anomaly reasoning results under various dimensions, the credibility of the anomaly detection results is quantified, thereby ensuring the accuracy of data anomaly detection.

[0026] According to an embodiment of the present disclosure, an embodiment of a data anomaly detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] In this embodiment, a data anomaly detection method is provided, which can be used in computer devices, such as computers, servers, etc. Figure 1 is a flow chart of a data anomaly detection method according to an embodiment of the present disclosure. Figure 1 As shown, the process includes the following steps:

[0028] Step S101, obtaining target data to be detected.

[0029] The target data to be detected is the data generated by the data platform that has not been detected for anomalies. The target data can be incremental data of the data platform, or data generated for a certain business or application, etc., which is not specifically limited here.

[0030] Specifically, the data platform is connected to each business scenario and each application program, and can collect business data generated by each business scenario and application data generated by each application program. The computer device is connected to the data platform in communication, and can obtain the current target data to be detected from the data platform periodically or in real time.

[0031] Step S102, sampling the target data from multiple dimensions to obtain sampled data in each dimension.

[0032] Dimensions are sampling dimensions of target data, which may include time trends, value distributions, and business rules. For any dimension, sample the target data once or multiple times according to a pre-set time window to obtain sampled data within the time window. If multiple samplings are performed, the time windows for each sampling can be the same or different, and there can be overlapping or covering relationships between different time windows.

[0033] Step S103, performing abnormal reasoning on the sampled data in each dimension to generate abnormal reasoning results in each dimension.

[0034] The abnormal reasoning results are used to characterize the abnormal information in the sampled data, including data abnormal point information (such as a list of data abnormal points, statistical information of data abnormal points, contextual information of data abnormal points) and the reasoning process of abnormal judgment, etc. Specifically, in each dimension, independent abnormal reasoning is performed on the corresponding sampled data, and the data abnormal point information in the sampled data and the reasoning process of abnormal judgment can be determined, and the abnormal reasoning results generated by reasoning on the sampled data in each dimension are obtained.

[0035] Step S104, performing consistency evaluation on the abnormal reasoning results in each dimension to determine the data anomaly detection result.

[0036] Extract abnormal indicators and the reasoning process for determining abnormalities from the abnormal reasoning results of each dimension, perform statistical analysis on the abnormal indicators, determine the consistency of the abnormal indicators, and perform similarity analysis on the reasoning process to analyze the consistency of abnormal judgments in different dimensions. At the same time, according to the business scenario of the target data, determine the business rules under the business scenario, and verify the rationality of each abnormal reasoning result according to the business rules to determine the verification consistency of the business rules. Then, combined with the consistency evaluation results under each dimension, measure the consistency degree under different dimensions, and generate corresponding data anomaly detection results.

[0037] It should be noted that multiple sampling can be performed. If there are multiple sampling data, multiple rounds of consistency evaluation are performed from different dimensions. The final data anomaly detection result is determined according to the results of the multiple rounds of consistency evaluation to improve the stability and reliability of the detection result.

[0038] The data anomaly detection method provided in this embodiment performs anomaly detection on the same target data from different dimensions by sampling the target data multiple times from multiple dimensions and performing anomaly inference using the multiple sampled data in each dimension. This method does not require setting different detection methods for different data sets, thereby enhancing the adaptability of data detection. By performing consistency evaluation on the anomaly inference results in each dimension, it is convenient to improve the detection reliability through cross-validation, thereby improving the accuracy of data anomaly detection and ensuring the accuracy of data anomaly detection.

[0039] In this embodiment, a data anomaly detection method is provided, which can be used in computer devices, such as computers, servers, etc. Figure 2 is a flow chart of a data anomaly detection method according to an embodiment of the present disclosure. Figure 2 As shown, the process includes the following steps:

[0040] Step S201, obtaining target data to be detected. Please refer to the relevant description of the corresponding steps in the above embodiment for details, which will not be repeated here.

[0041] Step S202: Sample the target data from multiple dimensions to obtain sampled data in each dimension. Please refer to the description of the corresponding steps in the above embodiment for details, which will not be repeated here.

[0042] Step S203, performing abnormal reasoning on the sampled data in each dimension to generate abnormal reasoning results in each dimension.

[0043] Specifically, the above step S203 includes:

[0044] Step S2031: Generate dimension prompt information corresponding to each dimension based on the data characteristics of each dimension.

[0045] Data characteristics are the characteristics of target data under different dimensions, such as data distribution, range, type (such as numerical, categorical, text), etc. Dimension prompt information is a descriptive prompt information for data characteristics, which is used to guide the abnormal reasoning process of sampled data. Specifically, for the time trend dimension, the dimension prompt information "Analyze the trend consistency and sudden changes in the time series" can be generated; for the numerical distribution dimension, the dimension prompt information "Check the statistical distribution characteristics and identify deviations that do not match the expected probability pattern" can be generated; for the business rule dimension, the dimension prompt information "Verify compliance with business rules and constraints in specific fields" can be generated.

[0046] In some optional implementations, the above step S2021 includes:

[0047] Step a1, obtaining a prompt word template.

[0048] Step a2, in response to the information editing operation on the prompt word template, obtain the dimension prompt information corresponding to each dimension.

[0049] The prompt word template is a pre-set text structure used to guide or prompt the user to input specific information. The prompt word template includes input prompts, output prompts, and task prompts. Figure 3 As shown, the input prompt is used to guide the user to edit specific input information, which can be prompt information for multiple dimensions; the output prompt is used to guide the user to edit specific output information, which can include prompt information such as the reason for determining data anomalies, anomaly scores, and anomaly suggestions; the task prompt is used to guide the user to edit the detection task.

[0050] The user edits the input prompt information and output prompt information in the prompt word template according to the data characteristics under each dimension. Accordingly, the computer device can respond to the information editing operation initiated by the user on the prompt word template, generate task prompt information, input prompt information and output prompt information at the corresponding position of the prompt word template, and obtain the dimension prompt information corresponding to each dimension.

[0051] In the above implementation, it supports setting corresponding dimensional prompt information according to different dimensions, so as to generate diversified prompt information from different dimensions and ensure the diversity and independence of abnormal reasoning results.

[0052] Step S2032: perform abnormal reasoning on the sampled data according to the prompt information of each dimension to generate an abnormal reasoning result.

[0053] The dimensional prompt information corresponding to each dimension is combined to guide the subsequent abnormal reasoning process of the sampled data, and the sampled data obtained by each sampling is evaluated for abnormality, and the abnormal reasoning results under each dimension are generated, wherein the abnormal reasoning results include the abnormal score of the sampled data, the abnormal cause, and the abnormal suggestions to be taken.

[0054] In some optional implementations, abnormal reasoning is performed on multiple sampling data in each dimension, and multiple abnormal reasoning results in each dimension are generated through a pre-trained abnormality detection model.

[0055] Among them, the training methods of the anomaly detection model include:

[0056] Step b1, constructing a training sample set for the data anomaly detection task, the training sample set includes anomaly features and anomaly labeled samples.

[0057] Step b2: fine-tune the anomaly detection model according to a preset fine-tuning strategy based on the anomaly features and anomaly labeled samples.

[0058] The anomaly detection model is used to detect anomalies in sampled data. The anomaly detection model can adopt a large language model architecture, such as the Transform model structure with a decoder-only architecture, and use the causal self-attention mechanism to ensure that the anomaly detection model can better focus on historical information. At the same time, it can ensure that the anomaly detection model can support longer context windows to process data features with longer sequences.

[0059] During the training phase of the anomaly detection model, the raw data collected from the data platform is preprocessed by data cleaning, normalization, feature selection, etc., to reduce the noise and redundant information in the raw data and ensure the quality of the target data generated after preprocessing, which is conducive to improving the subsequent model training efficiency.

[0060] Specifically, data cleaning is used to remove noise and erroneous data from the original data. In this process, not only the original data is strictly screened and verified, but also the accuracy and consistency of the data input into the anomaly detection model are ensured through multiple verification mechanisms. Standardization or normalization technology is used to accurately scale the original data to a unified standard range to eliminate the influence of different feature dimensions. Feature selection algorithms such as principal component analysis (PCA) and LASSO are used to select the most representative and discriminative data features from the original data to reduce data dimensions and redundant information, so as to improve the training efficiency and effect of the anomaly detection model.

[0061] Based on the target data obtained after preprocessing, a training sample set for fine-tuning the anomaly detection model is constructed. The training sample set contains the abnormal features required for abnormal reasoning and manually annotated anomaly labeled samples. The preset fine-tuning strategy is a pre-set model training strategy for vertical domain detection tasks. Specifically, the anomaly detection model is fine-tuned on the training sample set according to the preset fine-tuning strategy to obtain the final anomaly detection model, and it is deployed in the computer device to use the anomaly detection model to perform anomaly detection on the data in the data platform, process the abnormal data, and ensure data quality and data reliability.

[0062] Among them, the preset fine-tuning strategy can be to adaptively adjust the model architecture according to the data characteristics of the anomaly detection task; it can also be to introduce a specific loss function during the training process; it can also be to introduce a specific regularization method during the training process to optimize the anomaly detection model's ability to detect abnormal data.

[0063] In the above implementation, a training sample set is constructed using the preprocessed target data, and the training sample set is used to train an anomaly detection model based on a large language model architecture, so that the anomaly detection model has the ability to automatically extract data features, thereby achieving higher detection accuracy in large-scale data.

[0064] Step S204: Perform consistency evaluation on the abnormal reasoning results in each dimension to determine the data abnormality detection result. For details, please refer to the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0065] The data anomaly detection method provided in this embodiment combines the data characteristics of each dimension to generate corresponding dimension prompt information, so as to perform independent reasoning in each dimension according to the dimension prompt information, thereby ensuring the diversity and independence of the abnormal reasoning results, which is conducive to improving the reliability of the abnormal reasoning results.

[0066] In this embodiment, a data anomaly detection method is provided, which can be used in computer devices, such as computers, servers, etc. Figure 4is a flow chart of a data anomaly detection method according to an embodiment of the present disclosure. Figure 4 As shown, the process includes the following steps:

[0067] Step S301, obtaining target data to be detected. Please refer to the relevant description of the corresponding steps in the above embodiment for details, which will not be repeated here.

[0068] Step S302: Sample the target data from multiple dimensions to obtain sampled data in each dimension. Please refer to the description of the corresponding steps in the above embodiment for details, which will not be repeated here.

[0069] Step S303, perform abnormal reasoning on the sampled data in each dimension to generate abnormal reasoning results in each dimension. For details, please refer to the relevant description of the corresponding steps in the above embodiment, which will not be repeated here.

[0070] Step S304: perform consistency evaluation on the abnormal reasoning results in each dimension to determine the data anomaly detection result.

[0071] Specifically, the above step S304 includes:

[0072] Step S3041, obtaining abnormal indicators and reasoning texts corresponding to abnormal reasoning results in each dimension.

[0073] Abnormal indicators are statistical indicators that characterize data abnormalities, including mean, median, and standard deviation. Among them, the mean is the average value of abnormal data of abnormal inference results corresponding to multiple sampling data, which is used to reflect the overall abnormality of the target data; the median is the middle value of abnormal data of abnormal inference results corresponding to multiple sampling data, which is used to reduce the impact of extreme data values; the standard deviation is the average difference between abnormal data points and their mean, which is used to evaluate the consistency of data abnormality judgment.

[0074] The reasoning text is the text generated for the abnormal reasoning process. As described above, the anomaly detection model can perform anomaly detection on the target data, determine the abnormal judgment cause related to the target data, and obtain the reasoning text corresponding to the abnormal judgment cause.

[0075] Step S3042: Perform statistical consistency evaluation based on the abnormal indicators to generate statistical consistency evaluation results.

[0076] The statistical consistency evaluation results are used to characterize the consistency of the abnormal indicators in each dimension in the statistical dimension. The statistical consistency scores are given to each abnormal reasoning result according to the abnormal indicators, and the statistical consistency evaluation results are determined by the statistical consistency scores.

[0077] In some optional implementations, the above step S3042 includes:

[0078] Step c1, calculating the coefficient of variation corresponding to the abnormal reasoning result based on the abnormal index.

[0079] Step c2, determining a statistical consistency score based on the ratio between the coefficient of variation and a preset threshold, and using the statistical consistency score to represent the statistical consistency evaluation result.

[0080] The coefficient of variation is used to describe the relative volatility of data. When the coefficient of variation is small, it means that the fluctuation of the data outliers relative to the abnormal mean is small; when the coefficient of variation is large, it means that the fluctuation of the data outliers relative to the abnormal mean is large. Specifically, the specific calculation method of the coefficient of variation (CV) is: CV = standard deviation / mean.

[0081] A corresponding variation threshold can be set for the coefficient of variation, such as 0.2. According to the ratio between the coefficient of variation and the preset threshold, the statistical consistency score can be calculated. The specific calculation expression is as follows:

[0082] Score_stat=1-min(CV / CV_threshold, 1)

[0083] Among them, Score_stat is the statistical consistency score; CV_threshold is the preset threshold; CV is the coefficient of variation.

[0084] Based on the above calculation expression, it can be determined that the score interval of the statistical consistency score is [0,1]. The higher the statistical consistency score, the higher the degree of statistical consistency.

[0085] In the above implementation, by calculating the statistical consistency score, it is convenient to determine the consistency of abnormal reasoning results in different dimensions in statistical analysis.

[0086] Step S3043: perform reasoning consistency evaluation on the reasoning text corresponding to the abnormal reasoning result in each dimension to generate a reasoning consistency evaluation result.

[0087] The reasoning consistency evaluation results are used to evaluate the consistency of the reasoning process of each abnormal reasoning result in each dimension. The similarity of the reasoning texts corresponding to each abnormal reasoning result is calculated to determine the similarity of the reasoning texts corresponding to each abnormal reasoning result. The reasoning consistency of each abnormal reasoning result is scored according to the similarity, and the reasoning consistency evaluation results are determined through the reasoning consistency scoring.

[0088] In some optional implementations, the above step S3043 includes:

[0089] Step d1, vectorize the reasoning text in each dimension to obtain vectorized reasoning text.

[0090] Step d2, determining the text similarity between the vectorized reasoning texts.

[0091] Step d3, extracting keywords from the inference text in each dimension and determining the keyword overlap rate.

[0092] Step d4, based on the weighted results of text similarity and keyword overlap rate, determine the reasoning consistency score, and use the reasoning consistency score to represent the reasoning consistency evaluation result.

[0093] The pre-trained vectorization model is used to vectorize the inference text corresponding to each abnormal inference result, and the inference text is converted into a vector representation to obtain the vectorized inference text corresponding to each abnormal inference result.

[0094] Text similarity is used to characterize the similarity between vectorized reasoning texts. Specifically, according to the vector similarity calculation method, the cosine similarity between all vectorized reasoning texts is calculated, and the cosine similarity is used as the text similarity. Of course, the text similarity between vectorized reasoning texts can also be determined by using methods such as Euclidean distance, Manhattan distance, and Pearson correlation coefficient.

[0095] Keywords are words or phrases that represent the main content of the reasoning text. By analyzing the semantics of the reasoning text corresponding to each abnormal reasoning result, keywords that can represent the semantics of the reasoning text are extracted from each reasoning text. The repeated keywords extracted from the reasoning text corresponding to each abnormal reasoning result are counted, and the keyword overlap rate is determined according to the number of repeated keywords and the total number of keywords.

[0096] The reasoning consistency is scored according to the text similarity and the keyword overlap rate to obtain the reasoning consistency score. Specifically, the reasoning consistency score is calculated as follows:

[0097] Score_reason=w1*avg_cosine_sim+w2*keyword_overlap_ratio

[0098] Among them, w1 is the weight coefficient of text similarity, which can be set to 0.6; w2 is the weight coefficient of keyword overlap rate, which can be set to 0.4; avg_cosine_sim is the average value of multiple text similarities; keyword_overlap_ratio is the keyword overlap rate.

[0099] In the above implementation, the text similarity and keyword overlap rate between the inference texts are calculated to determine the inference consistency score, which facilitates determining the consistency of abnormal inference results of different dimensions during the inference process.

[0100] Step S3044, performing consistency evaluation of business rules on the abnormal reasoning results in each dimension to generate a rule consistency evaluation result.

[0101] The business rule consistency evaluation results are used to evaluate the consistency of each abnormal reasoning result in each dimension in terms of business compliance. For any dimension, the corresponding abnormal reasoning results are checked against the business rule constraints to determine whether each abnormal reasoning result complies with the business rule. The consistency of each abnormal reasoning result is scored according to the compliance with the business rule, and the business rule consistency evaluation results are determined through the business rule consistency scoring.

[0102] In some optional implementations, the above step S3044 includes:

[0103] Step e1, checking whether the abnormal reasoning result complies with the business rules.

[0104] Step e2: If the abnormal reasoning result complies with the business rules, the rule compliance of the abnormal reasoning result is determined according to a preset method, and the rule consistency evaluation result is represented by the rule compliance.

[0105] For the abnormal reasoning results obtained under any dimension, verify whether they comply with the predefined business rules and detect whether they violate the known business constraints. If the abnormal reasoning results can meet the business rules and their business constraints, the rule compliance of the abnormal reasoning results can be calculated based on the rule weight matrix. The specific calculation method is as follows:

[0106] Score_rule=Σ(rule_weight_i*rule_compliance_i) / Σ(rule_weight_i)

[0107] Among them, Score_rule represents the rule compliance; rule_weight_i represents the weight of the i-th business rule; rule_compliance_i represents the compliance of the i-th business rule (0 or 1).

[0108] The setting method for rule_weight_i is: if the business rule is a hard rule, set its weight to 1.0, indicating that the rule must be met; if the business rule is a soft rule, set its weight to 0.3-0.7, indicating that it is recommended to meet the rule; if the business rule is a reference rule, set its weight to 0.1-0.3, indicating that it is optional to meet the rule. It should be noted that each business rule supports online update and invalidation processing, and the weight corresponding to each business rule can be dynamically adjusted based on historical data performance.

[0109] In the above implementation, the rule compliance of the abnormal reasoning result is calculated so as to determine the consistency of the abnormal reasoning results of different dimensions in terms of business rules by using the rule compliance.

[0110] Step S3045, generating a data anomaly detection result based on the fusion result of the statistical consistency evaluation result, the reasoning consistency evaluation result and the rule consistency evaluation result.

[0111] The statistical consistency evaluation results, the reasoning consistency evaluation results, and the rule consistency evaluation results are integrated according to the corresponding weights to obtain a consistency comprehensive score, and the data anomaly detection result is determined based on the consistency comprehensive score. Specifically, the calculation method of the consistency comprehensive score is as follows:

[0112] Score = w11*statistical consistency + w21*inference consistency + w31*rule consistency

[0113] Among them, the weight coefficients w11, w21 and w31 can be dynamically adjusted according to the actual business scenario.

[0114] Step S305, obtaining the consistency degree represented by the fusion result.

[0115] The consistency degree is used to characterize the consistency of the anomaly detection results generated by different dimensions. Specifically, the consistency degree includes high consistency, moderate consistency and low consistency. If Score ≥ 0.8, it means that the anomaly detection results generated by different dimensions are highly consistent; if 0.5 ≤ Score < 0.8, it means that the anomaly detection results generated by different dimensions are moderately consistent; if Score < 0.5, it means that the anomaly detection results generated by different dimensions are lowly consistent.

[0116] Step S306: Determine an abnormal response strategy for the data abnormality detection result based on the consistency level.

[0117] Different degrees of consistency correspond to different exception response strategies. Specifically, the exception response strategy includes automatically triggering the exception response mechanism; starting the secondary sampling verification mechanism and adding the business rule verification mechanism; triggering the manual review mechanism. The corresponding exception response strategy is triggered according to the consistency of the anomaly detection results generated in different dimensions.

[0118] In some optional implementations, the above step S306 includes:

[0119] Step f1: If the consistency level meets the first level condition, an abnormal response to the data abnormality detection result is automatically triggered.

[0120] The consistency degree meets the first level condition, which means that the comprehensive consistency score meets the preset value corresponding to the first level. For scenarios where the anomaly detection results generated in different dimensions are highly consistent, the average value of the statistical consistency score representing the statistical consistency evaluation results is directly used as the final anomaly score, and the anomaly response mechanism for the data anomaly detection results is automatically triggered.

[0121] Step f2: If the consistency level meets the second level condition, a secondary anomaly verification strategy for the target data is initiated.

[0122] The consistency degree meets the second level condition, which means that the comprehensive consistency score meets the preset value corresponding to the second level. For scenarios where the anomaly detection results generated by different dimensions are moderately consistent, the length of the sampling time window is changed, the secondary sampling verification mechanism is started, and additional business rules are added for secondary verification of the business rules.

[0123] Step f3: if the consistency level meets the third level condition, the context information of the target data is collected to trigger the review process for the target data.

[0124] The consistency degree meets the third level condition, which means that the consistency comprehensive score meets the preset value corresponding to the third level. For scenarios where the anomaly detection results generated by different dimensions are low in consistency, the manual review mechanism is triggered and additional contextual information is collected to ensure the accuracy of anomaly data detection.

[0125] The data anomaly detection method provided in this embodiment combines statistical consistency, reasoning consistency and rule consistency to conduct a comprehensive evaluation of abnormal reasoning results, thereby realizing the quantification of the credibility of abnormal reasoning results. By integrating statistical consistency, reasoning consistency and rule consistency, the consistency degree represented by the fusion result is determined, and the corresponding abnormal response strategy is determined according to the consistency degree. Different processing strategies can be adaptively selected according to the consistency degree, realizing dynamic decision-making of abnormal response strategies.

[0126] As a specific application example of the present disclosure, Figure 5 As shown, the data generated in the collection business is preprocessed and then input into the anomaly detection model. The preprocessed data is sampled multiple times according to a preset time window to obtain corresponding sampled data.

[0127] The anomaly detection model is pre-set with a prompt project consisting of prompt word templates, which constructs dimensional prompt information for time trends, numerical distributions, and business rules. The sampled data is inferred to be abnormal according to the corresponding dimensional prompt information, and the abnormal inference results corresponding to multiple sampled data are obtained. The abnormal inference results of different dimensions are evaluated for consistency, and the consistency degree is determined based on the consistency evaluation results. If there is a high degree of consistency, the anomaly detection result is directly output; if there is a moderate degree of consistency, a secondary sampling verification is performed; if there is a low degree of consistency, a manual review is triggered. In this way, the final anomaly detection result is determined based on the degree of consistency.

[0128] In this embodiment, a data anomaly detection device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0129] This embodiment provides a data anomaly detection device, such as Figure 6 As shown, including:

[0130] The acquisition module 401 is used to acquire target data to be detected.

[0131] The sampling module 402 is used to sample the target data from multiple dimensions to obtain sampled data in each dimension.

[0132] The abnormal reasoning module 403 is used to perform abnormal reasoning on the sampled data in each dimension and generate abnormal reasoning results in each dimension.

[0133] The evaluation module 404 is used to perform consistency evaluation on the abnormal reasoning results in each dimension to determine the data anomaly detection results.

[0134] In some optional implementations, the abnormal reasoning module 403 includes:

[0135] The prompt generating unit is used to generate dimension prompt information corresponding to each dimension based on the data characteristics of each dimension.

[0136] The reasoning subunit is used to perform abnormal reasoning on the sampled data according to the prompt information of each dimension and generate abnormal reasoning results.

[0137] In some optional embodiments, the above device further includes:

[0138] The training module is used to train the anomaly detection model. Specifically, the sampling data in each dimension is subjected to anomaly reasoning, and the anomaly reasoning results in each dimension are generated by the pre-trained anomaly detection model. The training module includes:

[0139] The sample set construction unit is used to construct a training sample set for the data anomaly detection task. The training sample set includes anomaly features and anomaly labeled samples.

[0140] The model fine-tuning unit is used to fine-tune the anomaly detection model according to a preset fine-tuning strategy based on the anomaly features and anomaly labeled samples.

[0141] In some optional implementations, the prompt generating unit includes:

[0142] The template acquisition subunit is used to acquire the prompt word template.

[0143] The editing subunit is used to obtain dimension prompt information in response to the information editing operation on the prompt word template.

[0144] In some optional implementations, the evaluation module 404 includes:

[0145] The reasoning information acquisition unit is used to obtain abnormal indicators and reasoning texts corresponding to abnormal reasoning results in each dimension.

[0146] The statistical consistency evaluation unit is used to perform statistical consistency evaluation based on abnormal indicators and generate statistical consistency evaluation results.

[0147] The reasoning consistency evaluation unit is used to perform reasoning consistency evaluation on the reasoning texts corresponding to the abnormal reasoning results in each dimension and generate reasoning consistency evaluation results.

[0148] The rule consistency evaluation unit is used to perform consistency evaluation of business rules on abnormal reasoning results in various dimensions and generate rule consistency evaluation results.

[0149] The fusion unit is used to generate a data anomaly detection result based on the fusion result of the statistical consistency evaluation result, the reasoning consistency evaluation result and the rule consistency evaluation result.

[0150] In some optional implementations, the statistical consistency evaluation unit includes:

[0151] The coefficient of variation determination subunit is used to calculate the coefficient of variation corresponding to the abnormal reasoning result based on the abnormal indicator.

[0152] The first calculation subunit is used to determine a statistical consistency score based on a ratio between a coefficient of variation and a preset threshold value, and to characterize a statistical consistency evaluation result with the statistical consistency score.

[0153] In some optional implementations, the reasoning consistency evaluation unit includes:

[0154] The vector subunit is used to vectorize the reasoning text in each dimension to obtain the vectorized reasoning text.

[0155] The similarity determination subunit is used to determine the text similarity between the vectorized reasoning texts.

[0156] The extraction subunit is used to extract keywords from the inference text under each dimension and determine the keyword overlap rate.

[0157] The weighted calculation subunit is used to determine the reasoning consistency score based on the weighted results of text similarity and keyword overlap rate, and to represent the reasoning consistency evaluation result with the reasoning consistency score.

[0158] In some optional implementations, the rule consistency evaluation unit includes:

[0159] The rule detection subunit is used to detect whether the abnormal reasoning results comply with the business rules.

[0160] The compliance determination subunit is used to determine the rule compliance of the abnormal reasoning result in a preset manner if the abnormal reasoning result complies with the business rules, and to characterize the rule consistency evaluation result with the rule compliance.

[0161] In some optional embodiments, the above device further includes:

[0162] The consistency degree acquisition module is used to obtain the consistency degree represented by the fusion result.

[0163] The response strategy determination module is used to determine an abnormal response strategy for the data anomaly detection result based on the consistency degree.

[0164] In some optional implementations, the response strategy determination module includes:

[0165] The first response unit is used to automatically trigger an abnormal response to the data abnormality detection result if the consistency degree meets the first level condition.

[0166] The second response unit is used to start a secondary anomaly verification strategy for the target data if the consistency level meets the second level condition.

[0167] The third response unit is used to collect context information of the target data and trigger a review process for the target data if the consistency level meets the third level condition.

[0168] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0169] The data anomaly detection device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0170] The data anomaly detection device provided by the embodiment of the present disclosure samples the target data multiple times from multiple dimensions and uses the multiple sampled data in each dimension to perform anomaly inference, so that anomaly detection can be performed on the same target data from different dimensions, without setting different detection methods for different data sets, thereby enhancing the adaptability of data detection. By performing consistency evaluation on the anomaly inference results in each dimension, it is convenient to improve the detection reliability through cross-validation, thereby improving the accuracy of data anomaly detection and ensuring the accuracy of data anomaly detection.

[0171] The present disclosure also provides a computer device having the above Figure 6 The data anomaly detection device shown.

[0172] See also Figure 7 , Figure 7 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0173] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0174] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0175] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0176] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0177] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0178] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0179] The computer device also includes a communication interface, which is used for the computer device to communicate with other devices or a communication network.

[0180] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0181] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0182] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data anomaly detection method, characterized in that: The method comprises: Acquire target data to be detected; Sampling the target data from multiple dimensions to obtain sampled data in each dimension; Performing abnormal reasoning on the sampled data in each dimension to generate abnormal reasoning results in each dimension; The abnormal reasoning results under the various dimensions are evaluated for consistency to determine the data anomaly detection results.

2. The method according to claim 1, characterized in that The performing abnormal reasoning on the sampled data in each dimension to generate abnormal reasoning results in each dimension includes: Based on the data characteristics of each dimension, generate dimension prompt information corresponding to each dimension; The sampled data is subjected to abnormal reasoning according to the prompt information of each dimension to generate the abnormal reasoning result.

3. The method according to claim 2, characterized in that The generating dimension prompt information corresponding to each dimension based on the data characteristics of each dimension includes: Get the prompt word template; In response to the information editing operation on the prompt word template, dimension prompt information corresponding to each dimension is obtained.

4. The method according to claim 2, characterized in that: The abnormal reasoning is performed on the sampled data in each dimension, and the abnormal reasoning results in each dimension are generated by a pre-trained abnormal detection model; The training method of the anomaly detection model includes: Constructing a training sample set for a data anomaly detection task, wherein the training sample set includes anomaly features and anomaly labeled samples; Based on the abnormal features and the abnormal annotated samples, the abnormal detection model is fine-tuned according to a preset fine-tuning strategy.

5. The method according to claim 1, characterized in that: The performing consistency evaluation on the abnormal reasoning results under the various dimensions to determine the data anomaly detection results includes: Obtaining abnormal indicators and reasoning text corresponding to the abnormal reasoning results in each dimension; Performing statistical consistency evaluation based on the abnormal indicators to generate statistical consistency evaluation results; Performing reasoning consistency evaluation on the reasoning text under each dimension to generate a reasoning consistency evaluation result; Performing consistency evaluation of business rules on the abnormal reasoning results under each dimension to generate rule consistency evaluation results; A data anomaly detection result is generated based on a fusion result of the statistical consistency evaluation result, the reasoning consistency evaluation result, and the rule consistency evaluation result.

6. The method according to claim 5, characterized in that The performing statistical consistency assessment based on the abnormal indicator to generate a statistical consistency assessment result includes: Calculate the coefficient of variation corresponding to the abnormal reasoning result based on the abnormal indicator; Based on the ratio between the coefficient of variation and a preset threshold, a statistical consistency score is determined, and the statistical consistency score is used to characterize the statistical consistency evaluation result.

7. The method according to claim 5, characterized in that The performing reasoning consistency evaluation on the reasoning text under each dimension to generate a reasoning consistency evaluation result includes: Vectorize the reasoning text in each dimension to obtain a vectorized reasoning text; Determining text similarity between the vectorized reasoning texts; Extracting keywords of the inference text under various dimensions and determining the keyword overlap rate; Based on the weighted result of the text similarity and the keyword overlap rate, a reasoning consistency score is determined, and the reasoning consistency evaluation result is represented by the reasoning consistency score.

8. The method according to claim 5, characterized in that The performing consistency evaluation of the business rules on the abnormal reasoning result to generate a rule consistency evaluation result includes: Detecting whether the abnormal reasoning result complies with business rules; If the abnormal reasoning result complies with the business rule, the rule compliance of the abnormal reasoning result is determined in a preset manner, and the rule consistency evaluation result is characterized by the rule compliance.

9. The method according to claim 5, characterized in that Also includes: Obtaining the consistency degree represented by the fusion result; Based on the consistency level, an abnormality response strategy for the data abnormality detection result is determined.

10. The method according to claim 9, characterized in that The determining, based on the consistency level, an abnormal response strategy for the data abnormality detection result includes: If the consistency level meets the first level condition, an abnormal response to the data abnormality detection result is automatically triggered; If the consistency level meets the second level condition, a secondary anomaly verification strategy for the target data is initiated; If the consistency level meets the third level condition, the context information of the target data is collected to trigger a review process for the target data.

11. A data anomaly detection device, characterized in that: The device comprises: An acquisition module, used to acquire target data to be detected; A sampling module, used to sample the target data from multiple dimensions to obtain sampled data in each dimension; An abnormal reasoning module is used to perform abnormal reasoning on the sampled data in each dimension and generate abnormal reasoning results in each dimension; The evaluation module is used to perform consistency evaluation on the abnormal reasoning results under the various dimensions and determine the data anomaly detection results.

12. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data anomaly detection method according to any one of claims 1 to 10 by executing the computer instructions.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data anomaly detection method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the data anomaly detection method according to any one of claims 1 to 10.

Images