A data auditing method, system, device, and media based on a real-time data warehouse layered architecture.

Abstract

This application provides a data auditing method, system, device, and medium based on a real-time data warehouse layered architecture, relating to the field of data processing technology. In complex scenarios involving multiple data layers, multiple stream triggers, and parallel computing, the method constructs an anomaly detection framework and anomaly analysis grid. This framework can quickly locate the root cause level and node of abnormal data and perform pre-level and intelligent attribution analysis of abnormal events. It solves the technical shortcomings of stream computing, such as auditing delays, inability to quickly and effectively locate the root cause of problems, and high error rates from manual intervention. This achieves rapid and accurate anomaly data auditing, effectively ensuring the data consistency of important real-time data assets.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data auditing method, system, device and medium based on a real-time data warehouse hierarchical architecture. Background Technology

[0002] Data auditing plays a central role in ensuring data quality within a data governance system; however, existing data auditing technologies have the following shortcomings:

[0003] 1) Severe delay: Traditional batch audits have a delay of at least hours, making it impossible to detect and quickly locate abnormal issues in a timely manner, resulting in a significant business risk exposure;

[0004] 2) Limitations of single-point detection: Traditional auditing methods are often based on single-level data monitoring logic, which cannot form an effective detection mechanism for multi-factor anomalies. After an anomaly is discovered, it still takes time to locate the specific anomaly level and node, which greatly prolongs the event processing time and can easily lead to the spread of abnormal data and cause greater impact.

[0005] 3) Cost of manual classification and grading: Traditional auditing methods rely heavily on manual identification of anomalies and handling of problems. They cannot pre-classify and grade audit events based on business scenarios, causes of problems and handling methods. Manual identification is costly and carries the risk of errors and omissions.

[0006] Therefore, the aforementioned technical problems urgently need to be solved. Summary of the Invention

[0007] In view of the above problems, this application is made to provide a data auditing method, system, device, and medium based on a real-time data warehouse layered architecture that overcomes or at least partially solves the above problems. The technical solution is as follows:

[0008] Firstly, a data auditing method based on a real-time data warehouse layered architecture is provided, the method comprising:

[0009] Construct one or more business scenarios for real-time data warehouse stream computing and launch stream computing tasks at each level of the real-time data warehouse;

[0010] By utilizing data stream bypass computation, offline data result traceability, and real-time monitoring of task execution status, a multi-dimensional anomaly detection framework covering the entire data chain of the real-time data warehouse is constructed around various business scenarios. The anomaly detection framework includes an anomaly detection module and multiple detection metrics.

[0011] Set the lifecycle of the anomaly detection module, execute the anomaly detection module, and judge anomaly events in real time based on multiple detection indicators. If an anomaly event exists, execute the next step of anomaly analysis grid tracing logic.

[0012] The anomaly analysis grid tracing logic specifically involves receiving an anomaly event from the anomaly detection module, marking the current node where the anomaly event occurred, and then analyzing the impact of the upstream nodes of the current node on the anomaly event of the current node based on the reverse propagation of the pre-built anomaly analysis grid, until the anomaly source node is traced back.

[0013] Based on the source node and event of the anomaly, anomaly cases are matched in a pre-defined knowledge base to obtain the matched anomaly cases.

[0014] Anomalies are pre-classified based on multiple factors, including business scenario, task operation and maintenance level, matched anomaly cases, and impact, and anomaly handling mechanisms are matched accordingly.

[0015] The system calls a pre-defined large model, inputs the matched abnormal cases, the pre-defined results of abnormal events, and the matched abnormal handling mechanisms into the large model, and outputs intelligent attribution analysis and alarm information.

[0016] In one possible implementation, multiple detection metrics include data quality metrics, data consistency metrics, and operational monitoring metrics;

[0017] Data quality metrics are used to verify the quality of data at each level of the real-time data warehouse based on data quality verification rules. These levels of the real-time data warehouse include the Operational Data Storage (ODS) layer, the Data Warehouse Details (DWD) layer, the Data Warehouse Service (DWS) layer, and the Data Application (APP) layer.

[0018] Data consistency metrics are used to demonstrate the consistency of data transmission or computation between data levels. Data consistency metrics include the number of records, hash values ​​of key fields, global MD5 hash, predictive metrics, and distribution volatility. Specifically, data consistency is detected through the following audit indicator group:

[0019] The source data real-time audit indicator group specifically monitors the consistency of data collected from the upstream data warehouse in real time.

[0020] The real-time data audit indicator group at each level specifically monitors the consistency of real-time data when it is distributed from each level of the real-time data warehouse.

[0021] The target-side real-time audit indicator group specifically monitors the data consistency when data from the real-time data warehouse is distributed to other systems.

[0022] The final result offline audit indicator group specifically adopts the offline data link audit method, which takes the full calculation results of the offline data warehouse in two batches and the incremental calculation results of each data level in the real-time data warehouse for data verification. The full calculation results of the offline data warehouse and the range of the two batches of changed data are used as the comparison benchmark to calculate the coverage and accuracy of the real-time incremental data and verify the final consistency of the data in the real-time calculation task.

[0023] Operational monitoring metrics are used to describe the execution status of various tasks in real-time computing scenarios. Specifically, operational monitoring is achieved through the following monitoring metric groups:

[0024] The real-time computing task running status monitoring index group specifically detects the running status and resource status of real-time computing tasks, including indicators such as real-time task status, machine resource usage, message queue consumption latency, back pressure of each task operator, and checkpoint integrity verification.

[0025] The dimension scheduling task operation status monitoring indicator group specifically detects the operation status of data exchange, offline computing tasks and corresponding scheduling, including indicators such as scheduling workflow and the operation status of each task, scheduling queue resource usage, and task version verification.

[0026] In one possible implementation, anomaly event judgment is performed in real time based on multiple detection indicators, including:

[0027] After obtaining the data quality rule verification results at each level, the credibility of data quality anomaly events and the impact assessment are performed through an independent data quality indicator aggregation node.

[0028] After testing various data consistency indicators, a separate data consistency indicator aggregation node is used to determine the credibility of data consistency anomalies and assess their impact.

[0029] After the operation monitoring indicators are detected, the impact assessment of abnormal events in task monitoring is performed through an operation monitoring indicator detection summary node.

[0030] The aggregated data of the above data quality indicators, data consistency indicators, and operation monitoring indicators are submitted to a unified anomaly detection node for further aggregation. All relevant indicators are combined to determine the credibility of anomalies, the number of anomalies, and the impact of anomalies, and alarm information for anomalies is generated.

[0031] In one possible implementation, the anomaly analysis grid is constructed through the following steps:

[0032] The grid's vertical axis represents the hierarchical nodes of the real-time data warehouse. The set of hierarchical nodes is defined as follows:

[0033]

[0034] in For hierarchical nodes, x represents the real-time data warehouse data level, i represents the logical node number, and k is a positive integer;

[0035] The horizontal axis consists of parallel computing nodes that operate on the same operator. The set of parallel computing nodes is defined as follows:

[0036]

[0037] in For parallel computing nodes, j represents the parallel computing node number;

[0038] Define the edge set as:

[0039] in Hierarchical nodes with hierarchical nodes The edges between them represent data flow dependencies;

[0040] A dynamic weight matrix is ​​constructed based on the dependencies between nodes in different levels:

[0041]

[0042] in It is a dynamic weight matrix. for Time-level nodes For hierarchical nodes Influencing factors.

[0043] In one possible implementation, ideally, the influence factor in the grid... The value is 1, and the base value of each node is based on... The presence of anomalies at any given time is determined by a {0,1} tuple; however, in actual production environments, factors such as data skew, data latency, and single points of failure necessitate the introduction of adjustment terms to dynamically adjust the influencing factors and node baseline values; the steps for constructing the anomaly analysis grid also include:

[0044] First, consider data skewness, i.e., the proportion of data traffic from upstream nodes, and introduce an impact factor correction coefficient. :

[0045]

[0046] in This represents the data flow from node i at level x to node j at level y; This represents the data flow from node i at level x to node m at level y;

[0047] Considering the data latency caused by backpressure in stream computing, which in turn leads to delays in the propagation of abnormal events, the node base value is redefined. :

[0048]

[0049] in, This represents the tuple of the ideal node cardinality {0,1}. This indicates that the slope can be adjusted;

[0050] Introducing node basis value correction coefficients :

[0051]

[0052] in Related to the node's abnormal event itself; Variables representing data quality anomalies; Variables representing data inconsistency anomalies; Represents abnormal variables in runtime monitoring;

[0053] Therefore, the final calculated node anomaly factor is:

[0054]

[0055] in This refers to node anomaly factors.

[0056] In one possible implementation, upon receiving an anomaly event from the anomaly detection module, the current node where the anomaly event occurred is marked; based on the reverse propagation of a pre-built anomaly analysis grid, the influence of the current node's upstream nodes on the current node's anomaly event is analyzed until the anomaly source node is traced back, including:

[0057] Reverse propagation triggering specifically involves receiving an abnormal event from the anomaly detection module, marking the current node where the abnormal event occurred, loading the dynamic weight matrix of the upstream nodes of the current node, and calculating the anomaly factor of the current node. ;

[0058] Backward propagation specifically involves using a depth-first search algorithm to calculate the anomaly factor of each node on the branch along the dependent edges, and marking the node whose anomaly factor exceeds a first preset threshold or whose difference between the upstream anomaly factor and 0 is less than a second preset threshold as the source node.

[0059] The source node merging and multi-source judgment are performed after the depth-first search is completed. If the difference in contribution of multiple nodes is less than the third preset threshold, they are marked as concurrency anomalies and the set of anomaly source nodes is output.

[0060] The source tracing evolution is specifically a process of continuous reverse propagation and source tracing over time, updating the abnormal factors and influence coefficients of each node in real time, and correcting the abnormal source node until it is stable.

[0061] In one possible implementation, based on the anomaly source node and the anomaly event, anomaly case matching is performed in a pre-defined knowledge base to obtain matched anomaly cases, including:

[0062] Based on the anomaly source node and the anomaly event, the data quality indicators, data consistency indicators, and operational monitoring indicators of the anomaly event are vectorized into features. Anomaly case matching is then performed in a pre-defined knowledge base to obtain matched anomaly cases. The vectorization method is based on event type classification, defining... This is a vector of data quality anomaly events. For the first s Each data quality anomaly event type, then for: ;

[0063] Similarly, definition This is a consistency anomaly event vector. For running monitoring of abnormal event vectors;

[0064] ;

[0065] ;

[0066] in, For the first s A consistency anomaly event type; For the first s Each type of abnormal event in the operation monitoring.

[0067] Secondly, a data auditing system based on a real-time data warehouse hierarchical architecture is provided, the system comprising:

[0068] The startup unit is used to build one or more business scenarios for real-time data warehouse streaming computing and to start streaming computing tasks at each level of the real-time data warehouse.

[0069] The building unit is used to construct an anomaly detection framework that covers the entire data chain of the real-time data warehouse in multiple dimensions, around various business scenarios, through data stream bypass computing, offline data result traceability, and real-time monitoring of task execution status; the anomaly detection framework includes an anomaly detection module and multiple detection indicators;

[0070] The execution and judgment unit is used to set the lifecycle of the anomaly detection module, execute the anomaly detection module, and judge anomaly events in real time based on multiple detection indicators. If an anomaly event exists, the next step of the anomaly analysis grid tracing logic is executed.

[0071] The anomaly analysis grid tracing unit is used to mark the current node where the anomaly event occurred after receiving an anomaly event from the anomaly detection module; it analyzes the impact of the upstream nodes of the current node on the anomaly event of the current node by reverse propagation according to the pre-built anomaly analysis grid, until the source node of the anomaly is traced back.

[0072] The intelligent attribution analysis and operation and maintenance unit is used to match abnormal cases in a preset knowledge base based on the abnormal source node and abnormal event to obtain the matched abnormal cases; it determines the abnormal event level based on multiple factors such as business scenario, task operation and maintenance level, matched abnormal cases, and impact, and matches the abnormal handling mechanism; it calls the preset large model, inputs the matched abnormal cases, the results of the abnormal event level, and the matched abnormal handling mechanism into the large model, and outputs intelligent attribution analysis and alarm information.

[0073] Thirdly, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the data auditing method based on a real-time data warehouse hierarchical architecture as described in any of the preceding claims.

[0074] Fourthly, a storage medium is provided that stores a computer program, wherein the computer program is configured to execute the data auditing method based on the real-time data warehouse hierarchical architecture described above at runtime.

[0075] By utilizing the above technical solutions, the data auditing method, system, device, and medium based on a real-time data warehouse layered architecture provided in this application embodiment can quickly locate the root cause level and node of abnormal data and perform pre-level and intelligent attribution analysis of abnormal events in complex scenarios with multiple data levels, multiple stream triggers, and parallel computing. This solves the technical defects in stream computing such as auditing delays, inability to quickly and effectively locate the root cause of problems, and high error rates of manual intervention, achieving fast and accurate abnormal data auditing and effectively ensuring the data consistency of important real-time data assets. Attached Figure Description

[0076] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0077] Figure 1 A flowchart of the data auditing method based on a real-time data warehouse hierarchical architecture provided in an embodiment of this application is shown;

[0078] Figure 2 This illustration shows a case classification rule provided in an embodiment of this application;

[0079] Figure 3 This paper illustrates a structural diagram of a data audit system based on a real-time data warehouse hierarchical architecture, as provided in an embodiment of this application.

[0080] Figure 4 A structural diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0081] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0082] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."

[0083] To solve the above-mentioned technical problems, the implementation idea of ​​this application is as follows:

[0084] 1) Multi-dimensional cross-validation of data anomaly detection: By using methods such as data stream bypass computation, offline data result tracing, and real-time monitoring of task execution status, a multi-dimensional anomaly detection framework covering the entire real-time data warehouse is built around various business scenarios. This enables cross-validation between different data levels in the real-time data warehouse, as well as cross-validation between computation results and task status, avoiding false alarms caused by timeliness errors and balancing the timeliness and reliability of the anomaly detection mechanism.

[0085] 2) Anomaly Analysis Grid for Tracing Anomaly Sources: An anomaly analysis grid model is constructed by combining task concurrency operator monitoring metrics, upstream and downstream related task monitoring metrics, and data anomaly status. After the anomaly detection mechanism is triggered, the anomaly analysis grid reverse propagation algorithm is used to trace and locate the source node of each anomaly data source, realizing the tracing of multi-factor anomaly event nodes under multi-stream complex computing conditions.

[0086] 3) Intelligent operation and maintenance based on expert experience and large model attribution analysis: After locating the source node of the anomaly, the system automatically matches and classifies anomaly cases based on the expert experience database, and uses a large model to assist in attribution analysis, so that subsequent operation and maintenance personnel can intervene or the automated program can handle the classification.

[0087] Based on the above implementation ideas, this application provides a data auditing method based on a real-time data warehouse layered architecture, such as... Figure 1 As shown, the data auditing method based on a real-time data warehouse layered architecture may include the following steps S101 to S107:

[0088] Step S101: Construct one or more business scenarios for real-time data warehouse streaming computing and start streaming computing tasks at each level of the real-time data warehouse.

[0089] This step involves building one or more business scenarios for real-time data warehouse streaming computing, which can create a basic data output environment, including dimensional data tasks and scheduling configurations prepared for streaming computing tasks.

[0090] Step S102 involves constructing a multi-dimensional anomaly detection framework covering the entire data chain of the real-time data warehouse by using data stream bypass computation, offline data result tracing, and real-time monitoring of task execution status, centered around various business scenarios. The anomaly detection framework includes an anomaly detection module and multiple detection indicators.

[0091] Step S103: Set the lifecycle of the anomaly detection module, execute the anomaly detection module, and judge the anomaly events in real time based on multiple detection indicators. If there are anomalies, execute the next step of anomaly analysis grid tracing logic.

[0092] If no abnormal event occurs during this step, abnormal detection will continue throughout the lifecycle of the abnormality detection module.

[0093] Step S104, the anomaly analysis grid tracing logic, specifically involves receiving an anomaly event from the anomaly detection module, marking the current node where the anomaly event occurred; and then, based on the pre-built anomaly analysis grid, analyzing the impact of the upstream nodes of the current node on the anomaly event of the current node, until tracing back to the source node of the anomaly.

[0094] Step S105: Based on the source node and the event of the anomaly, perform anomaly case matching in the preset knowledge base to obtain the matching anomaly case.

[0095] Step S106: Based on multiple factors such as business scenario, task operation and maintenance level, matched abnormal cases, and impact, the abnormal event is pre-classified and an abnormal handling mechanism is matched.

[0096] Step S107: Call the preset large model, input the matched abnormal cases, the predetermined results of abnormal events, and the matched abnormal handling mechanism into the large model, and output intelligent attribution analysis and alarm information.

[0097] In this step, the large model, also known as the large language model, has three characteristics: first, as the name suggests, it is large in scale, with network parameters reaching tens of billions, hundreds of billions, or even more; second, it is general, meaning it is not limited to specific problems or domains; and third, it is emergent, meaning it generates unexpected new capabilities.

[0098] The large model can be an open-source large model. The parameter scale of the large model can be flexibly configured according to actual needs, such as different orders of magnitude such as 671 billion, 14 billion, 32 billion, 7 billion or 1.5 billion. The selection of the number of parameters mainly depends on the computing resource conditions and model performance requirements. This embodiment does not impose any restrictions on this.

[0099] After step S107, abnormal events and cases can be archived and related models can be optimized. The archiving here includes storing the basic attributes of abnormal events and abnormal cases, such as identifier, time, nodes involved, feature values ​​of abnormal detection, matching template, attribution results, classification results, handler, processing time, processing results, etc., and storing them in the database.

[0100] The optimization of related models, such as adjusting the coefficients of the matrices in the anomaly analysis grid, is a model performance optimization; it involves adding new templates, including template vectorization and maintenance of some basic information of the corresponding scenario, such as the scenario / task operation and maintenance level; and it also involves adding some knowledge items to the large model, etc. This embodiment does not limit these.

[0101] In complex scenarios involving multiple data levels, multiple stream triggers, and parallel computing, this embodiment constructs an anomaly detection framework and anomaly analysis grid. This framework can quickly locate the root cause level and node of abnormal data and perform pre-level and intelligent attribution analysis of abnormal events. It solves the technical defects in stream computing, such as audit delays, inability to quickly and effectively locate the root cause of problems, and high error rates of manual intervention. This enables fast and accurate auditing of abnormal data and effectively ensures the data consistency of important real-time data assets.

[0102] This application embodiment provides a possible implementation method, wherein the multiple detection indicators mentioned in step S102 above may include (1) data quality indicators; (2) data consistency indicators; and (3) operation monitoring indicators, specifically:

[0103] (1) Data quality indicators are based on data quality verification rules to verify the quality of data at each level of the real-time data warehouse. The levels of the real-time data warehouse include the Operational Data Store (ODS) layer, the Data Warehouse Detail (DWD) layer, the Data Warehouse Service (DWS) layer, and the Application (APP) layer. Among them, the ODS and APP layers are the most important as the data entry and data exit points of the real-time data warehouse. Data quality verification rules for these two levels can often achieve the greatest effect.

[0104] (2) Data consistency indicators are used to describe the consistency verification of data transmission or computation between data levels. The data feature items of the data consistency indicators include the number of records, hash values ​​of key fields, global MD5 (a widely used cryptographic hash function that can convert data of arbitrary length into a fingerprint of fixed length, mainly used to verify data integrity), predictive indicator items, distribution volatility, etc. Specifically, data consistency detection is achieved through the following audit indicator group:

[0105] The source data real-time audit indicator group specifically monitors the consistency of data collected from the upstream data warehouse in real time.

[0106] The real-time data audit indicator group at each level specifically monitors the consistency of real-time data when it is distributed from each level of the real-time data warehouse.

[0107] The target-side real-time audit indicator group specifically monitors the data consistency when data from the real-time data warehouse is distributed to other systems.

[0108] The final results offline audit indicator group specifically adopts the offline data link audit method, which takes the full calculation results of the offline data warehouse in two batches and the incremental calculation results of each data level in the real-time data warehouse for data verification. The full calculation results of the offline data warehouse and the range of the two batches of changed data are used as the comparison benchmark to calculate the coverage and accuracy of the real-time incremental data and verify the final consistency of the data in the real-time calculation task.

[0109] (3) Operation monitoring indicators are used to describe the execution status of various tasks in real-time computing scenarios. Specifically, operation monitoring is achieved through the following monitoring indicator groups:

[0110] The real-time computing task running status monitoring index group specifically detects the running status and resource status of real-time computing tasks, including indicators such as real-time task status, machine resource usage, message queue consumption latency, back pressure of each task operator, and checkpoint integrity verification.

[0111] The dimension scheduling task operation status monitoring indicator group specifically detects the operation status of data exchange, offline computing tasks and corresponding scheduling, including indicators such as scheduling workflow and the operation status of each task, scheduling queue resource usage, and task version verification.

[0112] This embodiment constructs a multi-dimensional anomaly detection framework covering the entire data chain of a real-time data warehouse by means of data stream bypass computation, offline data result tracing, and real-time monitoring of task execution status. This framework infers and analyzes abnormal events by implementing the above three types of indicators.

[0113] This application embodiment provides a possible implementation method in which abnormal event judgment is performed in real time based on multiple detection indicators in step S103, which may specifically include the following steps A1 to A4:

[0114] Step A1: After obtaining the data quality rule verification results at each level, the credibility of data quality anomaly events and the impact assessment are performed through an independent data quality indicator aggregation node.

[0115] Step A2: After checking various data consistency indicators, the credibility of data consistency anomalies and the impact assessment are performed through an independent data consistency indicator aggregation node.

[0116] Step A3: After the running monitoring indicators are detected, the impact of abnormal events in task monitoring is assessed through a running monitoring indicator detection summary node.

[0117] Step A4: The aggregated data of the above data quality indicators, data consistency indicators, and operation monitoring indicators are submitted to the unified anomaly detection node for further aggregation. All relevant indicators are combined to determine the credibility of anomalies, the number of anomalies, and the impact of anomalies, and alarm information for anomalies is generated.

[0118] This embodiment establishes independent summary nodes (such as data quality indicator summary nodes, data consistency indicator summary nodes, and operation monitoring indicator detection summary nodes), enabling the system to perform specialized credibility judgments for anomalies across different dimensions. This design allows for the use of the most suitable detection methods for various indicators, thereby reducing false alarms. The final unified anomaly detection node performs cross-indicator correlation analysis, identifying hidden anomalies that are difficult to detect from a single dimension. For example, a minor data quality deviation (such as a slight increase in field missing rate) coupled with task execution delays will significantly increase its overall risk level, making alerts more accurate.

[0119] This application provides a possible implementation method for constructing an anomaly analysis grid through the following steps:

[0120] The grid's vertical axis represents the hierarchical nodes of the real-time data warehouse. The set of hierarchical nodes is defined as follows:

[0121]

[0122] in For hierarchical nodes, x represents the real-time data warehouse data level, i represents the logical node number, and k is a positive integer;

[0123] The horizontal axis consists of parallel computing nodes that operate on the same operator. The set of parallel computing nodes is defined as follows:

[0124]

[0125] in For parallel computing nodes, j represents the parallel computing node number;

[0126] Define the edge set as:

[0127] in Hierarchical nodes with hierarchical nodes The edges between them represent data flow dependencies;

[0128] A dynamic weight matrix is ​​constructed based on the dependencies between nodes in different levels:

[0129]

[0130] in It is a dynamic weight matrix. for Time-level nodes For hierarchical nodes Influencing factors.

[0131] This application provides a possible implementation method in the embodiment of which, under ideal conditions, the influence factor in the grid... The value is 1, and the base value of each node is based on... The presence of anomalies at any given time is determined by a {0,1} tuple; however, in actual production environments, factors such as data skew, data latency, and single points of failure necessitate the introduction of adjustment terms to dynamically adjust the influencing factors and node baseline values; the steps for constructing the anomaly analysis grid also include:

[0132] First, consider data skewness, i.e., the proportion of data traffic from upstream nodes, and introduce an impact factor correction coefficient. :

[0133]

[0134] in This represents the data flow from node i at level x to node j at level y; This represents the data flow from node i at level x to node m at level y;

[0135] Considering the data latency caused by backpressure in stream computing, which in turn leads to delays in the propagation of abnormal events, the node base value is redefined. :

[0136]

[0137] in, This represents the tuple of the ideal node cardinality {0,1}. This indicates that the slope can be adjusted;

[0138] Introducing node basis value correction coefficients :

[0139]

[0140] in Related to the node's abnormal event itself; Variables representing data quality anomalies; Variables representing data inconsistency anomalies; Represents abnormal variables in runtime monitoring;

[0141] Therefore, the final calculated node anomaly factor is:

[0142]

[0143] in This refers to node anomaly factors.

[0144] This application embodiment provides a possible implementation method. After receiving the abnormal event from the abnormality detection module in step S104, the current node where the abnormal event occurred is marked; the influence of the upstream nodes of the current node on the abnormal event of the current node is analyzed according to the reverse propagation of the pre-constructed abnormality analysis grid, until the source node of the abnormality is traced back. Specifically, it may include the following steps B1 to B4:

[0145] Step B1, reverse propagation triggering, specifically involves receiving an abnormal event from the anomaly detection module, marking the current node where the abnormal event occurred, loading the dynamic weight matrix of the upstream node of the current node, and calculating the anomaly factor of the current node. ;

[0146] Step B2, back propagation, specifically uses a depth-first search algorithm to calculate the anomaly factor of the nodes on the branch along the dependent edges, and marks the nodes whose anomaly factor exceeds the first preset threshold or whose difference between the upstream anomaly factor and 0 is less than the second preset threshold as the source node.

[0147] Step B3: Source node merging and multi-source judgment. Specifically, after the depth-first search is completed, source nodes are merged. If the difference in contribution of multiple nodes is less than the third preset threshold, they are marked as concurrent abnormal sources, and the set of abnormal source nodes is output.

[0148] Step B4, source tracing evolution, specifically involves continuously performing reverse propagation and source tracing evolution over time, updating the abnormal factors and influence coefficients of each node in real time, and correcting the abnormal source node until it stabilizes.

[0149] This embodiment significantly improves the accuracy, efficiency, and interpretability of identifying the source of abnormal events in complex systems through dynamic, multi-level source tracing analysis. This method can not only accurately locate single abnormal sources but also effectively handle concurrent abnormalities from multiple sources, and achieve dynamic evolution and correction of the tracing results, thereby providing continuous and reliable decision support for system operation and maintenance and risk control.

[0150] This application embodiment provides a possible implementation method. Step S105 involves matching abnormal cases in a preset knowledge base based on the abnormal source node and the abnormal event to obtain the matched abnormal cases. Specifically, this may include the following step C1:

[0151] Step C1 involves vectorizing the data quality indicators, data consistency indicators, and operational monitoring indicators of the abnormal event based on the anomaly source node and the abnormal event. This vectorization is then used to match anomaly cases in a pre-defined knowledge base, which includes historical case templates. The vectorization method is based on event type classification, defining... This is a vector of data quality anomaly events. For the first s Each data quality anomaly event type, then for: ;

[0152] Similarly, definition This is a consistency anomaly event vector. For running monitoring of abnormal event vectors;

[0153] ;

[0154] ;

[0155] in, For the first s A consistency anomaly event type; For the first s Each type of abnormal event in the operation monitoring.

[0156] Furthermore, based on the data quality anomaly event vector, consistency anomaly event vector, and operation monitoring anomaly event vector defined above, a feature matrix of a single anomaly node is constructed, thereby obtaining the feature matrix of the entire node when the anomaly event occurs. The cosine similarity method is used to transform the current full node feature matrix. With the knowledge base historical case template matrix Perform similarity comparison: Then, set an appropriate threshold according to the actual scenario to filter out the template candidate set; for the candidate template set, since the abnormal events are approximately independent, the confidence score can be calculated by the Naive Bayes algorithm to achieve good results.

[0157] The posterior probability of classifying a given abnormal event as a template event can be obtained and used as the confidence level for that event. The template case with the highest confidence level is selected as the matching result for the knowledge base case. If there are scenarios where the confidence level is lower than a preset threshold or where the matching mechanism fails due to a mismatch between the abnormal features and the knowledge base case template, feature vector extraction and template storage are performed on the newly added abnormal scenarios. Simultaneously, the model needs to be periodically optimized based on newly added event data to gradually improve the accuracy and confidence level of event matching.

[0158] This application embodiment provides a possible implementation method. Before determining the pre-classification level of abnormal events based on multiple factors such as business scenario, task operation and maintenance level, matched abnormal cases, and impact in step S106, event classification rules are formulated. Specifically:

[0159] See Figure 2 As shown, the classification includes the classification dimensions / levels, scenario / task operation and maintenance level, abnormal case matching, scope of impact, and operation and maintenance handling mechanism (i.e., abnormal handling mechanism). The event classification adopts the "highest principle": when any one of the three classification dimensions of scenario / task operation and maintenance level, abnormal case matching, and scope of impact triggers the high-level event standard, the highest level corresponding to the three dimensions is taken as the final level of the abnormal event, ensuring that high-risk abnormalities are dealt with first.

[0160] This application embodiment provides a possible implementation method. Step S107 calls a preset large model, inputs the matched abnormal cases, the predetermined results of the abnormal events, and the matched abnormal handling mechanism into the large model, and outputs intelligent attribution analysis and alarm information. Specifically, it can be:

[0161] After completing the knowledge base case matching and event pre-classification, the large model integrates the abnormal event classification results, abnormal case details and historical case handling data to generate attribution conclusions, differentiated handling plans and expected durations, and encapsulates them into standard alarm information. It also adapts to the visual dashboard and office communication software alarms to achieve a closed loop of "analysis-recommendation-notification-visualization".

[0162] In the attribution analysis and recommendation phase, based on the entire data chain associated with the anomaly source node, three types of monitoring data characteristics—data quality, data consistency, and operational monitoring—are extracted, along with relevant task log content. These, along with anomaly event judgment criteria and the expected level of anomaly events, are used as input items for the large model for attribution analysis. Combining historical case handling experience with relevant emergency documents in the expert knowledge base, key operations and risk points in the handling process are highlighted, and the expected handling time is provided. This facilitates operations and maintenance personnel in assessing risks based on actual business conditions and notifying relevant personnel.

[0163] Here, standard alarm information encapsulation includes four core elements:

[0164] 1) Mark the anomaly level and provide a description of the anomaly case;

[0165] 2) Describe the root cause, the source of the anomaly, and its downstream impact;

[0166] 3) List the proposed solutions and operational procedures step by step;

[0167] 4) Provide links to a knowledge base and emergency documentation;

[0168] The visualization interface connects to the operation and maintenance panel, which includes a node topology diagram displayed in a hierarchical architecture, a trend chart of indicator fluctuations, a progress bar for handling tracking, a historical case comparison module, and a quick access point for automated operations. It also supports multi-dimensional filtering and data export for review and model optimization.

[0169] This application provides a possible implementation method, and the intelligent attribution analysis and operation and maintenance process is as follows: (1) Construct a full node feature matrix; (2) Match the historical case template of the knowledge base; (3) Determine whether the confidence level is met. If yes, then (4) Automated pre-level, determine the operation and maintenance processing mechanism, and (5) Intelligent attribution analysis and processing suggestions, and continue (7); If the confidence level is not met, then (6) Determine it as an unknown case, and continue (7); (7) Visual output; (8) Execute the operation and maintenance process; (9) Execute the composite procedure; (10) Determine whether the composite procedure is passed. If yes, then (11) Archive the abnormal event and update the event template library; If no, then return (1).

[0170] By combining the technical solution design of a real-time data warehouse layered architecture, this embodiment addresses the shortcomings of traditional data auditing, such as lag, single-point limitations, and reliance on manual intervention, and forms significant advantages in multiple dimensions:

[0171] Excellent in both timeliness and reliability: By constructing a multi-factor anomaly detection framework through technologies such as data stream bypass calculation and real-time monitoring of task execution status, it can achieve second-level anomaly judgment, which significantly shortens the response time compared to the traditional hour-level batch audit method. At the same time, it relies on the aggregation and verification of three types of indicators: data quality, consistency, and operation monitoring to avoid false alarms caused by timeliness errors, thereby eliminating business risk exposure and ensuring the reliability of detection results.

[0172] Real-time data warehouse end-to-end monitoring: Covers real-time monitoring of the source, all levels, and the target, and combines the analysis of abnormal events across the entire data chain, which is more helpful for anomaly case analysis than the analysis of anomalies at a single node.

[0173] Rapid source tracing in complex scenarios: Constructing an anomaly analysis grid and introducing a reverse propagation algorithm can trace the source node of anomalies in multi-stream parallel computing scenarios, solving the problem that traditional single-point detection cannot locate cross-level and multi-factor anomalies, and shortening the handling time of anomaly cases;

[0174] Intelligent operation and maintenance: It realizes automatic classification of abnormal events, replacing the traditional manual classification, standardizing the standard operation and maintenance process, reducing operational risks, and at the same time, the big model generates adaptive optimization suggestions based on the business knowledge base to assist in handling decisions;

[0175] Scalability and continuous optimization: Supports the introduction of new business scenarios and the addition of other abnormal event types to adapt to the evolving needs of complex real-time data warehouse scenarios.

[0176] It should be noted that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In practical applications, all the above possible implementation methods can be arbitrarily combined in a combined manner to form possible embodiments of this application, which will not be described in detail here.

[0177] Based on the data auditing methods based on the real-time data warehouse layered architecture provided in the above embodiments, and based on the same inventive concept, this application also provides a data auditing system based on the real-time data warehouse layered architecture.

[0178] Figure 3 This is a structural diagram of the data audit system based on a real-time data warehouse layered architecture provided in an embodiment of this application. For example... Figure 3 As shown, the data audit system based on the real-time data warehouse layered architecture may specifically include a startup unit 310, a construction unit 320, an execution and judgment unit 330, an anomaly analysis grid tracing unit 340, and an intelligent attribution analysis and operation and maintenance unit 350.

[0179] The startup unit 310 is used to build one or more business scenarios for real-time data warehouse streaming computing and to start streaming computing tasks at each level of the real-time data warehouse.

[0180] Building unit 320 is used to construct a multi-dimensional anomaly detection framework covering the entire data chain of the real-time data warehouse through data stream bypass computing, offline data result tracing, and real-time monitoring of task execution status, around various business scenarios; the anomaly detection framework includes an anomaly detection module and multiple detection indicators;

[0181] The execution and judgment unit 330 is used to set the lifecycle of the anomaly detection module, execute the anomaly detection module, and judge anomaly events in real time based on multiple detection indicators. If an anomaly event exists, the next step of the anomaly analysis grid tracing logic is executed.

[0182] The anomaly analysis grid tracing unit 340 is used to mark the current node where the anomaly event occurred after receiving an anomaly event from the anomaly detection module; and to analyze the impact of the upstream nodes of the current node on the anomaly event of the current node by reverse propagation according to the pre-constructed anomaly analysis grid, until the source node of the anomaly is traced back.

[0183] The intelligent attribution analysis and operation and maintenance unit 350 is used to match abnormal cases in a preset knowledge base based on the abnormal source node and abnormal event to obtain the matched abnormal cases; it determines the abnormal event level based on multiple factors such as business scenario, task operation and maintenance level, matched abnormal cases, and impact, and matches the abnormal handling mechanism; it calls the preset large model, inputs the matched abnormal cases, the results of the abnormal event level, and the matched abnormal handling mechanism into the large model, and outputs intelligent attribution analysis and alarm information.

[0184] This application provides a possible implementation method, and the multiple detection indicators include data quality indicators, data consistency indicators, and operation monitoring indicators;

[0185] Data quality metrics are used to verify the quality of data at each level of the real-time data warehouse based on data quality verification rules. These levels of the real-time data warehouse include the Operational Data Storage (ODS) layer, the Data Warehouse Details (DWD) layer, the Data Warehouse Service (DWS) layer, and the Data Application (APP) layer.

[0186] Data consistency metrics are used to demonstrate the consistency of data transmission or computation between data levels. Data consistency metrics include the number of records, hash values ​​of key fields, global MD5 hash, predictive metrics, and distribution volatility. Specifically, data consistency is detected through the following audit indicator group:

[0187] The source data real-time audit indicator group specifically monitors the consistency of data collected from the upstream data warehouse in real time.

[0188] The real-time data audit indicator group at each level specifically monitors the consistency of real-time data when it is distributed from each level of the real-time data warehouse.

[0189] The target-side real-time audit indicator group specifically monitors the data consistency when data from the real-time data warehouse is distributed to other systems.

[0190] The final result offline audit indicator group specifically adopts the offline data link audit method, which takes the full calculation results of the offline data warehouse in two batches and the incremental calculation results of each data level in the real-time data warehouse for data verification. The full calculation results of the offline data warehouse and the range of the two batches of changed data are used as the comparison benchmark to calculate the coverage and accuracy of the real-time incremental data and verify the final consistency of the data in the real-time calculation task.

[0191] Operational monitoring metrics are used to describe the execution status of various tasks in real-time computing scenarios. Specifically, operational monitoring is achieved through the following monitoring metric groups:

[0192] The real-time computing task running status monitoring index group specifically detects the running status and resource status of real-time computing tasks, including indicators such as real-time task status, machine resource usage, message queue consumption latency, back pressure of each task operator, and checkpoint integrity verification.

[0193] The dimension scheduling task operation status monitoring indicator group specifically detects the operation status of data exchange, offline computing tasks and corresponding scheduling, including indicators such as scheduling workflow and the operation status of each task, scheduling queue resource usage, and task version verification.

[0194] This application embodiment provides a possible implementation, wherein the execution and judgment unit 330 is further configured to:

[0195] After obtaining the data quality rule verification results at each level, the credibility of data quality anomaly events and the impact assessment are performed through an independent data quality indicator aggregation node.

[0196] After testing various data consistency indicators, a separate data consistency indicator aggregation node is used to determine the credibility of data consistency anomalies and assess their impact.

[0197] After the operation monitoring indicators are detected, the impact assessment of abnormal events in task monitoring is performed through an operation monitoring indicator detection summary node.

[0198] The aggregated data of the above data quality indicators, data consistency indicators, and operation monitoring indicators are submitted to a unified anomaly detection node for further aggregation. All relevant indicators are combined to determine the credibility of anomalies, the number of anomalies, and the impact of anomalies, and alarm information for anomalies is generated.

[0199] This application provides a possible implementation method, which may also include anomaly analysis grid building units ( Figure 3 (Not shown in the image) is used to construct an anomaly analysis mesh through the following steps:

[0200] The grid's vertical axis represents the hierarchical nodes of the real-time data warehouse. The set of hierarchical nodes is defined as follows:

[0201]

[0202] in For hierarchical nodes, x represents the real-time data warehouse data level, i represents the logical node number, and k is a positive integer;

[0203] The horizontal axis consists of parallel computing nodes that operate on the same operator. The set of parallel computing nodes is defined as follows:

[0204]

[0205] in For parallel computing nodes, j represents the parallel computing node number;

[0206] Define the edge set as:

[0207] in Hierarchical nodes with hierarchical nodes The edges between them represent data flow dependencies;

[0208] A dynamic weight matrix is ​​constructed based on the dependencies between nodes in different levels:

[0209]

[0210] in It is a dynamic weight matrix. for Time-level nodes For hierarchical nodes Influencing factors.

[0211] This application provides a possible implementation method in the embodiment of which, under ideal conditions, the influence factor in the grid... The value is 1, and the base value of each node is based on... The presence of anomalies at any given time is determined by a {0,1} tuple; however, in actual production environments, factors such as data skew, data latency, and single points of failure are present, so adjustment terms are introduced to dynamically adjust the influencing factors and node base values; the anomaly analysis grid construction unit is also used for:

[0212] First, consider data skewness, i.e., the proportion of data traffic from upstream nodes, and introduce an impact factor correction coefficient. :

[0213]

[0214] in This represents the data flow from node i at level x to node j at level y; This represents the data flow from node i at level x to node m at level y;

[0215] Considering the data latency caused by backpressure in stream computing, which in turn leads to delays in the propagation of abnormal events, the node base value is redefined. :

[0216]

[0217] in, This represents the tuple of the ideal node cardinality {0,1}. This indicates that the slope can be adjusted;

[0218] Introducing node basis value correction coefficients :

[0219]

[0220] in Related to the node's abnormal event itself; Variables representing data quality anomalies; Variables representing data inconsistency anomalies; Represents abnormal variables in runtime monitoring;

[0221] Therefore, the final calculated node anomaly factor is:

[0222]

[0223] in This refers to node anomaly factors.

[0224] This application embodiment provides a possible implementation, wherein the anomaly analysis grid tracing unit 340 is further used for:

[0225] Reverse propagation triggering specifically involves receiving an abnormal event from the anomaly detection module, marking the current node where the abnormal event occurred, loading the dynamic weight matrix of the upstream nodes of the current node, and calculating the anomaly factor of the current node. ;

[0226] Backward propagation specifically involves using a depth-first search algorithm to calculate the anomaly factor of each node on the branch along the dependent edges, and marking the node whose anomaly factor exceeds a first preset threshold or whose difference between the upstream anomaly factor and 0 is less than a second preset threshold as the source node.

[0227] The source node merging and multi-source judgment are performed after the depth-first search is completed. If the difference in contribution of multiple nodes is less than the third preset threshold, they are marked as concurrency anomalies and the set of anomaly source nodes is output.

[0228] The source tracing evolution is specifically a process of continuous reverse propagation and source tracing over time, updating the abnormal factors and influence coefficients of each node in real time, and correcting the abnormal source node until it is stable.

[0229] This application embodiment provides a possible implementation, wherein the intelligent attribution analysis and maintenance unit 350 is further used for:

[0230] Based on the anomaly source node and the anomaly event, the data quality indicators, data consistency indicators, and operational monitoring indicators of the anomaly event are vectorized into features. Anomaly case matching is then performed in a pre-defined knowledge base to obtain matched anomaly cases. The vectorization method is based on event type classification, defining... This is a vector of data quality anomaly events. For the first s Each data quality anomaly event type, then for: ;

[0231] Similarly, definition This is a consistency anomaly event vector. For running monitoring of abnormal event vectors;

[0232] ;

[0233] ;

[0234] in, For the first s A consistency anomaly event type; For the first s Each type of abnormal event in the operation monitoring.

[0235] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the data auditing method based on a real-time data warehouse hierarchical architecture of any of the above embodiments.

[0236] In an exemplary embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The illustrated electronic device 400 includes a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, for example, via a bus 402. Optionally, the electronic device 400 may also include a transceiver 404. It should be noted that in practical applications, the transceiver 404 is not limited to one type, and the structure of this electronic device 400 does not constitute a limitation on the embodiments of this application.

[0237] Processor 401 may be a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0238] Bus 402 may include a pathway for transmitting information between the aforementioned components. Bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 402 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0239] The memory 403 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0240] The memory 403 stores computer program code that executes the scheme of this application, and its execution is controlled by the processor 401. The processor 401 executes the computer program code stored in the memory 403 to implement the content shown in the foregoing method embodiments.

[0241] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0242] Based on the same inventive concept, this application also provides a storage medium storing a computer program, wherein the computer program is configured to execute the data audit method based on the real-time data warehouse hierarchical architecture of any of the above embodiments at runtime.

[0243] Those skilled in the art will clearly understand that the specific working process of the systems, devices, and modules described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0244] Those skilled in the art will understand that the technical solution of this application, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several program instructions to cause an electronic device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of this application when running the program instructions. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0245] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as electronic devices like personal computers, servers, or network devices) associated with program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the electronic device, the electronic device executes all or part of the steps of the methods described in the embodiments of this application.

[0246] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of this application, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to leave the protection scope of this application.

Claims

1. A data auditing method based on a real-time data warehouse layered architecture, characterized in that, The method includes: Construct one or more business scenarios for real-time data warehouse stream computing and launch stream computing tasks at each level of the real-time data warehouse; By utilizing data stream bypass computation, offline data result traceability, and real-time monitoring of task execution status, a multi-dimensional anomaly detection framework covering the entire data chain of the real-time data warehouse is constructed around various business scenarios. The anomaly detection framework includes an anomaly detection module and multiple detection metrics. Set the lifecycle of the anomaly detection module, execute the anomaly detection module, and judge anomaly events in real time based on multiple detection indicators. If an anomaly event exists, execute the next step of anomaly analysis grid tracing logic. The anomaly analysis grid tracing logic specifically involves receiving an anomaly event from the anomaly detection module, marking the current node where the anomaly event occurred, and then analyzing the impact of the upstream nodes of the current node on the anomaly event of the current node based on the reverse propagation of the pre-built anomaly analysis grid, until the anomaly source node is traced back. Based on the source node and event of the anomaly, anomaly cases are matched in a pre-defined knowledge base to obtain the matched anomaly cases. Anomalies are pre-classified based on multiple factors, including business scenario, task operation and maintenance level, matched anomaly cases, and impact, and anomaly handling mechanisms are matched accordingly. Call the preset large model, input the matched abnormal cases, the predetermined results of abnormal events, and the matched abnormal handling mechanism into the large model, and output intelligent attribution analysis and alarm information; The anomaly analysis grid is constructed through the following steps: The grid's vertical axis represents the hierarchical nodes of the real-time data warehouse. The set of hierarchical nodes is defined as follows: in For hierarchical nodes, x represents the real-time data warehouse data level, i represents the logical node number, and k is a positive integer; The horizontal axis consists of parallel computing nodes that operate on the same operator. The set of parallel computing nodes is defined as follows: in For parallel computing nodes, j represents the parallel computing node number; Define the edge set as: in Hierarchical nodes with hierarchical nodes The edges between them represent data flow dependencies; A dynamic weight matrix is ​​constructed based on the dependencies between nodes in different levels: in It is a dynamic weight matrix. for Time-level nodes For hierarchical nodes Influencing factors.

2. The method according to claim 1, characterized in that, Multiple monitoring indicators include data quality indicators, data consistency indicators, and operational monitoring indicators; Data quality metrics are used to verify the quality of data at each level of the real-time data warehouse based on data quality verification rules. These levels of the real-time data warehouse include the Operational Data Storage (ODS) layer, the Data Warehouse Details (DWD) layer, the Data Warehouse Service (DWS) layer, and the Data Application (APP) layer. Data consistency metrics are used to demonstrate the consistency of data transmission or computation between data levels. Data consistency metrics include the number of records, hash values ​​of key fields, global MD5 hash, predictive metrics, and distribution volatility. Specifically, data consistency is detected through the following audit indicator group: The source data real-time audit indicator group specifically monitors the consistency of data collected from the upstream data warehouse in real time. The real-time data audit indicator group at each level specifically monitors the consistency of real-time data when it is distributed from each level of the real-time data warehouse. The target-side real-time audit indicator group specifically monitors the data consistency when data from the real-time data warehouse is distributed to other systems. The final result offline audit indicator group specifically adopts the offline data link audit method, which takes the full calculation results of the offline data warehouse in two batches and the incremental calculation results of each data level in the real-time data warehouse for data verification. The full calculation results of the offline data warehouse and the range of the two batches of changed data are used as the comparison benchmark to calculate the coverage and accuracy of the real-time incremental data and verify the final consistency of the data in the real-time calculation task. Operational monitoring metrics are used to describe the execution status of various tasks in real-time computing scenarios. Specifically, operational monitoring is achieved through the following monitoring metric groups: The real-time computing task running status monitoring index group specifically detects the running status and resource status of real-time computing tasks, including indicators such as real-time task status, machine resource usage, message queue consumption latency, back pressure of each task operator, and checkpoint integrity verification. The dimension scheduling task operation status monitoring indicator group specifically detects the operation status of data exchange, offline computing tasks and corresponding scheduling, including indicators such as scheduling workflow and the operation status of each task, scheduling queue resource usage, and task version verification.

3. The method according to claim 2, characterized in that, Anomaly events are identified in real time based on multiple detection indicators, including: After obtaining the data quality rule verification results at each level, the credibility of data quality anomaly events and the impact assessment are performed through an independent data quality indicator aggregation node. After testing various data consistency indicators, a separate data consistency indicator aggregation node is used to determine the credibility of data consistency anomalies and assess their impact. After the operation monitoring indicators are detected, the impact assessment of abnormal events in task monitoring is performed through an operation monitoring indicator detection summary node. The aggregated data of the above data quality indicators, data consistency indicators, and operation monitoring indicators are submitted to a unified anomaly detection node for further aggregation. All relevant indicators are combined to determine the credibility of anomalies, the number of anomalies, and the impact of anomalies, and alarm information for anomalies is generated.

4. The method according to claim 1, characterized in that, In an ideal scenario, the influence factor in the grid The value is 1, and the base value of each node is based on... The presence of anomalies at any given time is determined by a {0,1} tuple; however, in actual production environments, factors such as data skew, data latency, and single points of failure necessitate the introduction of adjustment terms to dynamically adjust the influencing factors and node baseline values; the steps for constructing the anomaly analysis grid also include: First, consider data skewness, i.e., the proportion of data traffic from upstream nodes, and introduce an impact factor correction coefficient. : in This represents the data flow from node i at level x to node j at level y; This represents the data flow from node i at level x to node m at level y; Considering the data latency caused by backpressure in stream computing, which in turn leads to delays in the propagation of abnormal events, the node base value is redefined. : in, This represents the tuple of the ideal node cardinality {0,1}. This indicates that the slope can be adjusted; Introducing node basis value correction coefficients : in Related to the node's abnormal event itself; Variables representing data quality anomalies; Variables representing data inconsistency anomalies; Represents abnormal variables in runtime monitoring; Therefore, the final calculated node anomaly factor is: in This refers to node anomaly factors.

5. The method according to claim 4, characterized in that, Upon receiving an anomaly event from the anomaly detection module, the current node where the anomaly event occurred is marked. Based on the reverse propagation of the pre-built anomaly analysis grid, the influence of the current node's upstream nodes on the current node's anomaly event is analyzed until the source node of the anomaly is traced back, including: Reverse propagation triggering specifically involves receiving an abnormal event from the anomaly detection module, marking the current node where the abnormal event occurred, loading the dynamic weight matrix of the upstream nodes of the current node, and calculating the anomaly factor of the current node. ; Backpropagation specifically involves using a depth-first search algorithm to calculate the anomaly factor of each node on the branch along the dependent edges, and marking the node whose anomaly factor exceeds a first preset threshold or whose difference between the upstream anomaly factor and 0 is less than a second preset threshold as the source node. The source node merging and multi-source judgment are performed after the depth-first search is completed. If the difference in contribution of multiple nodes is less than the third preset threshold, they are marked as concurrency anomalies and the set of anomaly source nodes is output. The source tracing evolution is specifically a process of continuous reverse propagation and source tracing over time, updating the abnormal factors and influence coefficients of each node in real time, and correcting the abnormal source node until it is stable.

6. The method according to claim 1, characterized in that, Based on the anomaly source node and the anomaly event, anomaly cases are matched against a pre-defined knowledge base to obtain the matched anomaly cases, including: Based on the anomaly source node and the anomaly event, the data quality indicators, data consistency indicators, and operational monitoring indicators of the anomaly event are vectorized into features. Anomaly case matching is then performed in a pre-defined knowledge base to obtain matched anomaly cases. The vectorization method is based on event type classification, defining... This is a vector of data quality anomaly events. For the first s Each data quality anomaly event type, then for: ; Similarly, definition This is a consistency anomaly event vector. For running monitoring of abnormal event vectors; ； ； in, For the first s A consistency anomaly event type; For the first s Each type of abnormal event in the operation monitoring.

7. A data auditing system based on a real-time data warehouse layered architecture, characterized in that, The system includes: The startup unit is used to build one or more business scenarios for real-time data warehouse streaming computing and to start streaming computing tasks at each level of the real-time data warehouse. The building unit is used to construct an anomaly detection framework that covers the entire data chain of the real-time data warehouse in multiple dimensions, around various business scenarios, through data stream bypass computing, offline data result traceability, and real-time monitoring of task execution status; the anomaly detection framework includes an anomaly detection module and multiple detection indicators; The execution and judgment unit is used to set the lifecycle of the anomaly detection module, execute the anomaly detection module, and judge anomaly events in real time based on multiple detection indicators. If an anomaly event exists, the next step of the anomaly analysis grid tracing logic is executed. The anomaly analysis grid tracing unit is used to mark the current node where the anomaly event occurred after receiving an anomaly event from the anomaly detection module; it analyzes the impact of the upstream nodes of the current node on the anomaly event of the current node by reverse propagation according to the pre-built anomaly analysis grid, until the source node of the anomaly is traced back. The intelligent attribution analysis and operation and maintenance unit is used to match abnormal cases in a preset knowledge base based on the abnormal source node and abnormal event to obtain the matched abnormal cases; it determines the abnormal event level based on multiple factors such as business scenario, task operation and maintenance level, matched abnormal cases, and impact, and matches the abnormal handling mechanism; it calls the preset large model, inputs the matched abnormal cases, the results of the abnormal event level, and the matched abnormal handling mechanism into the large model, and outputs intelligent attribution analysis and alarm information. The system further includes an anomaly analysis grid construction unit, used to construct an anomaly analysis grid through the following steps: The grid's vertical axis represents the hierarchical nodes of the real-time data warehouse. The set of hierarchical nodes is defined as follows: in For hierarchical nodes, x represents the real-time data warehouse data level, i represents the logical node number, and k is a positive integer; The horizontal axis consists of parallel computing nodes that operate on the same operator. The set of parallel computing nodes is defined as follows: in For parallel computing nodes, j represents the parallel computing node number; Define the edge set as: in Hierarchical nodes with hierarchical nodes The edges between them represent data flow dependencies; A dynamic weight matrix is ​​constructed based on the dependencies between nodes in different levels: in It is a dynamic weight matrix. for Time-level nodes For hierarchical nodes Influencing factors.

8. An electronic device, characterized in that, The system includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the data auditing method based on a real-time data warehouse hierarchical architecture as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the data auditing method based on the real-time data warehouse hierarchical architecture as described in any one of claims 1 to 6 at runtime.

Images