Data auditing method and platform, electronic equipment and computer readable storage medium

Abstract

The invention provides a data auditing method, an auditing system, electronic equipment and a computer readable storage medium. The method comprises the steps of determining a target data set; configuring an auditing task for the target data set; wherein the auditing task is used for indicating auditing data in the target data set; uniformly dividing the auditing task into a plurality of sub-tasks, and distributing each sub-task into a first node; the number of the first nodes is set as required; calling the plurality of first nodes, and executing the plurality of subtasks in parallel to obtain a plurality of auditing results; and collecting and aggregating the plurality of auditing results to obtain an auditing report. According to the invention, the auditing efficiency can be improved, and the error probability can be reduced.

Description

Technical Field

[0001] This application relates to computer technology, and in particular to data auditing methods, platforms, electronic devices, and computer-readable storage media. Background Technology

[0002] Data auditing methods in related technologies involve manually collecting data from different business systems, exporting it to spreadsheets such as Excel, and then writing automated programs or scripts to clean, compare, and analyze the data to identify and correct inconsistencies or errors. This method relies on manual operation and basic technical tools, and while it can ensure data quality to a certain extent, it is also inefficient and prone to errors. Summary of the Invention

[0003] This application provides a data auditing method, auditing platform, electronic device, and computer-readable storage medium. Through this application, the efficiency of auditing can be improved and the probability of errors can be reduced.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a data auditing method, the method comprising: Determine the target dataset; Configure an audit task for the target dataset; wherein the audit task is used to instruct the audit of data in the target dataset; The audit task is evenly divided into multiple sub-tasks, and each sub-task is assigned to a first node; wherein the number of first nodes is set as needed. Multiple first nodes are invoked to execute the multiple subtasks in parallel, resulting in multiple audit results; Collect and aggregate the multiple audit results to obtain an audit report.

[0005] This application provides an auditing system, the auditing system comprising: An audit system is used to identify the target dataset; The audit system is also used to configure audit tasks for the target dataset; The audit system is also used to evenly divide the audit task into multiple sub-tasks and assign each sub-task to a first node; wherein the number of the first nodes is set as needed. Multiple first nodes are used to execute the multiple subtasks in parallel to obtain multiple audit results; The audit system is also used to collect and aggregate the multiple audit results to obtain an audit report.

[0006] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data auditing method provided in the embodiments of this application.

[0007] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the data auditing method provided in this application.

[0008] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data auditing method provided in this application.

[0009] The embodiments of this application have the following beneficial effects: This application can utilize multiple first nodes configured on demand to effectively process large-scale datasets and expand computing resources to accommodate increasing data volumes; it can also effectively handle different amounts of data under different business needs, solving the performance bottleneck problem encountered in related technologies when the data volume is large; furthermore, by processing data in parallel through multiple first nodes, the audit speed can be accelerated; at the same time, if there is incremental (to be audited) data in the target dataset, the audit platform can obtain and execute data audit from the target dataset, solving the time delay problem caused by batch processing of Excel spreadsheets in related technologies, achieving a high degree of automation, reducing manual operation and intervention, reducing human error, and avoiding the need for repeated development due to changes in audit rules. Attached Figure Description

[0010] Figure 1 This is a first flowchart illustrating the data auditing method provided in this application embodiment; Figure 2 This is a second flowchart illustrating the data auditing method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the architecture of a business system to be audited, provided in this application; Figure 4 This is a flowchart of a star-raven optimization algorithm provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the audit platform provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0011] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] Embodiments of this application provide a data auditing method, which is applied to an auditing platform, with reference to... Figure 1 As shown, the method includes the following steps: Step 101: Determine the target dataset.

[0016] In this embodiment of the application, the target dataset includes data to be audited.

[0017] For example, the target dataset includes business (application) data transmitted by multiple business (application) systems or multiple business (application) devices, and this business (application) data needs to be audited; for example, business (application) system A includes data 1, data 2, and data 3 to be audited; business (application) system B includes data 7, data 4, and data 5 to be audited; then, the target dataset includes data 7, data 4, data 5, data 1, data 2, and data 3 to be audited.

[0018] In this embodiment of the application, the audit platform includes an audit system and a business (application) system. Here, the audit system can be directly connected to multiple business (application) systems and obtain the data to be audited from the business (application) systems to form a target dataset. Of course, the audit platform may also include a database system connected to the business (application) systems. The data in the business (application) systems is stored in the database system, and the audit system obtains the data to be audited from the database system.

[0019] The aforementioned database system is used to store the data for this business (application). It should be noted that this application does not limit the type of database; for example, it could be an SQL database, a MySQL database, etc.

[0020] In this embodiment, the audit system can be a server or a server cluster. The audit system can be deployed entirely in the cloud or distributed across any environment; this application does not impose any limitations on this. The entity executing this application can be the audit system.

[0021] Step 102: Configure audit tasks for the target dataset.

[0022] Among them, the audit task is used to indicate the data in the audit target dataset.

[0023] In this embodiment of the application, the audit system can automatically generate audit tasks for the target dataset, that is, as long as the target dataset exists or there is data in the target dataset, an audit task is generated immediately; of course, an audit task can also be generated based on the instructions input by the user.

[0024] In this embodiment, the audit system can divide the data to be audited into multiple audit tasks based on the different types of data to be audited, and use different audit tasks to complete the audit of different types of data. Of course, the audit system can also generate one audit task that covers all the data to be audited, that is, the number of audit tasks generated in this application is not specifically limited.

[0025] Step 103: Divide the audit task into multiple sub-tasks evenly, and assign each sub-task to the first node; The number of first nodes can be set as needed.

[0026] In this embodiment, the first node is a computing node in the distributed architecture of the audit platform. The computing node can be located in the audit system or it can be independent of the audit system. If the audit system has audit requirements, the relevant tasks can be directly sent to the first node so that the first node can perform data audit and obtain audit results.

[0027] Here, each first node is configured with an independent data processing unit and storage resources.

[0028] In this embodiment, if the configured audit task includes multiple tasks, each task can be directly assigned according to the configuration information of the first node; alternatively, the multiple tasks can be further subdivided into evenly distributed subtasks, and then assigned according to the configuration information of the first node. If the configured audit task includes one task, the task can be directly assigned to the matching first node; alternatively, the task can be divided into multiple subtasks, and then assigned according to the configuration information of the first node.

[0029] In this embodiment of the application, uniform subtasks refer to subtasks whose computational load is all within a certain range, thus ensuring load balancing.

[0030] In this embodiment of the application, on-demand configuration means that the number of first nodes can be adjusted according to the number of subtasks. When there are many subtasks or a large amount of data to be processed, multiple first nodes can be configured to ensure the audit speed.

[0031] In this embodiment of the application, each subtask can only be assigned to one first node (i.e., a computing node).

[0032] In this embodiment of the application, when each first node executes a subtask, its resource usage (such as CPU, memory, storage, etc.) does not exceed its capacity.

[0033] In this embodiment of the application, a specific subtask may require the resources of a matching first node or a first node that meets specific conditions to be executed.

[0034] Step 104: Call multiple first nodes to execute multiple subtasks in parallel and obtain multiple audit results.

[0035] In this embodiment of the application, parallel execution means that each first node can be executed at any time, unaffected by the audit results of other nodes.

[0036] Step 105: Collect and aggregate multiple audit results to obtain an audit report.

[0037] In this embodiment of the application, the collected audit results are comprehensively analyzed to generate a detailed audit report; the audit report records a detailed log of all audit activities, including rule execution, results and any manual intervention.

[0038] This application provides a data auditing method. The method includes: determining a target dataset; configuring auditing tasks for the target dataset; wherein the auditing tasks are used to instruct the auditing of data in the target dataset; evenly dividing the auditing tasks into multiple subtasks and assigning each subtask to a first node; setting the number of first nodes as needed; calling multiple first nodes to execute multiple subtasks in parallel to obtain multiple audit results; collecting and aggregating multiple audit results to obtain an audit report. In other words, this application can effectively handle large-scale datasets by utilizing multiple first nodes configured on demand, and can also expand computing resources to accommodate increasing data volumes; it can also effectively handle different amounts of data under different business needs, solving the performance bottleneck problem encountered in related technologies when dealing with large amounts of data; furthermore, by processing data in parallel through multiple first nodes, the auditing speed can be accelerated; simultaneously, if there is incremental (to be audited) data in the target dataset, the auditing platform can obtain and execute data auditing from the target dataset, solving the time delay problem caused by batch processing of Excel spreadsheets in related technologies. It has a high degree of automation, reduces manual operation and intervention, reduces human error, and avoids the need for repeated development due to changes in auditing rules.

[0039] Embodiments of this application provide a data auditing method, which is applied to an auditing platform, with reference to... Figure 2 As shown, the method includes the following steps: Step 201: Call the second node to obtain multiple audit data from at least one first system and preprocess the audit data.

[0040] In this embodiment, the second node is the control node in the distributed architecture of the audit platform. This control node can be located within the audit system, or it can be independent of it. If the audit system has audit requirements, the control node is responsible for task allocation, data management, and coordination of the various computing nodes. The distributed architecture in this application includes one control node and multiple computing nodes.

[0041] In this embodiment, data preprocessing mainly includes data cleaning, data integration, data transformation, and data reduction. The data preprocessing method can be flexibly selected according to actual needs, and this embodiment does not impose specific limitations.

[0042] In this embodiment of the application, the control node imports data from different data sources into the audit system (such as the Hadoop Distributed File System in the audit system, which includes the control node) through a predefined data interface, for example, by using an ETL tool.

[0043] In this embodiment of the application, before the data enters the audit process, the second node performs preprocessing operations such as cleaning, standardization, and deduplication to ensure data quality.

[0044] In this application embodiment, the first system refers to one or more business (application) systems that contain data to be audited.

[0045] In this embodiment of the application, the second node can directly obtain the audit data from at least one first system; or it can obtain the audit data from a database system connected to at least one first system (i.e., indirectly).

[0046] Step 202: Determine the target dataset based on the preprocessed data.

[0047] In this embodiment of the application, the target dataset may be composed of multiple preprocessed data.

[0048] In this embodiment of the application, the target dataset can be a newly constructed dataset or a historical dataset. Preprocessed data is added to the historical dataset to determine the historical dataset as the target dataset.

[0049] In this embodiment, the auditing platform can distribute the data in the target dataset into N buckets to achieve load balancing. Specifically, this application can use a hash function to distribute the data in the dataset to be audited into different buckets based on a certain key value. This includes: a hash function receiving the data to be audited and converting it into a fixed-length hash value. This hash value is usually an integer. The hash value calculated by the hash function is used to determine which bucket the data should be placed in. For example, in a hash table, the hash value is used as an array index. To map the hash value to a specific bucket, a modulo operation is typically used. Specifically, if there are a total of N buckets, the hash value H is calculated as H % N, and the result is the bucket number where the data should be placed. The hash function should distribute the data to be audited as evenly as possible across the buckets.

[0050] Step 203: Configure audit tasks for the target dataset.

[0051] In this embodiment of the application, step 203, configuring the audit task for the target dataset, includes the following steps: Step A1: Determine the first information; wherein the first information includes one or more of the following: business requirement information of the first system; data quality information of the data to be audited.

[0052] In this embodiment of the application, data quality information is also referred to as data quality indicators, which include, but are not limited to: Completeness: Check whether the data records are complete and whether any necessary information is missing; Consistency: Ensure that data remains logically consistent across different records, data sources, or points in time; Compliance: Verify whether the data complies with specific laws, standards, or business rules; Accuracy: Assessing whether data accurately reflects real-world conditions or measurements.

[0053] In this embodiment of the application, business requirement information Step A2: Based on the first piece of information, determine the audit rules or audit strategy.

[0054] This application can store user-defined audit rules using a database schema or configuration files.

[0055] Here, the audit rules can be selected from the user-defined audit rules that match the first information, or they can be rules generated by the audit platform based on the first information.

[0056] This application can be integrated with a rules engine to interpret and execute audit rules, such as Drools, Jess, or other business rules management systems.

[0057] This application allows users to configure audit rules based on different data quality metrics.

[0058] Step A3: Configure audit tasks for the target dataset based on audit rules.

[0059] In this embodiment of the application, there is a one-to-one correspondence between audit rules or audit strategies and audit tasks, and multiple audit tasks can be obtained according to different audit rules or audit strategies.

[0060] Step 204: Use an intelligent optimization algorithm to train the reference model to obtain the target model.

[0061] The target model is used to indicate the allocation of the corresponding first node for the subtask.

[0062] In this embodiment, an intelligent optimization algorithm is used to solve the reference model for the allocation of subtasks among multiple first nodes. Then, the reference model is fed back and iteratively optimized based on the actual operation of the solved reference model to generate a target model that can handle audit tasks and audit data, adapting to changing environments and needs.

[0063] In some embodiments, the intelligent optimization algorithm includes a star raven optimization algorithm employing an adaptive T-distribution mutation strategy; in the foraging and storage strategy of the star raven optimization algorithm, the adaptive T-distribution mutation strategy is integrated into the position update rule, and the degree of freedom parameter and mutation probability of the T-distribution are adjusted during the iteration process of the star raven optimization algorithm.

[0064] This application considers that no metaheuristic optimization algorithm can handle all optimization problems, and that the Star Raven Optimization Algorithm also suffers from an imbalance between global and local exploitation capabilities, making it prone to getting trapped in local optima. Therefore, this proposal adopts an adaptive T-distribution mutation strategy to improve the Star Raven Optimization Algorithm, avoiding getting trapped in local optima and enhancing global search capabilities. This allows the algorithm to have better global exploitation capabilities in the early stages of iteration and good local search capabilities in the later stages of iteration, while also improving the convergence speed. The position update method is as follows: ;in, It represents the position of the optimal solution in the j-th dimension after adaptive T-distribution mutation perturbation. It represents the position of the optimal solution in the j-th dimension before the mutation perturbation, and the number of iterations serves as the degree of freedom parameter of the T-distribution.

[0065] This application integrates an adaptive T-distribution mutation strategy into the position update rule in the foraging and storage strategies of the Star Raven optimization algorithm, setting a mutation probability. This is used to control the frequency of individual mutations in the algorithm. In subsequent iterations, the degrees of freedom parameters and mutation probabilities of the T-distribution are dynamically adjusted based on the algorithm's search history and the quality of the current solution to balance global exploration and local exploitation.

[0066] Step 205: Integrate the target model into the parallel audit execution engine.

[0067] Step 206: Call the parallel audit execution engine to evenly divide the audit task into multiple subtasks.

[0068] Step 207: Invoke the parallel audit execution engine and assign each subtask to the first node.

[0069] In this embodiment of the application, during the process of allocating audit tasks and ensuring that the tasks are evenly distributed among the computing nodes, a decision variable and an objective function are defined to ensure dynamic load balancing.

[0070] It should be noted that for each audit task, the decision variable indicates which computing node the task is assigned to.

[0071] It should be noted that designing the objective function to minimize the load difference across all computing nodes can be done by minimizing the difference between the maximum load and the average load of all nodes. First, a load metric is defined. To quantify the load on each node, this metric includes CPU utilization, memory usage, disk I / O, network bandwidth usage, etc. The average load across all nodes is also considered. The value can be calculated using the following formula: N represents the total number of nodes. The node with the highest load among all nodes can be represented as... Its load is The objective function This is used to minimize the difference between the maximum load and the average load, i.e.: To further promote load balancing, the final objective function can be derived: ,in, It is a weighting factor used to balance the difference between maximum and average load and the magnitude of load fluctuation.

[0072] In this embodiment, if assigning the first subtask to the third node fails, the parallel audit execution engine is invoked to reassign the first subtask to the fourth node; wherein, the multiple first nodes include the third node and the fourth node; and the multiple subtasks include the first subtask. In this embodiment, the audit task is decomposed into multiple subtasks, each of which can be executed independently on different computing nodes. The parallel audit execution engine intelligently allocates subtasks to various computing nodes to ensure load balancing and optimal resource utilization. If a node fails, the engine should be able to reallocate its tasks to other nodes.

[0073] The parallel audit execution engine in this application is designed with scalability in mind, allowing for the addition of more computing nodes to handle larger workloads as needed. Furthermore, the engine can adapt to different audit algorithms and strategies to meet the audit requirements of various data types and business needs. Through the parallel audit execution engine, organizations can significantly improve data processing speed, shorten audit cycles, enhance resource utilization, and ultimately achieve more efficient business operations and risk management.

[0074] Step 208: Determine the second piece of information; The second piece of information includes the audit method, the content to be audited, the logic construction tool corresponding to the audit, and the execution time corresponding to the audit. Audit methods include one of the following: using the first data source in the target data source as a benchmark, comparing other data sources in the target data source with the first data source; or comparing the target data sources with each other. In this embodiment, the content to be audited can be a data field selected by the user from the auditable data, or it can be all the auditable data in the first node. It should be noted that this application performs a full audit on all data from multiple data sources during the first audit. If all data sources use MySQL databases, subsequent audit tasks will import incremental data into the central data warehouse based on the binlog logs in MySQL, achieving automated auditing.

[0075] The logic construction tools in this application include comparison operators, logical operators, and regular expressions. The comparison operators include equal to, not equal to, greater than, and less than. The logical operators include AND, OR, and NOT. Regular expressions are used for complex pattern matching.

[0076] In this application, the execution time can be either manually triggered by the user or automatically executed according to a schedule.

[0077] Step 209: Call multiple first nodes, and based on the second information, execute multiple sub-tasks in parallel to obtain multiple audit results.

[0078] Step 210: Collect and aggregate multiple audit results to obtain an audit report.

[0079] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0080] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0081] This application proposes a data auditing method based on distributed intelligent scheduling. This method can effectively distribute auditing tasks to multiple computing nodes by combining intelligent optimization algorithms, thereby achieving parallel processing and improving auditing speed. Furthermore, by customizing configurable auditing rule parameters, it can effectively improve the accuracy of auditing results.

[0082] 1. Distributed Architecture Design: This application constructs a distributed system consisting of multiple computing nodes and a central node. The computing nodes are responsible for executing specific data auditing tasks, and each node is equipped with an independent data processing unit and storage resources. The central node, as the core of the system, is responsible for task allocation, data management, and coordinating the work of each computing node.

[0083] Figure 3 This is a schematic diagram of the architecture of a business system to be audited, as provided in this application. Figure 3 As shown, business data from business systems A, B, ..., M are stored in a central data warehouse. The audit system retrieves the business data to be audited from the central data warehouse. The audit system, equipped with a parallel audit execution engine, distributes audit tasks to N nodes (computing nodes of a distributed system) based on an intelligent optimization algorithm to achieve load balancing, and aggregates the audit results from the N nodes to generate an audit report.

[0084] 2. Data Import and Preprocessing Module: The central node imports data from different data sources into its Hadoop Distributed File System through predefined data interfaces, such as using ETL tools. Before the data enters the audit process, preprocessing operations such as cleaning, standardization, and deduplication are performed to ensure data quality.

[0085] 3. Data Sharding and Load Balancing Module: Based on a hash function, the dataset to be audited is distributed into different files (buckets) according to a certain key value to achieve load balancing. Furthermore, dynamic load balancing can be performed based on the computing power of each node.

[0086] 4. Intelligent Optimization Algorithm Module: Select and implement an intelligent optimization algorithm for the dynamic allocation of audit tasks among computing nodes. This proposal uses the Squirrel Optimization Algorithm, which simulates two behaviors of squirrels. The first behavior occurs in summer and autumn, primarily involving collecting and storing squirrel seeds (food); the second behavior occurs in winter and spring, primarily involving searching and retrieving storage locations. The optimization algorithm corresponds to two strategies: a foraging and storage strategy and a cache search and retrieval strategy. Each strategy includes two different population behaviors: exploration and exploitation.

[0087] 4.1 Population Initialization The population initialization of the Star Raven optimization algorithm is the same as that of most intelligent optimization algorithms. Some points, namely Star Raven individuals, are randomly generated in the search space of the problem.

[0088] (1) In formula (1), Let j be the position of the i-th star crow at the t-th iteration; where i = 1, 2, ..., N; j = 1, 2, ..., D; To find the upper bound of the optimal parameters; is the lower bound of the optimization parameters; RM is a random number in the interval [0, 1] to ensure that the initial positions are evenly distributed in the search space.

[0089] 4.2 Foraging and Storage Strategies 4.2.1 First Stage Exploration (2) In formula (2), This represents the new position of the i-th star crow at the t-th iteration; t represents the current iteration number; Tmax is the maximum number of iterations. These are random numbers generated based on levy flight; The algorithm control parameter is set to 0.05 to avoid getting trapped in local minima; A, B, and C are three indices randomly selected from the population to facilitate the exploration of high-quality food sources. , , and A random real number in the range [0,1]. Let be the mean of all solutions in the current population along the j-th dimension of the t-th generation; This represents the j-th position of the star crow at index A during the t-th iteration; This represents the j-th position of the star crow at index B during the t-th iteration; This represents the j-th position of the star crow at index C during the t-th iteration; According to , A random number between (0,1) and (0,1) The generated value; It is a random number generated based on Levy flight; It is a random number that follows a normal distribution; (3) In formula (3), and It is a random real number in the range [0,1].

[0090] 4.2.2 First Phase Development (4) In formula (4), This represents the new position of the i-th star crow during the t-th iteration; It is the best individual in the current population. Let be the storage of the i-th star crow in the t-th iteration; l is a factor that decreases linearly from 1 to 0, used to introduce diversity in the development of NOA, which can avoid local minima that may occur when searching in a certain direction, and at the same time can speed up the convergence speed of the algorithm.

[0091] 4.3 Cache Search and Retrieval Strategies The cache search and retrieval strategy is based on two reference points chosen by the individual Star Raven to remember the location of stored food, and the calculation formula is as follows: (5) (6) (7) (8) In the above formula, and All of these are the storage locations of the i-th star crow in the t-th iteration; A vector of random values ​​in the range [0,1]; The value is a random radian in the range [0,π]. is the probability used to determine the proportion of global exploration in the search space; A and B are two different individuals randomly selected from the Star Raven population; T is the maximum number of iterations. , , It is a random number between [0,1]; the initial reference point is generated by formula (2).

[0092] 4.3.1 Second Phase Exploration (9) (10) (11) In formulas (9) to (11), C is the best solution obtained so far in the j-th dimension (i.e., the best individual in the current population), and C is an individual randomly selected from the Star Crow population. , and (i=3, 4, 5, 6, 7, 8) are all random numbers between [0, 1].

[0093] 4.3.2 Second Phase Development (12) (13) (14) Figure 4 This is a flowchart of a star-raven optimization algorithm provided in an embodiment of this application; as shown... Figure 4 As shown, the population is first initialized, then the fitness is updated; then, based on the relationship between fitness (σ) and threshold (σ1), the search stage is selected; if the first stage search corresponding to the foraging and storage strategy is selected, It is a random number between [0,1]. As the number of iterations decreases linearly to 0; if Greater than The foraging phase begins - Survey 1; if... Not greater than In the storage phase - development 1; if the second phase search corresponding to the storage area and retrieval strategy is selected, It is a random number between [0,1]. =0.4; if Greater than To locate the storage area - Survey 2; if Not greater than The process begins with retrieving food – development 2; then, based on the results of each stage, it is determined whether the maximum number of iterations has been reached. If the maximum number of iterations has not been reached, the fitness is updated, and the selection search phase begins; if the maximum number of iterations has been reached, the process ends.

[0094] This application considers that no metaheuristic optimization algorithm can handle all optimization problems, and that the Star Raven Optimization Algorithm also suffers from an imbalance between global and local exploitation capabilities and is prone to getting trapped in local optima. Therefore, this proposal adopts an adaptive T-distribution mutation strategy to improve the Star Raven Optimization Algorithm, avoiding getting trapped in local optima and enhancing global search capabilities. This allows the algorithm to have better global exploitation capabilities in the early stages of iteration and good local search capabilities in the later stages of iteration, while also improving the convergence speed of the algorithm. The position update method is as follows: (14) in, It represents the position of the optimal solution in the j-th dimension after adaptive T-distribution mutation perturbation. It represents the position of the optimal solution in the j-th dimension before the mutation perturbation, and the number of iterations serves as the degree of freedom parameter of the T-distribution.

[0095] In the foraging and storage strategies of the Star Raven optimization algorithm, the adaptive T-distribution mutation strategy is integrated into the position update rule, and a mutation probability is set. This is used to control the frequency of individual mutations in the algorithm. In subsequent iterations, the degrees of freedom parameters and mutation probabilities of the T-distribution are dynamically adjusted based on the algorithm's search history and the quality of the current solution to balance global exploration and local exploitation.

[0096] 5. Audit task allocation strategy: Use intelligent optimization algorithms to allocate audit tasks to ensure that tasks are evenly distributed among computing nodes.

[0097] 5.1 Define decision variables For each audit task, define a decision variable to indicate which computing node the task is assigned to.

[0098] 5.2 Determine the objective function The objective function is designed to minimize the load variation across all computing nodes; this can be the minimization of the difference between the maximum load and the average load of all nodes. First, a load metric is defined. To quantify the load on each node, this metric includes CPU utilization, memory usage, disk I / O, network bandwidth usage, etc. The average load across all nodes is also considered. The value can be calculated using the following formula: N represents the total number of nodes. The node with the highest load among all nodes can be represented as... Its load is The objective function This is used to minimize the difference between the maximum load and the average load, i.e.: To further promote load balancing, the final objective function can be derived: ,in, It is a weighting factor used to balance the difference between maximum and average load and the magnitude of load fluctuation.

[0099] 5.3 Constraints (1) Each audit subtask can only be assigned to one computing node.

[0100] (2) The resource usage (such as CPU, memory, storage, etc.) of each computing node shall not exceed its capacity.

[0101] (3) A specific task may require specific resources or meet specific conditions.

[0102] 5.4 Feedback and Iteration To address the problem of distributing audit tasks among child nodes, the Star Raven optimization algorithm is employed to solve the model. Subsequently, the model is iteratively optimized based on actual operational data to adapt to changing environments and requirements. Finally, the optimized model is integrated into the resource management and task scheduling system, i.e., the parallel audit execution engine, to achieve automated audit task allocation.

[0103] 6. Parallel Audit Execution Engine The audit task is broken down into multiple subtasks, each of which can be executed independently on different compute nodes. The parallel audit execution engine intelligently distributes subtasks across the compute nodes to ensure load balancing and optimal resource utilization. If a node fails, the engine should be able to reallocate its tasks to other nodes. After the parallel audit is completed, the results from each node need to be collected and aggregated to form the final audit report.

[0104] The engine was designed with scalability in mind, allowing for the addition of more computing nodes to handle larger workloads as needed. Furthermore, the engine can adapt to different audit algorithms and strategies to meet the audit requirements of various data types and business needs. Through the parallel audit execution engine, organizations can significantly improve data processing speed, shorten audit cycles, enhance resource utilization, and ultimately achieve more efficient business operations and risk management.

[0105] 7. Data Audit Module 7.1 Audit Methods Depending on the method of comparing data sources, it can be divided into the following two types: Master-slave audit: Select a primary data source as a benchmark and compare other data sources against it.

[0106] Cross-checking: Comparing data sources to identify differences or inconsistencies.

[0107] 7.2 Field-level auditing Allows users to select the data fields that need to be audited, and provides a list of fields for users to select.

[0108] 7.3 Audit Rule Configuration Allow users to configure audit rules based on different data quality metrics: Completeness: Check whether the data records are complete and whether any necessary information is missing.

[0109] Consistency: Ensure that data remains logically consistent across different records, data sources, or points in time.

[0110] Compliance: Verify whether the data complies with specific laws, standards, or business rules.

[0111] Accuracy: Assessing whether data accurately reflects real-world conditions or measurements.

[0112] 7.4 Rule Logic Construction Tools that provide the building logic, such as: Comparison operators (equal to, not equal to, greater than, less than, etc.).

[0113] Logical operators (AND, OR, NOT, etc.).

[0114] Regular expressions are used for complex pattern matching.

[0115] 7.5 Rule Storage and Management Design a database schema or use configuration files to store user-defined audit rules.

[0116] 7.6 Rule Engine Integration Integrate a rules engine to interpret and execute audit rules, such as Drools, Jess, or other business rules management systems.

[0117] 7.7 Execution and Scheduling Audits can be triggered manually or executed automatically according to a schedule. The first audit will audit all data from multiple data sources. If all data sources use MySQL databases, subsequent audit tasks will import incremental data into the central data warehouse based on the binlog logs in MySQL, thus achieving automated auditing.

[0118] 8. Comprehensive Analysis and Audit Trail Module The collected audit results are comprehensively analyzed to generate a detailed audit report.

[0119] Record detailed logs of all audit activities, including rule enforcement, results, and any manual interventions.

[0120] This application establishes a distributed task allocation mechanism, which distributes data audit tasks to multiple computing nodes in real time. The Star Raven optimization algorithm enables intelligent scheduling of audit tasks among the nodes, and the algorithm is further improved by incorporating an adaptive T-distribution mutation strategy to prevent subtasks from getting trapped in local optima during intelligent scheduling. By monitoring node status in real time and dynamically adjusting task allocation, the efficiency and load balancing of the audit process are ensured.

[0121] This application features a configurable audit rule engine that extracts key features from data and allows users to customize audit rules, including checks on the completeness, consistency, compliance, and accuracy of data quality indicators. This flexibility is a key aspect of achieving accurate data auditing in this invention.

[0122] This application features scalable data source management, which expands the number of data sources available for auditing, adapting to and incorporating them into the audit process regardless of their size. This feature enhances the system's applicability and flexibility.

[0123] This application ensures the completeness of a series of data auditing steps, including task allocation, rule configuration, load balancing among nodes, multi-data source management, and audit execution.

[0124] Embodiments of this application provide an auditing platform that can be used to implement... Figures 1 to 2 A corresponding implementation provides a data auditing method, referring to... Figure 5 As shown, the audit platform 500 includes: Audit system 501 is used to determine the target dataset; Audit system 501 is used to configure audit tasks for the target dataset; wherein, the audit task is used to instruct the audit of data in the target dataset; Audit system 501 is used to evenly divide audit tasks into multiple subtasks and assign each subtask to a first node; the number of first nodes can be set as needed. Multiple first nodes 502 are used to execute multiple subtasks in parallel and obtain multiple audit results; Audit system 501 is used to collect and aggregate multiple audit results to generate an audit report.

[0125] In other embodiments of this application, the second node 503 is used to obtain multiple audit data from at least one first system and to preprocess the audit data; Audit system 501 is used to determine the target dataset based on preprocessed data; where the data in the target dataset is distributed into N buckets; where N is a positive integer.

[0126] In other embodiments of this application, the audit system 501 is used to determine first information; wherein, the first information includes one or more of the following: business requirement information of the first system; data quality information of the data to be audited; Audit system 501 is used to determine audit rules based on the first piece of information; Audit system 501 is used to configure audit tasks for a target dataset based on audit rules.

[0127] In other embodiments of this application, the audit system 501 is used to train the reference model using an intelligent optimization algorithm to obtain a target model; wherein the target model is used to indicate the allocation of the corresponding first node for the subtask. Audit system 501 is used to integrate the target model into the parallel audit execution engine; The parallel audit execution engine in Audit System 501 divides the audit task into multiple subtasks evenly. The parallel audit execution engine in audit system 501 assigns each subtask to the first node.

[0128] In other embodiments of this application, the parallel audit execution engine in the audit system 501 is used to reassign the first subtask to the fourth node if it fails to assign the first subtask to the third node; wherein, the multiple first nodes include the third node and the fourth node; and the multiple subtasks include the first subtask.

[0129] In other embodiments of this application, the intelligent optimization algorithm includes a star raven optimization algorithm that employs an adaptive T-distribution mutation strategy; in the foraging and storage strategy of the star raven optimization algorithm, the adaptive T-distribution mutation strategy is integrated into the position update rule, and the degree of freedom parameter and mutation probability of the T-distribution are adjusted during the iteration process of the star raven optimization algorithm.

[0130] In other embodiments of this application, the audit system 501 is used to determine second information; wherein, the second information includes audit method, content to be audited, logical construction tool corresponding to the audit, and execution time corresponding to the audit; the audit method includes one of the following: using the first data source in the target data source as a benchmark, comparing other data sources in the target data source with the first data source; or comparing the target data sources with each other; Multiple first nodes 502 are used to execute multiple sub-tasks in parallel based on the second information to obtain multiple audit results.

[0131] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0132] It should be noted that, in the embodiments of this application, if the above-described data auditing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0133] Figure 6 This is a schematic structural diagram of an electronic device provided in an embodiment of this application. The electronic device may be an audit platform or an audit system. Figure 6 The illustrated electronic device 600 includes a processor 610, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0134] Optionally, such as Figure 6 As shown, the electronic device 600 may further include a memory 620. The processor 610 can retrieve and run computer programs from the memory 620 to implement the methods described in the embodiments of this application.

[0135] The memory 620 can be a separate device independent of the processor 610, or it can be integrated into the processor 610.

[0136] Optionally, such as Figure 6 As shown, the electronic device 600 may also include a transceiver 630, which the processor 610 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0137] The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.

[0138] Optionally, the electronic device 600 may specifically be an audit platform or audit system in the embodiments of this application, and the electronic device 600 may implement the corresponding processes implemented by the audit platform or audit system in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0139] This application also provides a computer program product, including a computer program that can be executed by the processor 610 of the electronic device 600 to perform the steps described in any of the foregoing methods.

[0140] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0141] As one embodiment, the processor may include one or more general-purpose central processing units (CPUs). Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores used for processing data (e.g., executing instructions).

[0142] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be ROM, Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), or flash memory. The volatile memory can be Random Access Memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0143] This application also provides a computer-readable storage medium for storing computer programs.

[0144] Optionally, the computer-readable storage medium can be applied to the audit platform / audit system / electronic device in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes implemented by the audit platform / audit system / electronic device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0145] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0146] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0147] The data auditing method, auditing platform, electronic device, and computer-readable storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0148] It should be understood that the phrases "an embodiment," "an embodiment," "an embodiment of this application," "the foregoing embodiment," "some implementations," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, the phrases "an embodiment," "an embodiment," "an embodiment of this application," "the foregoing embodiment," "some implementations," or "some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0149] Unless otherwise specified, any step performed by the audit platform / audit system / electronic device in the embodiments of this application may be executed by the processor of the audit platform / audit system / electronic device. Unless otherwise specified, the embodiments of this application do not limit the order in which the audit system / electronic device performs the following steps. Furthermore, the methods used to process data in different embodiments may be the same or different methods.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0151] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0152] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0153] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0154] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0155] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0156] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0157] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0158] The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0159] It should be noted that in the various embodiments involved in this application, all steps or some steps may be performed, as long as a complete technical solution can be formed.

[0160] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data auditing method, characterized in that, The method includes: Determine the target dataset; Configure an audit task for the target dataset; wherein the audit task is used to instruct the audit of data in the target dataset; The audit task is evenly divided into multiple sub-tasks, and each sub-task is assigned to a first node; wherein the number of first nodes is set as needed. Multiple first nodes are invoked to execute the multiple subtasks in parallel, resulting in multiple audit results; Collect and aggregate the multiple audit results to obtain an audit report.

2. The method according to claim 1, characterized in that, The determination of the target dataset includes: The second node is invoked to obtain multiple audit data from at least one first system, and the audit data is preprocessed. Based on the preprocessed data, a target dataset is determined; wherein the data in the target dataset is distributed into N buckets; wherein N is a positive integer.

3. The method according to claim 2, characterized in that, Configuring audit tasks for the target dataset includes: The first information is determined; wherein the first information includes one or more of the following: business requirement information of the first system; data quality information of the data to be audited; Based on the first information, the audit rules are determined; Based on the audit rules, configure audit tasks for the target dataset.

4. The method according to claim 1, characterized in that, The step of evenly dividing the audit task into multiple sub-tasks and assigning each sub-task to the first node includes: An intelligent optimization algorithm is used to train the reference model to obtain the target model; wherein, the target model is used to indicate the allocation of the corresponding first node for the subtask; Integrate the target model into the parallel audit execution engine; The parallel audit execution engine is invoked to evenly divide the audit task into multiple subtasks; The parallel audit execution engine is invoked to assign each subtask to the first node.

5. The method according to claim 4, characterized in that, The method further includes: If assigning the first subtask to the third node fails, the parallel audit execution engine is invoked to reassign the first subtask to the fourth node; wherein, the plurality of first nodes includes the third node and the fourth node; and the plurality of subtasks includes the first subtask.

6. The method according to claim 4, characterized in that, The intelligent optimization algorithm includes a star raven optimization algorithm that adopts an adaptive T-distribution mutation strategy. In the foraging and storage strategies of the star raven optimization algorithm, the adaptive T-distribution mutation strategy is integrated into the position update rules, and the degree of freedom parameter and mutation probability of the T-distribution are adjusted during the iteration process of the star raven optimization algorithm.

7. The method according to claim 1, characterized in that, The process involves calling multiple first nodes, executing the multiple subtasks in parallel, and obtaining multiple audit results, including: Determine the second information; wherein the second information includes the audit method, the content to be audited, the logic construction tool corresponding to the audit, and the execution time corresponding to the audit; the audit method includes one of the following: using the first data source in the target data source as a benchmark, comparing other data sources in the target data source with the first data source; or comparing the target data sources with each other; The multiple first nodes are invoked, and the multiple subtasks are executed in parallel based on the second information to obtain multiple audit results.

8. An auditing platform, characterized in that, The audit platform includes: An audit system is used to identify the target dataset; The audit system is further configured to perform audit tasks on the target dataset; wherein the audit tasks are used to instruct the audit of data in the target dataset. The audit system is also used to evenly divide the audit task into multiple sub-tasks and assign each sub-task to a first node; wherein the number of the first nodes is set as needed. Multiple first nodes are used to execute the multiple subtasks in parallel to obtain multiple audit results; The audit system is also used to collect and aggregate the multiple audit results to obtain an audit report.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data auditing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the data auditing method according to any one of claims 1 to 7.

Images