Data quality monitoring system for data governance effect evaluation

Abstract

The invention discloses a data quality monitoring system for data governance effect evaluation. The operation logs of the edge nodes are obtained in real time through the log auditing module, problems or abnormal conditions occurring in the data processing process can be found in time, and the real-time performance and accuracy of data processing are guaranteed. The edge node data processing process is monitored and evaluated, so that errors possibly existing in data processing can be found and solved in time, and the data quality and reliability can be improved; through regular access test and modification test on the sensitive data of the edge node, potential security vulnerabilities or risks can be found in time. According to the scheme, efficient detection of node data quality is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data governance, and particularly relates to a data quality monitoring system for evaluating the effectiveness of data governance. Background Art

[0002] With the accelerating advancement of enterprise digital transformation, data plays an increasingly important role in enterprises. However, due to diverse data sources, large scale, complex formats, etc., it is easy to cause data quality problems, such as inaccuracy, incompleteness, inconsistency, etc. These problems will affect the decision-making ability, operation efficiency and customer satisfaction of enterprises. Therefore, it is necessary to establish an effective data governance mechanism to manage and improve data quality.

[0003] To ensure the effectiveness of data governance measures, an organization needs to establish a perfect data quality monitoring system. This system can monitor regularly, discover potential problems and handle them in a timely manner, thereby improving data credibility and availability.

[0004] However, the existing data quality monitoring systems for evaluating the effectiveness of data governance usually have difficulty storing data on different nodes according to data heat, which is not convenient for monitoring and optimizing key nodes, and it is difficult to efficiently detect the data quality of nodes. At the same time, it is difficult to conduct targeted evaluations on the management security quality of sensitive data, and it is not convenient to fully ensure the security of sensitive data. Summary of the Invention

[0005] In order to solve the deficiencies in the prior art, the present invention provides a data quality monitoring system for evaluating the effectiveness of data governance to solve the technical problems that it is not convenient to monitor and optimize key nodes, it is difficult to efficiently detect the data quality of nodes, and at the same time, it is difficult to conduct targeted evaluations on the management security quality of sensitive data, and it is not convenient to fully ensure the security of sensitive data.

[0006] To solve the above technical problems, the present invention adopts the following technical solutions.

[0007] The present invention first discloses a data quality monitoring system for evaluating the effectiveness of data governance, including a data integration layer, a data processing layer and a data quality monitoring layer;

[0008] The data integration layer includes:

[0009] A multi-source access module: collects data from databases, files and API data sources and integrates them into the edge nodes of the data integration layer;

[0010] Data processing layer: used to evaluate the access heat of the access data stored in each edge node, establish a heat data storage policy according to the evaluation result of the access heat, migrate the data in the edge node according to the heat data storage policy, desensitize the migrated data, transmit the desensitized data to the corresponding storage location, and send the access link of the corresponding sensitive data to the corresponding storage location;

[0011] Data quality monitoring layer, including:

[0012] Log audit module: According to the heat data storage policy, obtain the operation logs of the edge nodes to be detected during the detection time period, monitor the data processing process of the edge nodes, and evaluate the operation quality of data processing;

[0013] Whitelist management module: Establish a whitelist management mechanism to set the access permissions for sensitive data; set the detection time period with a preset time as the cycle, randomly select edge nodes as detection nodes during the detection time period, and perform access tests and modification tests on the sensitive data of other edge nodes outside the detection nodes.

[0014] As a further solution of the present invention: The data processing layer includes:

[0015] Data heat evaluation module: Obtain the access data of the data stored in the edge node, and evaluate the access heat of the data stored in each edge node;

[0016] Hierarchical storage module: used to establish a heat data storage policy according to the evaluation result of the access heat of the data stored in the edge node, and store the data in different edge nodes;

[0017] Desensitization management module: used to perform data desensitization processing on the data migrated in the edge node according to the sensitivity of the data and the heat data storage policy, transmit the desensitized data to the corresponding storage location, and locally store the sensitive data.

[0018] As a further solution of the present invention: The data heat evaluation module obtains the access data of the data stored in the edge node, and evaluates the access heat of the data stored in each edge node, including:

[0019] Set a data access detection time interval at every preset time period, and obtain the access data of the data stored in the edge node during the data access detection time interval, where the access data includes: the access frequency and download times of the detection data, and the upload times of the same type of data;

[0020] Evaluate the access heat of the data stored in each edge node through the following formula:

[0021]

[0022] Among them, K is the evaluation value of the access popularity of the data, f is the access frequency of the detected data in the data access detection time interval, f0 is the average access frequency of the corresponding edge node data in the data access detection time interval, da is the number of downloads of the detected data in the data access detection time interval, da0 is the average number of downloads of the corresponding edge node data in the data access detection time interval, db is the number of uploads of the same type of data of the detected data in the data access detection time interval, and db0 is the average number of uploads of the corresponding edge node data in the data access detection time interval.

[0023] As a further solution of the present invention: The hierarchical storage module establishes a popularity data storage policy according to the evaluation result of the access popularity of the data stored in the edge node, and stores the data in different edge nodes, including:

[0024] Obtain the access data of the data stored in the edge node for the access popularity evaluation result. If the access popularity evaluation value of the data is greater than the first threshold a, the access data is set as the first-level popularity data;

[0025] If the access popularity evaluation value of the data is less than or equal to a and greater than the second threshold b, the access data is set as the second-level popularity data;

[0026] If the access popularity evaluation value of the data is less than or equal to b and greater than the third threshold c, the access data is set as the third-level popularity data;

[0027] If the access popularity evaluation value of the data is less than or equal to c, the access data is set as the fourth-level popularity data;

[0028] Count the data volumes of the first-level popularity data, second-level popularity data, third-level popularity data, and fourth-level popularity data in each edge node, and establish a popularity data storage policy according to the statistical results, and store the data in different edge nodes.

[0029] As a further solution of the present invention: Establish a popularity data storage policy according to the statistical results, and store the data in different edge nodes, including:

[0030] Set the edge nodes whose data volume ranking of the first-level popularity data in each edge node accounts for the top 25% as the first-level popularity data edge nodes;

[0031] Among the edge nodes excluding the first-level popularity data edge nodes, rank the data volumes of the second-level popularity data in the edge nodes, and select the same number of edge nodes as the first-level popularity data edge nodes according to the ranking, and set them as the second-level popularity data edge nodes;

[0032] Among the edge nodes after excluding the edge nodes of the first-level popularity data and the edge nodes of the second-level popularity data, rank the data volume of the third-level popularity data in the edge nodes, and filter out the same number of edge nodes as the edge nodes of the first-level popularity data according to the ranking, and set them as the edge nodes of the third-level popularity data; and set the remaining edge nodes as the edge nodes of the fourth-level popularity data;

[0033] According to the edge nodes of different levels of popularity data set, establish the following popularity data storage strategy:

[0034] Based on the principle that only the corresponding level of popularity data is stored in the edge nodes of the first-level popularity data, the edge nodes of the second-level popularity data, the edge nodes of the third-level popularity data, and the edge nodes of the fourth-level popularity data, send the popularity data that does not belong to the corresponding level in the edge nodes of different levels of popularity data to the edge nodes of the level of popularity data corresponding to the data popularity.

[0035] As a further solution of the present invention: The desensitization management module performs data desensitization processing on the data migrated in the edge nodes according to the sensitivity of the data and the popularity data storage strategy, and transmits the desensitized data to the corresponding storage location, and locally stores the sensitive data, including:

[0036] Classify the sensitivity of the data stored in the nodes, classify personal identity information as first-level sensitive information, and classify financial information as second-level sensitive information. According to the results of the sensitivity classification, set the access rights of the first-level sensitive information and the second-level sensitive information, and each edge node controls the access to the sensitive data according to the access rights;

[0037] According to the popularity data storage strategy and the results of the sensitivity classification, perform data desensitization processing on the data migrated in the edge nodes;

[0038] Transmit the desensitized data to the corresponding storage location through the MD5 algorithm for encryption, encrypt and store the corresponding sensitive data locally, and send the access link to the corresponding storage location.

[0039] As a further solution of the present invention: The log audit module obtains the operation logs of the edge nodes to be detected during the detection time period according to the popularity data storage strategy, monitors the edge node data processing process, and performs data processing operation quality evaluation, including:

[0040] Analyze the operation logs of the data processing layer

[0041] According to the popularity data storage strategy, set the log detection time period every interval of time t, and obtain the operation logs of the data processing layer in the edge nodes of the first-level popularity data and the edge nodes of the second-level popularity data during the log detection time period;

[0042] Every interval of time 3t, set a log detection time period, and obtain the operation logs of the data processing layer in the edge nodes of the three-level heat data and the edge nodes of the four-level heat data during the log detection time period; among them, the operation logs of the data processing layer include data migration logs, data desensitization logs, and error logs;

[0043] Perform data processing operation quality evaluation through the following formula:

[0044]

[0045] Among them, α is the data processing operation quality evaluation value of the edge node, v is the average data migration speed of the edge node, v0 is the average data migration speed among all edge nodes of the same heat level as the edge node, S1 is the amount of data migrated by the edge node, z is the number of data migration error logs that occur in the edge node during the log detection time period, z0 is the number of data migration error logs that occur in all edge nodes of the same heat level as the edge node during the log detection time period, S2 is the amount of data desensitized by the edge node, x is the number of data desensitization error logs that occur in the edge node during the log detection time period, and x0 is the number of data desensitization error logs that occur in all edge nodes of the same heat level as the edge node during the log detection time period;

[0046] For edge nodes whose data processing operation quality evaluation value is greater than the preset threshold, perform data quality early warning.

[0047] As a further solution of the present invention: The red list management module establishes a red list management mechanism to set access permissions for sensitive data; set a detection time period with a preset time as the cycle, randomly select edge nodes as detection nodes during the detection time period, and perform access tests and modification tests on the sensitive data of other edge nodes except the detection nodes, including:

[0048] Set a first-level access permission for first-level sensitive information, and the first-level access permission includes: allowing authorized users to access;

[0049] Set a second-level access permission for second-level sensitive information, and the second-level access permission includes allowing authorized users and systems, as well as authenticated users to access;

[0050] Set a third-level access permission for ordinary data that is not first-level sensitive information and second-level sensitive information, and the third-level access permission includes allowing authenticated users and systems to access;

[0051] Establish a first-level sensitive information access red list and a second-level sensitive information access red list, and only allow users in the first-level sensitive information access red list to access the corresponding first-level sensitive information;

[0052] Randomly select one or more nodes from all edge nodes as detection nodes using a random number generation algorithm;

[0053] Conduct access tests on the first-level sensitive information and second-level sensitive information in the first-level heat data edge nodes and second-level heat data edge nodes;

[0054] Every interval of time t, set the first access detection time period, and conduct access tests and modification tests on the first-level heat data edge nodes and second-level heat data edge nodes during the first access detection time period;

[0055] Every interval of time t, set the second access detection time period, and conduct access tests and modification tests on the third-level heat data edge nodes and fourth-level heat data edge nodes during the second access detection time period.

[0056] As a further solution of the present invention: conducting access tests and modification tests includes:

[0057] Set the corresponding access permissions for the first-level sensitive information and second-level sensitive information on the selected detection nodes;

[0058] The detection nodes perform access operations and modification operations on the first-level sensitive information and second-level sensitive information in the edge nodes to be tested;

[0059] During the access detection time period, count the number of times of failed authorized access and successful unauthorized access to the first-level sensitive information and second-level sensitive information for each edge node, as well as the number of successful modification times for the first-level sensitive information and second-level sensitive information;

[0060] According to the statistical data, conduct node data security quality assessment through the following formula:

[0061]

[0062] Among them, β is the node data security quality assessment value, X1 is the number of times of failed authorized access to the first-level sensitive information and second-level sensitive information of the edge node, X0 is the average number of times of failed authorized access to the first-level sensitive information and second-level sensitive information among all edge nodes with the same heat level as the edge node, X2 is the number of times of successful unauthorized access to the first-level sensitive information and second-level sensitive information of the edge node, and X3 is the number of times of successful modification of the first-level sensitive information and second-level sensitive information of the edge node;

[0063] For edge nodes with a node data security quality assessment value greater than the preset threshold, conduct data quality early warning;

[0064] After the test is completed, restore the modified data to the original state and clean up all generated temporary files.

[0065] Accordingly, the present application also discloses a terminal, including a processor and a storage medium;

[0066] The storage medium is used to store instructions;

[0067] The processor is used to operate according to the instructions to execute the data quality monitoring system for data governance effectiveness evaluation described above.

[0068] Accordingly, the present application also discloses a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the data quality monitoring system for data governance effectiveness evaluation described above.

[0069] The beneficial effects of the present invention are as follows. Compared with the prior art, the present invention provides a data quality monitoring system for data governance effectiveness evaluation. The present invention can timely discover problems or abnormal situations in the data processing process by obtaining the operation logs of edge nodes in real time through the log audit module, ensuring the real-time and accuracy of data processing. Monitoring and evaluating the data processing process of edge nodes is convenient for timely discovering and solving possible errors in data processing, helping to improve data quality and reliability. At the same time, by monitoring the operation logs of edge nodes, it is convenient to quickly discover the nodes with abnormal data processing, improving the efficiency and accuracy of fault diagnosis; evaluating the quality of the data processing process of edge nodes is convenient for identifying potential performance defects and taking corresponding measures in time for performance tuning to improve the overall performance of the system. The present invention can timely discover potential security vulnerabilities or risks by regularly conducting access tests and modification tests on sensitive data of edge nodes, and take corresponding measures to repair and improve them; by conducting access tests and modification tests on sensitive data of edge nodes, it helps to discover possible system errors, abnormal operations or configuration problems, and solve the problems in time to ensure the normal operation of the system. By randomly selecting edge nodes as detection nodes during the period to be detected, the supervision intensity is enhanced, ensuring that each edge node complies with data security policies and regulations, and preventing illegal operations and abuse of sensitive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the system framework of the present invention;

[0071] Figure 2 It is a schematic diagram of the framework of the data processing layer of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0072] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0073] The embodiments described in this application are only a part of the embodiments of the present invention, not all of them. Based on the spirit of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present invention.

[0074] In view of the deficiencies of the prior art, the present invention proposes a data quality monitoring system for evaluating the effectiveness of data governance. Refer to Figure 1 As shown, the data quality monitoring system for evaluating the effectiveness of data governance disclosed by the present invention includes a data integration layer, a data processing layer, and a data quality monitoring layer;

[0075] Among them, the data integration layer includes:

[0076] Multi-source access module: Collect data from databases, files, and API data sources and integrate them into the edge nodes of the data integration layer;

[0077] Data processing layer: Used to evaluate the access heat of the access data stored in each edge node, establish a heat data storage strategy according to the access heat evaluation result, migrate the data in the edge node according to the heat data storage strategy, desensitize the migrated data, transmit the desensitized data to the corresponding storage location, and send the access link of the corresponding sensitive data to the corresponding storage location;

[0078] The data quality monitoring layer includes:

[0079] Log audit module: According to the heat data storage strategy, obtain the operation logs of the edge nodes to be detected during the detection time period, monitor the data processing process of the edge nodes, and evaluate the operation quality of data processing;

[0080] Red list management module: Establish a red list management mechanism to set the access permissions for sensitive data; set the detection time period with a preset time as the cycle, randomly select edge nodes as detection nodes during the detection time period, and conduct access tests and modification tests on the sensitive data of other edge nodes outside the detection nodes.

[0081] Specifically, in this embodiment, the data processing layer evaluates the access heat of the access data stored in each edge node, establishes a heat data storage policy according to the access heat evaluation result, migrates the data in the edge node according to the heat data storage policy, desensitizes the migrated data, transmits the desensitized data to the corresponding storage location, and sends the access link of the corresponding sensitive data to the corresponding storage location; by migrating the data with low access frequency and low data heat to specific edge nodes, and migrating the data with high data heat to specific edge nodes, different edge nodes are matched according to the different data heat for data storage in and out, and the data with high access frequency is placed in the edge node dedicated to storing "high-heat data" for storage, which is convenient for quality monitoring and performance improvement of the edge node dedicated to storing "high-heat data", and speeds up the data access speed and response time. The sensitive data is desensitized, and at the same time, the sensitive data is not migrated during the migration process, reducing the number of times the sensitive data is transmitted, which is convenient for protecting the privacy information from being leaked.

[0082] Through the log audit module of the data quality monitoring layer, according to the heat data storage policy, the operation logs of the edge nodes to be detected during the detection period are obtained, the edge node data processing process is monitored, and the data processing operation quality is evaluated; by obtaining the operation logs of the edge nodes in real time through the log audit module, problems or abnormal situations occurring in the data processing process can be found in time, ensuring the real-time and accuracy of data processing. Monitoring and evaluating the edge node data processing process is convenient for timely discovering and solving possible errors in data processing, which helps to improve the data quality and reliability. By monitoring the edge node operation logs, it is convenient to quickly find the nodes with abnormal data processing, improving the efficiency and accuracy of fault diagnosis. Evaluating the quality of the edge node data processing process is convenient for identifying potential performance bottlenecks, and taking corresponding measures in time for performance tuning to improve the overall system performance.

[0083] Establish a red - list management mechanism through the red - list management module to set access permissions for sensitive data; set a detection time period with a preset time as the cycle. Randomly select edge nodes as detection nodes during the detection time period to conduct access tests and modification tests on the sensitive data of other edge nodes outside the detection nodes. Setting access permissions for sensitive data through the red - list management mechanism facilitates effectively controlling that only authorized users or nodes can access sensitive data, thereby improving data security. By regularly conducting access tests and modification tests on the sensitive data of edge nodes, it is convenient to promptly discover potential security vulnerabilities or risks and take corresponding measures for repair and improvement; by conducting access tests and modification tests on the sensitive data of edge nodes, it helps to discover possible system errors, abnormal operations, or configuration problems and promptly solve the problems to ensure the normal operation of the system. By randomly selecting edge nodes as detection nodes within the detection time period, the supervision intensity is enhanced, ensuring that each edge node complies with data security policies and regulations and preventing illegal operations and abuse of sensitive data.

[0084] See Figure 2 , in a specific embodiment of the present invention, the data processing layer further includes:

[0085] Data heat - degree evaluation module: Obtain the access data of the data stored in the edge nodes, and evaluate the access heat - degree of the data stored in each edge node;

[0086] Hierarchical storage module, used to establish a heat - degree data storage policy according to the access heat - degree evaluation results of the data stored in the edge nodes, and store the data in different edge nodes;

[0087] Desensitization management module, used to perform data desensitization processing on the data migrated in the edge nodes according to the sensitivity of the data and the heat - degree data storage policy, and transmit the desensitized data to the corresponding storage location for local storage of sensitive data.

[0088] In a specific embodiment of the present invention, the data heat - degree evaluation module obtains the access data of the data stored in the edge nodes and evaluates the access heat - degree of the data stored in each edge node, including:

[0089] Set a data access detection time interval at every preset time period, and obtain the access data of the data stored in the edge nodes during the data access detection time interval. Among them, the access data includes: the access frequency and download times of the detection data, and the upload times of the same - type data;

[0090] Evaluate the access heat - degree of the data stored in each edge node through the following formula:

[0091]

[0092] Among them, K is the evaluation value of the access popularity of the data, f is the access frequency of the detected data in the data access detection time interval, f0 is the average access frequency of the corresponding edge node data in the data access detection time interval, da is the number of downloads of the detected data in the data access detection time interval, da0 is the average number of downloads of the corresponding edge node data in the data access detection time interval, db is the number of uploads of the same type of data of the detected data in the data access detection time interval, and db0 is the average number of uploads of the corresponding edge node data in the data access detection time interval.

[0093] In a specific embodiment of the present invention, the hierarchical storage module establishes a popularity data storage policy according to the access popularity evaluation result of the data stored in the edge node, and stores the data in different edge nodes, including:

[0094] Obtain the access data of the data stored in the edge node for the access popularity evaluation result. If the access popularity evaluation value of the data is greater than the first threshold a, the access data is set as the first-level popularity data;

[0095] If the access popularity evaluation value of the data is less than or equal to a and greater than the second threshold b, the access data is set as the second-level popularity data;

[0096] If the access popularity evaluation value of the data is less than or equal to b and greater than the third threshold c, the access data is set as the third-level popularity data;

[0097] If the access popularity evaluation value of the data is less than or equal to c, the access data is set as the fourth-level popularity data;

[0098] Count the data volumes of the first-level popularity data, second-level popularity data, third-level popularity data, and fourth-level popularity data in each edge node, and establish a popularity data storage policy according to the statistical results, and store the data in different edge nodes.

[0099] Specifically, in this embodiment, obtain the access data of the data stored in the edge node for the access popularity evaluation result. If the access popularity evaluation value of the data is greater than 2, the access data is set as the first-level popularity data;

[0100] If the access popularity evaluation value of the data is less than or equal to 2 and greater than 1.5, the access data is set as the second-level popularity data;

[0101] If the access popularity evaluation value of the data is less than or equal to 1.5 and greater than 0.5, the access data is set as the third-level popularity data;

[0102] If the access popularity evaluation value of the data is less than or equal to 0.5, the data is set as the fourth-level popularity data.

[0103] In a specific embodiment of the present invention, a popularity data storage strategy is established according to statistical results, and the data is stored in different edge nodes, including:

[0104] Among the edge nodes, the edge nodes whose data volume ranking of the first-level popularity data accounts for the top 25% are set as the first-level popularity data edge nodes;

[0105] Among the edge nodes excluding the first-level popularity data edge nodes, rank the data volume of the second-level popularity data in the edge nodes, and screen out the same number of edge nodes as the first-level popularity data edge nodes according to the ranking, and set them as the second-level popularity data edge nodes;

[0106] Among the edge nodes excluding the first-level popularity data edge nodes and the second-level popularity data edge nodes, rank the data volume of the third-level popularity data in the edge nodes, and screen out the same number of edge nodes as the first-level popularity data edge nodes according to the ranking, and set them as the third-level popularity data edge nodes; and set the remaining edge nodes as the fourth-level popularity data edge nodes;

[0107] According to the edge nodes of different levels of popularity data set, establish the following popularity data storage strategy:

[0108] Based on the principle that only the corresponding level of popularity data is stored in the first-level popularity data edge nodes, the second-level popularity data edge nodes, the third-level popularity data edge nodes, and the fourth-level popularity data edge nodes, the popularity data that does not belong to the corresponding level in the edge nodes of different levels of popularity data is sent to the edge nodes of the level of popularity data corresponding to the data popularity.

[0109] In a specific embodiment of the present invention, the desensitization management module performs data desensitization processing on the data to be migrated in the edge nodes according to the sensitivity of the data and the popularity data storage strategy, and transmits the desensitized data to the corresponding storage location, and locally stores the sensitive data, including:

[0110] Classify the sensitivity of the data stored in the node, classify personal identity information as first-level sensitive information, and classify financial information as second-level sensitive information. According to the results of the sensitivity classification, set the access permissions for the first-level sensitive information and the second-level sensitive information, and each edge node controls the access to the sensitive data according to the access permissions;

[0111] According to the popularity data storage strategy and the results of the sensitivity classification, perform data desensitization processing on the data to be migrated in the edge nodes;

[0112] Transmit the desensitized data to the corresponding storage location through the MD5 algorithm for encryption, encrypt and store the corresponding sensitive data locally, and send the access link to the corresponding storage location.

[0113] In a specific embodiment of the present invention, the log auditing module obtains the operation logs of edge nodes to be detected during a detection time period according to the heat data storage policy, monitors the data processing process of the edge nodes, and conducts a data processing operation quality assessment, including:

[0114] Analyze the operation logs of the data processing layer;

[0115] According to the heat data storage policy, every time interval t, set a log detection time period, and obtain the operation logs of the data processing layer in the edge nodes of the first-level heat data and the edge nodes of the second-level heat data during the log detection time period;

[0116] Every time interval 3t, set a log detection time period, and obtain the operation logs of the data processing layer in the edge nodes of the third-level heat data and the edge nodes of the fourth-level heat data during the log detection time period; wherein, the operation logs of the data processing layer include data migration logs, data desensitization logs, and error logs;

[0117] Perform a data processing operation quality assessment through the following formula:

[0118]

[0119] Among them, α is the data processing operation quality assessment value of the edge node, v is the average data migration speed of the edge node, v0 is the average data migration speed among all edge nodes of the same heat level as the edge node, S1 is the amount of data migrated by the edge node, z is the number of data migration error logs that occur in the edge node during the log detection time period, z0 is the number of data migration error logs that occur among all edge nodes of the same heat level as the edge node during the log detection time period, S2 is the amount of data desensitized by the edge node, x is the number of data desensitization error logs that occur in the edge node during the log detection time period, and x0 is the number of data desensitization error logs that occur among all edge nodes of the same heat level as the edge node during the log detection time period;

[0120] For edge nodes whose data processing operation quality assessment value is greater than a preset threshold, conduct a data quality warning.

[0121] Specifically, in this embodiment, the amount of data S1 migrated by the edge node is in GB, the amount of data S2 desensitized by the edge node is in MB, and for edge nodes whose data processing operation quality assessment value is greater than the preset threshold of 5, conduct a data quality warning.

[0122] In a specific embodiment of the present invention, the red list management module establishes a red list management mechanism to set access permissions for sensitive data; sets a detection time period with a preset time as a cycle, randomly selects edge nodes as detection nodes during the detection time period, and conducts access tests and modification tests on the sensitive data of other edge nodes except the detection nodes, including:

[0123] Set first-level access permissions for first-level sensitive information, and the first-level access permissions include: allowing authorized users to access;

[0124] Set second-level access permissions for second-level sensitive information, and the second-level access permissions include allowing authorized users, systems, and authenticated users to access;

[0125] Set third-level access permissions for ordinary data that is neither first-level sensitive information nor second-level sensitive information, and the third-level access permissions include allowing authenticated users and systems to access;

[0126] Establish a first-level sensitive information access red list and a second-level sensitive information access red list, and only allow users in the first-level sensitive information access red list to access the corresponding first-level sensitive information;

[0127] Randomly select one or more nodes from all edge nodes as detection nodes using a random number generation algorithm;

[0128] Conduct access tests on the first-level sensitive information and second-level sensitive information in the first-level hot data edge nodes and second-level hot data edge nodes;

[0129] Every interval of time t, set a first access detection time period, and conduct access tests and modification tests on the first-level hot data edge nodes and second-level hot data edge nodes during the first access detection time period;

[0130] Every interval of time t, set a second access detection time period, and conduct access tests and modification tests on the third-level hot data edge nodes and fourth-level hot data edge nodes during the second access detection time period.

[0131] In a specific embodiment of the present invention, the conducting of access tests and modification tests further includes:

[0132] Set the corresponding first-level sensitive information and second-level sensitive information access permissions on the selected detection nodes;

[0133] The detection nodes perform access operations and modification operations on the first-level sensitive information and second-level sensitive information in the edge nodes being tested;

[0134] During the access detection time period, count the number of times of failed authorized access and successful unauthorized access to the first-level sensitive information and second-level sensitive information for each edge node, as well as the number of successful modification times for the first-level sensitive information and second-level sensitive information;

[0135] According to the statistical data, perform node data security quality assessment through the following formula:

[0136]

[0137] Where β is the node data security quality assessment value, X1 is the number of times of failed authorized access to the first-level sensitive information and second-level sensitive information of the edge node, X0 is the average number of times of failed authorized access to the first-level sensitive information and second-level sensitive information among all edge nodes with the same popularity level as the edge node, X2 is the number of times of successful unauthorized access to the first-level sensitive information and second-level sensitive information of the edge node, and X3 is the number of times of successful modification of the first-level sensitive information and second-level sensitive information of the edge node;

[0138] For edge nodes with a node data security quality assessment value greater than the preset threshold, issue a data quality warning;

[0139] After the test is completed, restore the modified data to its original state and clean up all generated temporary files.

[0140] Specifically, in this embodiment, during the access detection time period, count the number of times of failed authorized access and successful unauthorized access to the first-level sensitive information and second-level sensitive information for each edge node, as well as the number of successful modification times for the corresponding first-level sensitive information and second-level sensitive information, perform node data security quality assessment, and issue a data quality warning for edge nodes with a node data security quality assessment value greater than the preset value 1. A node data security quality assessment value greater than the preset value 1 indicates that there is successful unauthorized access to the first-level sensitive information and second-level sensitive information of the edge node, or successful unauthorized modification of the first-level sensitive information and second-level sensitive information, or an excessive number of times of failed authorized access to the first-level sensitive information and second-level sensitive information of the edge node. At this time, a data quality warning needs to be issued for the corresponding edge node.

[0141] The beneficial effects of the present invention are as follows. Compared with the prior art, the present invention can timely discover problems or abnormal situations occurring in the data processing process by obtaining the operation logs of edge nodes in real time through the log auditing module, ensuring the real-time performance and accuracy of data processing. Monitoring and evaluating the data processing process of edge nodes facilitates timely discovery and resolution of possible errors in data processing, contributing to improving data quality and reliability. At the same time, by monitoring the operation logs of edge nodes, it is easy to quickly discover the nodes with abnormal data processing, improving the efficiency and accuracy of fault diagnosis; evaluating the quality of the data processing process of edge nodes facilitates identifying potential performance defects and taking corresponding measures in a timely manner for performance tuning to enhance the overall performance of the system. The present invention conducts access tests and modification tests on the sensitive data of edge nodes regularly, facilitating timely discovery of potential security vulnerabilities or risks and taking corresponding measures for repair and improvement; by conducting access tests and modification tests on the sensitive data of edge nodes, it helps to discover possible system errors, abnormal operations or configuration problems and resolve the problems in a timely manner to ensure the normal operation of the system. By randomly selecting edge nodes as detection nodes within the time period to be detected, the supervision intensity is enhanced, ensuring that each edge node complies with data security policies and regulations and preventing illegal operations and abuse of sensitive data.

[0142] Based on the spirit of the present invention, those skilled in the art can easily conceive that a computer program product can be obtained based on the aforementioned data quality monitoring system for evaluating the effectiveness of data governance. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to implement various aspects of the present disclosure. That is, the present application further includes a terminal, including a processor and a storage medium; the storage medium is used for storing instructions; the processor is used for operating according to the instructions to execute the data quality monitoring system for evaluating the effectiveness of data governance as described above.

[0143] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example - but not limited to - an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device, such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not construed as an instantaneous signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0144] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0145] Computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present disclosure.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the specific implementation manners of the present invention or make equivalent substitutions, and any modification or equivalent substitution that does not deviate from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A data quality monitoring system for data governance effectiveness evaluation, characterized in that: Includes data integration layer, data processing layer and data quality monitoring layer; The data integration layer includes a multi-source access module for collecting data from databases, files and API data sources and integrating them into the edge nodes of the data integration layer; The data processing layer is used to evaluate the access heat of the access data of the data stored in each edge node, establish a heat data storage strategy according to the access heat evaluation result, migrate the data in the edge node according to the heat data storage strategy, desensitize the migrated data, transfer the desensitized data to the corresponding storage location, and send the access link of the corresponding sensitive data to the corresponding storage location; The data quality monitoring layer includes: A log audit module is used to obtain the operation log of the edge node detected during the detection time period according to the heat data storage strategy, monitor the data processing process of the edge node, and evaluate the data processing operation quality; The red list management module is used to establish a red list management mechanism and set access rights for sensitive data; set a detection time period with a preset time as a cycle, randomly select edge nodes as detection nodes during the detection time period, and perform access tests and modification tests on sensitive data of other edge nodes outside the detection nodes.

2. The data quality monitoring system for data governance effectiveness evaluation according to claim 1 is characterized in that: The data processing layer further comprises: A data heat evaluation module is used to obtain access data of data stored in edge nodes and perform access heat evaluation on the data stored in each edge node; A hierarchical storage module is used to establish a hot data storage strategy based on the access heat evaluation results of the edge node storage data, and store the data in different edge nodes; The desensitization management module is used to perform data desensitization processing on the data migrated in the edge node according to the sensitivity of the data and the hot data storage strategy, and transfer the desensitized data to the corresponding storage location to store the sensitive data locally.

3. The data quality monitoring system for data governance effectiveness evaluation according to claim 2 is characterized in that: The data heat evaluation module is further used for: Setting a data access detection time interval at each preset time interval, and obtaining access data of the data stored in the edge node during the data access detection time interval, wherein the access data includes: the access frequency and download times of the detection data, and the upload times of the same type of data; The access heat of the data stored in each edge node is evaluated using the following formula: Wherein, K is the access heat evaluation value of the data, f is the access frequency of the detected data in the data access detection time interval, f0 is the average access frequency of the corresponding edge node data in the data access detection time interval, da is the number of downloads of the detected data in the data access detection time interval, da0 is the average number of downloads of the corresponding edge node data in the data access detection time interval, db is the number of uploads of the same type of data as the detected data in the data access detection time interval, and db0 is the average number of uploads of the corresponding edge node data in the data access detection time interval.

4. The data quality monitoring system for data governance effectiveness evaluation according to claim 2 is characterized in that: The hierarchical storage module is further used for: Obtain access data of data stored in the edge node to perform access heat evaluation results. If the access heat evaluation value of the data is greater than a first threshold a, the access data is set as first-level heat data; If the access heat evaluation value of the data is less than or equal to a and greater than the second threshold value b, the access data is set as secondary heat data; If the access heat evaluation value of the data is less than or equal to b and greater than the third threshold c, the access data is set to level 3 heat data; If the access heat evaluation value of the data is less than or equal to c, the access data is set to level 4 heat data; The amount of data belonging to the first-level heat data, the second-level heat data, the third-level heat data, and the fourth-level heat data in each edge node is counted, and a heat data storage strategy is established based on the statistical results to store the data in different edge nodes.

5. The data quality monitoring system for data governance effectiveness evaluation according to claim 4 is characterized in that: The heat data storage strategy is established according to the statistical results to store the data in different edge nodes, including: The edge nodes whose data volume of first-level hot data ranks in the top 25% among all edge nodes are set as first-level hot data edge nodes; Among the edge nodes after excluding the edge nodes of the first-level heat data, the data volume of the second-level heat data in the edge nodes is ranked, and the edge nodes with the same number as the edge nodes of the first-level heat data are selected according to the ranking, and set as the edge nodes of the second-level heat data; Among the edge nodes after excluding the first-level heat data edge nodes and the second-level heat data edge nodes, the data volume of the third-level heat data in the edge nodes is ranked, and the edge nodes with the same number as the first-level heat data edge nodes are selected according to the ranking, and set as the third-level heat data edge nodes; and the remaining edge nodes are set as the fourth-level heat data edge nodes; According to the edge nodes of different levels of hot data, the following hot data storage strategy is established: Based on the principle that only heat data of corresponding levels are stored in the first-level heat data edge nodes, the second-level heat data edge nodes, the third-level heat data edge nodes and the fourth-level heat data edge nodes, the heat data that does not belong to the corresponding level in the heat data edge nodes of different levels is sent to the heat data edge nodes of the level corresponding to the data heat.

6. The data quality monitoring system for data governance effectiveness evaluation according to claim 5 is characterized in that: The desensitization management module is further used to: The data stored in the node is classified into sensitivity categories, personal identity information is classified into primary sensitive information, and financial information is classified into secondary sensitive information. Based on the results of sensitivity classification, access rights for primary sensitive information and secondary sensitive information are set. Each edge node controls access to sensitive data based on access rights. According to the hot data storage strategy and sensitivity classification results, data is desensitized for the data being migrated in the edge nodes; The desensitized data is encrypted using the MD5 algorithm and transmitted to the corresponding storage location, the corresponding sensitive data is encrypted and stored locally, and the access link is sent to the corresponding storage location.

7. The data quality monitoring system for data governance effectiveness evaluation according to claim 1 is characterized in that: The log audit module is further used to: Analyzing the operation log of the data processing layer; According to the heat data storage strategy, set the log detection time period at each interval t, and obtain the operation log of the data processing layer in the first-level heat data edge node and the second-level heat data edge node in the log detection time period; At every 3t interval, a log detection time period is set to obtain the operation logs of the data processing layer in the third-level heat data edge node and the fourth-level heat data edge node in the log detection time period; wherein the operation logs of the data processing layer include data migration logs, data desensitization logs and error logs; The data processing run quality assessment is performed using the following formula: Wherein, α is the data processing operation quality evaluation value of the edge node, v is the average data migration speed of the edge node, v0 is the average data migration speed among all edge nodes of the same heat level of the edge node, S1 is the amount of data migrated by the edge node, z is the number of data migration error logs that appear in the edge node during the log detection time period, z0 is the number of data migration error logs that appear in the edge node during the log detection time period among all edge nodes of the same heat level of the edge node, S2 is the amount of data desensitized by the edge node, x is the number of data desensitization error logs that appear in the edge node during the log detection time period, and x0 is the number of data desensitization error logs that appear in the edge node during the log detection time period among all edge nodes of the same heat level of the edge node; For edge nodes whose data processing operation quality assessment value is greater than the preset threshold, a data quality warning is issued.

8. The data quality monitoring system for data governance effectiveness evaluation according to claim 1 is characterized in that: The red list management module is further used to: Setting a first-level access permission for the first-level sensitive information, wherein the first-level access permission includes allowing authorized users to access; Setting a second-level access right for the second-level sensitive information, wherein the second-level access right includes allowing authorized users and systems, as well as authenticated users to access; For ordinary data that is not primary sensitive information or secondary sensitive information, a third-level access permission is set, wherein the third-level access permission includes allowing access by authenticated users and systems; Establish a red list for access to first-level sensitive information and a red list for access to second-level sensitive information, and only allow users on the red list for access to first-level sensitive information to access the corresponding first-level sensitive information; From all edge nodes, one or more nodes are randomly selected as detection nodes using a random number generation algorithm; Perform access tests on the first-level sensitive information and second-level sensitive information in the first-level heat data edge node and the second-level heat data edge node; At each interval t, a first access detection time period is set, and access test and modification test are performed on the first-level heat data edge node and the second-level heat data edge node in the first access detection time period; At each time interval t, a second access detection time period is set, and access test and modification test are performed on the third-level heat data edge node and the fourth-level heat data edge node in the second access detection time period.

9. The data quality monitoring system for data governance effectiveness evaluation according to claim 8 is characterized in that: The access test and modification test further include: Set corresponding first-level sensitive information and second-level sensitive information access permissions on the selected detection node; The detection node performs access operations and modification operations on the primary sensitive information and the secondary sensitive information in the edge node under test; During the access detection period, count the number of failed and unauthorized access attempts to the primary and secondary sensitive information of each edge node, as well as the number of successful modifications to the primary and secondary sensitive information; Based on statistical data, node data security quality assessment is performed using the following formula: Among them, β is the node data security quality assessment value, X1 is the number of failed access attempts to the primary sensitive information and secondary sensitive information of the edge node, X0 is the average number of failed access attempts to the primary sensitive information and secondary sensitive information of all edge nodes with the same heat level, X2 is the number of unsuccessful attempts to access the primary sensitive information and secondary sensitive information of the edge node, and X3 is the number of successful modifications to the primary sensitive information and secondary sensitive information of the edge node; For edge nodes whose node data security quality assessment value is greater than the preset threshold, data quality warning is issued; After the test is completed, the modified data will be restored to its original state and all temporary files generated will be cleaned up.

10. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is used to operate according to the instructions to implement the data quality monitoring system for data governance effectiveness evaluation according to any one of claims 1-9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the data quality monitoring system for data governance effectiveness evaluation as described in any one of claims 1 to 9.

Images