A data governance method and system based on big data

Abstract

The present application discloses a data governance method and system based on big data, the method comprising constructing a plurality of corresponding business management nodes according to the existing business system of an enterprise; obtaining initial data to be governed and performing standardized preprocessing to obtain preprocessed data to be governed; performing type and feature extraction on the preprocessed data to be governed to obtain type feature information; performing sensitive attribute verification on the preprocessed data to be governed to obtain corresponding data sensitive attribute identification, associating and matching the business management nodes of a data management network organization, and determining to obtain an initial data governance plan; matching corresponding data governance standards according to the type feature information and the data sensitive attribute identification, and optimizing and adjusting the initial data governance plan to obtain a target data governance plan; performing data governance on the preprocessed data to be governed according to the target data governance plan; the present application has the effect of standardized data governance process and high data quality, and meets the data governance needs of modern enterprises.

Description

Technical Field

[0001] The present application relates to the field of data governance technology, and in particular to a data governance method and system based on big data. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, organizations are facing an increasing amount of data and complex data structures. In this context, data governance has become one of the key tasks for managing and utilizing data. For example, in the daily operations of an enterprise, a large amount of data is generated. The goal of data governance is to ensure that data can be accurately collected, stored, managed, shared and used, so as to support the decision-making and business needs of enterprise organizations.

[0003] However, the governance of enterprise management data currently faces many challenges, such as data accuracy and completeness, data security and privacy protection, as well as problems such as irregular processes and low data quality. These problems make it difficult to meet the data governance needs of modern enterprises, and there is therefore room for improvement. Summary of the invention

[0004] In order to improve the existing problems of enterprise data governance such as irregular processes and low data quality and meet the data governance needs of modern enterprises, this application provides a data governance method and system based on big data.

[0005] In the first aspect, the invention objective of the present application is achieved by adopting the following technical solutions:

[0006] A data governance method based on big data, comprising:

[0007] Construct enterprise data governance standards according to the data governance process of each relevant business node of the enterprise's existing business system, and construct multiple business management nodes corresponding to the enterprise's existing business system, and the multiple business management nodes form a data management network organization; obtain initial data to be governed, perform standardized preprocessing on the initial data to be governed, and obtain preprocessed data to be governed; perform type and feature extraction on the preprocessed data to be governed, and obtain type feature information;

[0008] Perform sensitive attribute verification on the pre-processed data to be governed to obtain corresponding data sensitive attribute identifiers, and determine and obtain an initial data governance plan based on the association and matching of the data sensitive attribute identifiers with the business management nodes of the data management network organization; match corresponding data governance standards according to the type feature information of the pre-processed data to be governed and the data sensitive attribute identifiers, and optimize and adjust the initial data governance plan to obtain a target data governance plan;

[0009] Data governance is performed on the pre-processed data to be governed according to the target data governance solution.

[0010] By adopting the above technical solution, the enterprise data governance standard is a unified governance standard formulated with reference to the enterprise data standard rules. Based on the enterprise's existing business system and enterprise data governance standards, multiple business management nodes are associated and matched with the data governance standards corresponding to each business management node to form a data management network organization with clear division of labor and clear data governance standards for business nodes, so as to facilitate the refined and high-quality data governance of large amounts of enterprise data; when the initial data to be governed is obtained, the initial data to be governed is pre-processed in a standardized manner, and invalid and erroneous data information to be governed is filtered and cleaned to obtain pre-processed data to be governed. The next step is to divide the data types and extract data features to obtain type feature information. At the same time, the sensitive attributes of the pre-processed data to be governed are verified for data real-time and data sensitivity to obtain sensitive attributes of the associated data. Attribute identification is used to facilitate priority processing based on the real-time requirements of the pre-processed data to be governed, so as to meet the latency requirements of different types of data; then the pre-processed data to be governed is matched with the business management nodes of the data management network organization and the optimized and adjusted target data governance plan, and data governance is performed. The optimized and adjusted target data governance plan realizes change management based on continuous feedback of business needs and feedback, and the data quality is higher. This application forms a data management network organization through multiple business management nodes to build a data governance framework that meets the existing business scenarios of the enterprise. The data governance process is more standardized, and through a series of data governance processes such as standardized preprocessing, sensitive attribute verification, data feature extraction, associated data governance standards, and optimization of data governance plans, it is beneficial to improve the quality of enterprise data and the quality and efficiency of enterprise data governance.

[0011] In a preferred example of the present application: the data management network mechanism is provided with different levels for the plurality of the business management nodes, and the plurality of business management nodes of different levels form a business flow corresponding to the existing business system of the enterprise;

[0012] After the sensitive attribute verification is performed on the pre-processed data to be governed to obtain the corresponding data sensitive attribute identifier, and the business management node of the data management network organization is associated and matched based on the data sensitive attribute identifier, and an initial data governance solution is determined, the method includes:

[0013] Obtaining the data acquisition time in the data information of the pre-processed data to be managed; within a preset time period, creating multiple data transmission compression intervals according to the data acquisition time of multiple groups of the pre-processed data to be managed and the corresponding data sensitive attribute identifiers of the service management nodes of the same level of the current processing level;

[0014] According to the plurality of data transmission compression intervals, the plurality of groups of the pre-processed data to be managed are combined and compressed to obtain a plurality of compression sets of the pre-processed data to be managed;

[0015] Deliver the plurality of compressed sets of pre-processed data to be managed to a next-level business management node.

[0016] By adopting the above technical solution, in the context of large-scale enterprise data applications, there are many business management nodes at the same level, that is, enterprise data is scattered in different management user terminals. For the same data management specifications, data integration and unified management are carried out. This application corresponds the management user terminal to the business management node, and matches the node level for each business management node. For the data governance process, a business flow corresponding to the existing business system of the enterprise is formed to establish a data governance framework with clear data governance roles and governance responsibilities; specifically, this application has formulated a data transmission compression interval method with clear data transmission process and data transmission method, and the data transmission compression interval is a time interval; when faced with interactive data with strong interactivity in data processing timeliness and transmission interactivity between upper and lower nodes, Effectively ensure the timeliness of data transmission and effectively avoid data congestion and data overflow caused by frequent interaction of large amounts of data; this application creates multiple data transmission compression intervals based on the data acquisition time and corresponding data sensitive attribute identifiers of multiple groups of pre-processed data to be governed by business management nodes of the same level of the current processing level; based on multiple data transmission compression intervals, multiple groups of pre-processed data to be governed are merged and compressed to obtain multiple compression sets of pre-processed data to be governed; the pre-processed data to be governed with a large amount of data is compressed and output to the corresponding business management node in a time interval division manner, which not only effectively meets the data transmission interaction needs, but also alleviates the data processing pressure of the management user terminals of each business management node and improves data processing efficiency.

[0017] In a preferred example of the present application: the service management node includes a basic-level service management node; when the data sensitivity identifier of the basic-level service management node meets the preset optimization processing identifier condition;

[0018] The method of creating multiple data transmission compression intervals within a preset time period according to the data acquisition time of multiple groups of the pre-processed data to be managed and the corresponding data sensitive attribute identifiers of the business management nodes at the same level of the current processing level includes: determining the pre-processed data to be managed and the created data transmission compression intervals according to the data acquisition time of multiple groups of the pre-processed data to be managed by the business management nodes at the same level of the current processing level; the created data transmission compression intervals are all associated with the data to be compressed packages; the created data transmission compression intervals include expandable data transmission compression intervals;

[0019] Determine whether the pre-processed data to be managed currently is within the created data transmission compression interval;

[0020] If the data acquisition time of the currently processed pre-processed data to be managed is within the created data transmission compression interval, the currently processed pre-processed data to be managed is merged into the corresponding data to be compressed package;

[0021] If the data acquisition time of the currently processed pre-processed data to be managed is not within the created data transmission compression interval, but within the expandable interval of the created data transmission compression interval, the compression time interval range and the expandable interval time range of the created data transmission compression interval are updated based on the data acquisition time of the currently processed pre-processed data to be managed, and the currently processed pre-processed data to be managed is merged into the updated data transmission compression interval;

[0022] If the data acquisition time of the preprocessed data to be managed currently being processed is not within the created data transmission compression interval and is not within the expandable interval of the created data transmission compression interval, a new data transmission compression interval is created based on the data acquisition time of the preprocessed data to be managed currently being processed, and the preprocessed data to be managed currently being processed is merged into the new data transmission compression interval.

[0023] By adopting the above-mentioned technical scheme, the basic-level business management nodes are massive basic data for data detection and data identification, with a larger amount of data and higher processing requirements for the management user terminal server of each business management node; and when the basic-level data identification component and / or data detection component is transmitted to the corresponding business management node through different transmission networks (that is, the business management node of the basic business), the data acquisition time of the pre-processed data to be governed is distributed in disorder, and the transmission efficiency of the data transmission method that simply adopts a fixed data transmission compression interval cannot meet the transmission interaction standards of big data. Therefore, in order to improve the compression transmission efficiency of the data transmission compression interval, the present application adopts a method of dynamically calculating the data transmission compression interval. When it is determined that the data acquisition time of the currently processed pre-processed data to be governed is not within the created data transmission compression interval, the compression time interval range of the created data transmission compression interval and the expansion method of the expandable interval time range are updated based on the data acquisition time of the currently processed pre-processed data to be governed, so as to merge the currently processed pre-processed data to be governed into the expandable data transmission compression interval, thereby improving the compression and merging efficiency of the pre-processed data to be governed.

[0024] In a preferred example of the present application, the determination of whether the currently processed pre-processed data to be managed is within the created data transmission compression interval includes:

[0025] The expandable interval of the data transmission compression interval expands the data transmission compression interval into an increasing transmission compression interval or a decreasing transmission compression interval according to a preset dynamic compression duration, the time range of the increasing transmission compression interval is calculated based on the data acquisition time of the next pre-processed data to be processed; the time range of the decreasing transmission compression interval is calculated based on the data acquisition time of the next pre-processed data to be processed; the corresponding data transmission compression interval is determined to be an increasing transmission compression interval or a decreasing transmission compression interval according to the data acquisition time of the next pre-processed data to be processed; the maximum value of the maximum compression time of the increasing transmission compression interval and the decreasing transmission compression interval satisfies the sum of the values ​​of the minimum compression time and the dynamic compression duration;

[0026] When it is determined that the creation time of a certain data transmission compression interval meets the preset timeout period, the data transmission compression interval is compressed and output.

[0027] By adopting the above technical scheme, the expandable data transmission compression interval can be expanded into two situations, namely, expanded into an increasing-time transmission compression interval or a decreasing-time transmission compression interval, and determined as an increasing-time transmission compression interval or a decreasing-time transmission compression interval based on the data acquisition time of the next preprocessed data to be processed, that is, judging whether the data acquisition time of the preprocessed data to be processed currently is within the created data transmission compression interval, and further judging whether other preprocessed data to be processed can be merged forward or backward in time sequence according to the dynamic compression duration according to the judgment result; the present application provides an expandable dynamic data transmission compression interval method, which is beneficial to improving the compression and merging effect of the preprocessed data to be processed, and improving the data interaction efficiency between two adjacent business management nodes.

[0028] In a preferred example of the present application, the sensitive attribute verification of the pre-processed data to be governed is performed to obtain a corresponding data sensitive attribute identifier, and the business management node of the data management network organization is matched based on the data sensitive attribute identifier to determine an initial data governance solution, including:

[0029] Creating and associating a data index for the pre-processed data to be governed, matching a sensitive data index with a preset sensitive attribute identification rule based on the data index, and generating multiple sensitive data based on the successfully matched sensitive data index, and obtaining a sensitive attribute label for each sensitive data, wherein the sensitive attribute label includes a label type for characterizing a data type and a label level for characterizing a sensitivity level;

[0030] Determine, from the data governance solution groups corresponding to the most label types in the plurality of sensitive data, a plurality of pending data governance solutions that match the highest sensitivity level of the label level;

[0031] The business process step information of the pre-processed data to be governed is determined by the acquired data governance process configuration information, and the business management nodes connected to the data management network organization are matched according to the business process step information, and an initial data governance plan is obtained by screening from the several pending data governance plans.

[0032] By adopting the above technical solution, data security identification and differentiated data governance operations with different data security levels can be realized through business management nodes, which is conducive to associating and matching corresponding data governance solutions for highly real-time sensitive data and key data, and forming an initial data governance solution based on the business node association relationship of the enterprise's existing business system, which conforms to the actual node processing flow of the enterprise's existing business system and the security identification and security protection requirements for the governance data.

[0033] In a preferred example of the present application, the obtaining of initial data to be managed and the standardization preprocessing of the initial data to be managed to obtain preprocessed data to be managed include:

[0034] In the preset data quality specification management model, the initial data to be governed is preprocessed for data quality management to obtain preprocessed data to be governed; the data quality management includes data cleaning, data correction, data anomaly verification and detection; the preset data quality specification management model is constructed based on the random forest algorithm, the isolation forest algorithm and the data error correction algorithm, and is obtained through deep machine learning and training.

[0035] By adopting the above technical solutions, in the data quality standard management model, through intelligent data quality management methods, the initial data to be governed is cleaned, corrected, verified and detected for data anomalies, so as to efficiently achieve quality management of enterprise data governance.

[0036] In the second aspect, the invention objective of the present application is achieved by adopting the following technical solutions:

[0037] A data governance system based on big data is applied to the data governance method based on big data as described above, and the system includes:

[0038] A governance standard and management framework formulation module is used to construct enterprise data governance standards according to the data governance processes of various relevant business nodes of the enterprise's existing business system, and to construct multiple business management nodes corresponding to the enterprise's existing business system, and the multiple business management nodes form a data management network organization;

[0039] The preprocessing and information extraction module is used to obtain initial data to be managed, perform standardized preprocessing on the initial data to be managed, and obtain preprocessed data to be managed; perform type and feature extraction on the preprocessed data to be managed, and obtain type feature information;

[0040] A data attribute association and matching module is used to perform sensitive attribute verification on the pre-processed data to be governed, obtain corresponding data sensitive attribute identifiers, and associate and match the business management nodes of the data management network organization based on the data sensitive attribute identifiers to determine an initial data governance solution;

[0041] A data governance scheme optimization module, used to match the corresponding data governance standards according to the type feature information of the pre-processed data to be governed and the data sensitive attribute identifier, and optimize and adjust the initial data governance scheme to obtain a target data governance scheme;

[0042] A data governance execution module is used to perform data governance on the pre-processed data to be governed according to the target data governance solution.

[0043] By adopting the above technical solution, the node level is matched for each business management node, and the business flow corresponding to the existing business system of the enterprise is formed for the data governance process, so as to establish a data governance framework with clear data governance roles and governance responsibilities; specifically, the present application has formulated a data transmission compression interval method with clear data transmission process and data transmission method, and the data transmission compression interval is a time interval; when facing interactive data with strong data processing timeliness and interaction between upper and lower nodes, in order to effectively ensure the timeliness of data transmission and effectively avoid data congestion and data overflow caused by frequent interaction of large amounts of data; the present application creates multiple data transmission compression intervals according to the data acquisition time and corresponding data sensitive attribute identifiers of multiple groups of pre-processed data to be governed of business management nodes of the same level of the current processing level; according to multiple data transmission compression intervals, multiple groups of pre-processed data to be governed are merged and compressed, and multiple compression sets of pre-processed data to be governed are obtained accordingly; the pre-processed data to be governed with a large amount of data is compressed and output to the corresponding business management node in a time interval division manner, which not only effectively meets the data transmission interaction needs, but also alleviates the data processing pressure of the management user terminal of each business management node, and improves data processing efficiency.

[0044] In the third aspect, the invention objective of this application is achieved by adopting the following technical solutions:

[0045] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data governance method based on big data when executing the computer program.

[0046] In a fourth aspect, the invention objective of the present application is achieved by adopting the following technical solutions:

[0047] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned data governance method based on big data.

[0048] In summary, the present application includes at least one of the following beneficial technical effects:

[0049] 1. In the context of large-volume enterprise data applications, there are many business management nodes at the same level, that is, enterprise data is scattered in different management user terminals. For the same data management specifications, data integration and unified management are performed. This application corresponds the management user terminal to the business management node, and matches the node level for each business management node. A business flow corresponding to the existing business system of the enterprise is formed for the data governance process to establish a data governance framework with clear data governance roles and governance responsibilities. Specifically, this application has formulated a data transmission compression interval method with clear data transmission process and data transmission method. The data transmission compression interval is a time interval. In order to effectively ensure the timeliness of data processing and the strong interactivity of the transmission between upper and lower nodes, The timeliness of data transmission effectively avoids data congestion and data overflow caused by frequent interaction of large amounts of data; the present application creates multiple data transmission compression intervals according to the data acquisition time and corresponding data sensitive attribute identifiers of multiple groups of pre-processed data to be managed of the business management nodes of the same level of the current processing level; according to the multiple data transmission compression intervals, the multiple groups of pre-processed data to be managed are merged and compressed to obtain multiple compression sets of pre-processed data to be managed; the pre-processed data to be managed of large amounts of data are compressed and output to the corresponding business management nodes in a time interval division manner, which not only effectively meets the data transmission interaction requirements, but also alleviates the data processing pressure of the management user terminals of each business management node, and improves data processing efficiency;

[0050] 2. The present application has formulated a data transmission compression interval method with clear data transmission process and data transmission method, and the data transmission compression interval is a time interval; when faced with interactive data with strong data processing timeliness and interaction between upper and lower nodes, in order to effectively ensure the timeliness of data transmission and effectively avoid data congestion and data overflow caused by frequent interaction of large amounts of data; the present application creates multiple data transmission compression intervals based on the data acquisition time and corresponding data sensitive attribute identifiers of multiple groups of pre-processed data to be governed by business management nodes of the same level of the current processing level; based on multiple data transmission compression intervals, multiple groups of pre-processed data to be governed are merged and compressed to obtain multiple compression sets of pre-processed data to be governed; the pre-processed data to be governed with a large amount of data is compressed and output to the corresponding business management node in a time interval division manner, which not only effectively meets the data transmission interaction needs, but also alleviates the data processing pressure of the management user terminals of each business management node, and improves data processing efficiency.

[0051] 3. The basic-level business management nodes are massive basic data for data detection and data identification, with a larger amount of data and higher processing requirements for the management user terminal server of each business management node; and when the basic-level data identification component and / or data detection component is transmitted to the corresponding business management node through different transmission networks (i.e., the business management node of the basic business), the data acquisition time of the pre-processed data to be managed is distributed in disorder, and the transmission efficiency of the data transmission method that simply adopts a fixed data transmission compression interval cannot meet the transmission interaction standards of big data. Therefore, in order to improve the compression transmission efficiency of the data transmission compression interval, this application adopts a method of dynamically calculating the data transmission compression interval. When it is determined that the data acquisition time of the pre-processed data to be managed currently being processed is not within the created data transmission compression interval, the compression time interval range of the created data transmission compression interval and the expansion method of the expandable interval time range are updated based on the data acquisition time of the pre-processed data to be managed currently being processed, so as to merge the pre-processed data to be managed currently being processed into the expandable data transmission compression interval, thereby improving the compression and merging efficiency of the pre-processed data to be managed. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a data governance method based on big data in one embodiment of the present application;

[0053] Figure 2 This is a flowchart after step S3 in a data governance method based on big data in one embodiment of the present application;

[0054] Figure 3 It is a schematic diagram of a device in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The present application is further described in detail below in conjunction with the accompanying drawings.

[0056] In one embodiment, if Figure 1 As shown, the present application discloses a data governance method based on big data, which specifically includes the following steps:

[0057] S1: Build enterprise data governance standards based on the data governance processes of various relevant business nodes of the enterprise's existing business system, and build multiple business management nodes corresponding to the enterprise's existing business system. Multiple business management nodes form a data management network organization.

[0058] In this embodiment, the relevant business nodes correspond to data detection components or data identification components of the basic business, such as product detection data and fault detection data during the operation of the enterprise, basic data for daily operations, etc.; the business management nodes correspond to management user terminals of the enterprise's existing business systems; and the enterprise data governance standards are unified governance standards formulated with reference to enterprise data standard rules.

[0059] Specifically, based on the company's existing business systems and enterprise data governance standards, multiple business management nodes are associated and matched with the data governance standards corresponding to each business management node to form a data management network organization with clear division of labor and clear data governance standards for business nodes, so as to facilitate refined and high-quality data governance of large amounts of enterprise data.

[0060] S2: Acquire initial data to be governed, perform standardized preprocessing on the initial data to be governed, and obtain preprocessed data to be governed; perform type and feature extraction on the preprocessed data to be governed, and obtain type feature information.

[0061] In this embodiment, when the initial data to be governed is obtained, the initial data to be governed is preprocessed by normalization to filter out invalid and erroneous data information to be governed to obtain preprocessed data to be governed; the next step is to perform data type classification and data feature extraction to obtain type feature information.

[0062] S3: Perform sensitive attribute verification on the pre-processed data to be governed, obtain the corresponding data sensitive attribute identification, associate and match the business management nodes of the data management network organization based on the data sensitive attribute identification, and determine the initial data governance plan.

[0063] In this embodiment, sensitive attributes of data real-time and data sensitivity are checked on the pre-processed data to be governed to obtain sensitive attribute identifiers of associated data, so as to prioritize processing according to the real-time requirements of the pre-processed data to be governed itself and meet the latency requirements of different types of data.

[0064] S4: Match the corresponding data governance standards based on the type feature information of the pre-processed data to be governed and the data sensitive attribute identification, and optimize and adjust the initial data governance plan to obtain the target data governance plan.

[0065] In this embodiment, data governance is performed after the pre-processed data to be governed is matched with the business management nodes of the data management network organization and the optimized and adjusted target data governance plan. The optimized and adjusted target data governance plan realizes change management based on continuously fed back business needs and feedback, and the data governance quality is higher.

[0066] S5: Perform data governance on the pre-processed data to be governed according to the target data governance plan.

[0067] In this embodiment, the application forms a data management network organization through multiple business management nodes to build a data governance framework that conforms to the existing business scenarios of the enterprise. The data governance process is more standardized, and through a series of data governance processes such as standardized preprocessing, sensitive attribute verification, data feature extraction, associated data governance standards, and optimization of data governance solutions, it is beneficial to improve the quality of enterprise data and the quality and efficiency of enterprise data governance.

[0068] In one embodiment, if Figure 2 As shown, the data management network organization is provided with different levels for multiple business management nodes, and multiple business management nodes of different levels form a business flow corresponding to the existing business system of the enterprise; after step S3, the data governance method based on big data includes:

[0069] S31: Obtain the data acquisition time in the data information of the pre-processed data to be governed; within the preset time length, create multiple data transmission compression intervals based on the data acquisition time of multiple groups of pre-processed data to be governed and the corresponding data sensitive attribute identifiers of the business management nodes at the same level of the current processing level.

[0070] In this embodiment, the created data transmission compression intervals are all associated with data packages to be compressed; in the context of enterprise data applications with large amounts of data, there are also many business management nodes at the same level, that is, the enterprise data is scattered in different management user terminals. For the same data management specifications, data integration and unified management are performed. This application corresponds the management user terminals to the business management nodes, and matches the node level for each business management node. For the data governance process, a business flow corresponding to the existing business system of the enterprise is formed to establish a data governance framework with clear data governance roles and governance responsibilities.

[0071] Specifically, the data transmission compression interval is based on a fixed compression duration, which can be defined by the user.

[0072] S32: According to the multiple data transmission compression intervals, the multiple groups of pre-processed data to be managed are combined and compressed to obtain multiple compression sets of the pre-processed data to be managed.

[0073] In this embodiment, the present application has formulated a data transmission compression interval method with clear data transmission process and data transmission method, and the data transmission compression interval is a time interval; when faced with interactive data with strong interactivity in data processing timeliness and transmission interactivity between upper and lower nodes, in order to effectively ensure the timeliness of data transmission and effectively avoid data congestion and data overflow caused by frequent interactions of large amounts of data; the present application creates multiple data transmission compression intervals based on the data acquisition time and corresponding data sensitive attribute identifiers of multiple groups of pre-processed data to be governed by business management nodes of the same level of the current processing level; according to multiple data transmission compression intervals, multiple groups of pre-processed data to be governed are merged and compressed, and correspondingly multiple compression sets of pre-processed data to be governed are obtained.

[0074] S33: Deliver the plurality of compressed sets of pre-processed data to be managed to the next level of business management nodes.

[0075] Exemplarily, after the data transmission compression interval is created, data is processed according to the preset merge compression output rule. After the data transmission compression interval is created, the preset merge compression output rule makes the following judgment:

[0076] Step 01: Determine whether the [data acquisition time] t of the currently processed pre-processed data to be managed is within the [data transmission compression interval] [T x , T y ]Inside;

[0077] Step 02: If it does not exist, 1) directly compress and output the data to be compressed that is less than the data acquisition time of the data transmission compression interval, and hand it over to the next level of service management node for processing; 2) create a new data transmission compression interval, [Data transmission compression interval] [Tx', Ty'], Tx'=t, Ty'=t+C, C is a fixed compression duration;

[0078] Step 03, if [data acquisition time] t is within the created [data transmission compression interval] [T x , T y ], merging the group of pre-processed data to be managed into the corresponding existing data package to be compressed;

[0079] Step 04: Continue to process the next group of pre-processed data to be managed, and repeat the judgment process of steps 01 to 03.

[0080] In this embodiment, a large amount of pre-processed data to be governed is compressed and output to the corresponding business management node in a time interval division manner, which not only effectively meets the data transmission interaction needs, but also alleviates the data processing pressure of the management user terminals of each business management node and improves data processing efficiency.

[0081] In one embodiment, the service management node includes a basic-level service management node; when the data sensitivity identification of the basic-level service management node meets the preset optimization processing identification condition; in step S31, specifically including:

[0082] S311: Determine the currently processed pre-processed data to be managed and the created data transmission compression intervals according to the data acquisition time of multiple groups of pre-processed data to be managed of the business management nodes at the same level of the current processing level; the created data transmission compression intervals are all associated with the data to be compressed packages; the created data transmission compression intervals include expandable data transmission compression intervals.

[0083] In this embodiment, the basic-level business management nodes are massive basic data for data detection and data identification, with a larger amount of data and higher requirements for the management user terminal server processing of each business management node; and when the basic-level data identification components and / or data detection components are transmitted to the corresponding business management nodes (i.e., the business management nodes of the basic business) through different transmission networks, the data acquisition time of the pre-processed data to be managed is distributed in disorder, and the transmission efficiency of the data transmission method that simply uses a fixed data transmission compression interval cannot meet the transmission interaction standards of big data. Therefore, in order to improve the compression transmission efficiency of the data transmission compression interval, this application adopts a method of dynamically calculating the data transmission compression interval.

[0084] S312: Determine whether the pre-processed data to be managed currently is within the created data transmission compression interval.

[0085] S313: If the data acquisition time of the currently processed pre-processed data to be managed is within the created data transmission compression interval, the currently processed pre-processed data to be managed is merged into the corresponding data to be compressed package.

[0086] S314: If the data acquisition time of the currently processed pre-processed data to be governed is not within the created data transmission compression interval, but within the expandable interval of the created data transmission compression interval, update the compression time interval range and the expandable interval time range of the created data transmission compression interval based on the data acquisition time of the currently processed pre-processed data to be governed, and merge the currently processed pre-processed data to be governed into the updated data transmission compression interval.

[0087] Specifically, in order to improve the compression transmission efficiency of the data transmission compression interval, the present application adopts a method of dynamically calculating the data transmission compression interval. When it is determined that the data acquisition time of the pre-processed data to be governed currently being processed is not within the created data transmission compression interval, the compression time interval range and the expandable interval time range of the created data transmission compression interval are updated based on the data acquisition time of the pre-processed data to be governed currently being processed.

[0088] S315: If the data acquisition time of the pre-processed data to be managed currently being processed is not within the created data transmission compression interval and is not within the expandable interval of the created data transmission compression interval, then a new data transmission compression interval is created based on the data acquisition time of the pre-processed data to be managed currently being processed, and the pre-processed data to be managed currently being processed is merged into the new data transmission compression interval.

[0089] In this embodiment, in order to improve the merging and compression efficiency and transmission interaction efficiency of the pre-processed data to be managed, the expandable interval of the created data transmission compression interval cannot be expanded continuously, which is prone to data delay problems. Therefore, this application sets a timeout condition for each data transmission compression interval. When the data transmission compression interval meets the timeout condition, the corresponding data package to be compressed is immediately compressed and output, thereby improving the compression and merging efficiency of the pre-processed data to be managed.

[0090] In one embodiment, before step S312, the following steps are specifically included:

[0091] S3121: The expandable interval of the data transmission compression interval expands the data transmission compression interval into an increased-time transmission compression interval or a decreased-time transmission compression interval according to the preset dynamic compression duration. The time range of the increased-time transmission compression interval is calculated based on the data acquisition time of the next pre-processed data to be processed and managed; the time range of the decreased-time transmission compression interval is calculated based on the data acquisition time of the next pre-processed data to be processed and managed.

[0092] In this embodiment, the expandable data transmission compression interval can be expanded into two situations, namely, expanded into an increasing time transmission compression interval or a decreasing time transmission compression interval, and is determined as an increasing time transmission compression interval or a decreasing time transmission compression interval based on the data acquisition time of the next pre-processed data to be processed, that is, it is determined whether the data acquisition time of the pre-processed data to be processed is within the created data transmission compression interval, and based on the judgment result, the dynamic compression duration further determines whether other pre-processed data to be processed can be merged forward in time sequence, or merged backward in time sequence.

[0093] S3122: Determine the corresponding data transmission compression interval as an increasing-time transmission compression interval or a decreasing-time transmission compression interval according to the data acquisition time of the next pre-processed data to be processed; the maximum value of the maximum compression time of the increasing-time transmission compression interval and the decreasing-time transmission compression interval satisfies the sum of the minimum compression time and the dynamic compression duration.

[0094] Exemplarily, the calculation of the time-increasing transmission compression interval and the time-reducing transmission compression interval is as follows:

[0095] Assume that [data transmission compression interval] is [T x , T y ], the dynamic compression duration is D, and the expandable [data transmission compression interval] is [T y -D,T x +D]; Determine whether the [data acquisition time] t of the currently processed pre-processed data to be managed is within the [data transmission compression interval] [T x , T y ]Inside;

[0096] If T x ≤t≤T y , determine that the next piece of pre-processed data to be processed is within the existing data transmission compression interval and does not need to be expanded, that is, [data transmission compression interval] [T x , T y ] and scalable [data transmission compression interval] [T y -D,T x +D] unchanged;

[0097] If T y <t≤T x +D, determine that the data transmission compression interval corresponding to the next pre-processed data to be processed is the time-increasing transmission compression interval, and the data transmission compression interval [T x , T y ] is expanded to [T x , t], the scalable data transmission compression interval is [tD, T x +D];

[0098] T y -D<t<T x , determine the data transmission compression interval corresponding to the next pre-processed data to be processed as the reduced time transmission compression interval, the data transmission compression interval [T x , T y ] is expanded to [t, T y ], the scalable data transmission compression interval is [T y -C, t+C].

[0099] S3123: When it is determined that the creation time of a data transmission compression interval meets the preset timeout period, the data transmission compression interval is compressed and output.

[0100] Specifically, the present application provides an expandable and dynamic data transmission compression interval method, which is conducive to improving the compression and merging effect of pre-processed data to be managed, and improving the data interaction efficiency between two adjacent business management nodes.

[0101] In one embodiment, in step S3, sensitive attribute verification is performed on the pre-processed data to be governed to obtain corresponding data sensitive attribute identifiers, and the business management nodes of the data management network organization are associated and matched based on the data sensitive attribute identifiers to determine an initial data governance solution, including:

[0102] S301: Create and associate data indexes for pre-processed data to be governed, identify sensitive data indexes that match preset sensitive attribute identification rules based on the data indexes, generate multiple sensitive data based on the successfully matched sensitive data indexes, and obtain sensitive attribute labels for each sensitive data. The sensitive attribute labels include label types for characterizing data types and label levels for characterizing sensitivity levels.

[0103] In this embodiment, the sensitive attribute label is A ij , i is the label type, j is the label level, i and j are integers greater than or equal to 1; data security identification and differentiated data governance operations at different data security levels are realized through business management nodes, which is conducive to the association and matching of corresponding data governance plans for highly real-time sensitive data and key data, and forming an initial data governance plan based on the business node association relationship of the enterprise's existing business system, which conforms to the actual node processing flow of the enterprise's existing business system and the security identification and security protection requirements for governance data.

[0104] Specifically, the sensitive attribute identification rules start with the data type and set different sensitive attribute identification rules for different types of data, such as setting regular expressions for email detection and sensitive words for Chinese detection. The column name rules are configured based on the standardized data dictionary. Sensitive data identification is achieved through data indexing technology, such as full-text search engine, and different sensitive attribute labels are added to different indexes to improve the efficiency and accuracy of sensitive data identification.

[0105] Exemplarily, indexing technology, such as the full-text search engine (Lucene), realizes sensitive data identification and adds attributes to the index, which mainly includes creating indexes, querying indexes and marking sensitive data. Among them, based on the Lucene framework, a data index is created for the sensitive identified source data, including creating a document object (Document) for each data and adding attributes, including but not limited to location attributes, content attributes and quantity attributes, dividing the vocabulary into understandable words; all the vocabulary units obtained from the document analysis are indexed, and finally the corresponding Document and related attributes can be found by searching only the indexed vocabulary units.

[0106] Sensitive attribute identification rule matching is performed based on the index, that is, whether there is an index content hit is queried based on the specific rules of the sensitive attribute identification rule. Based on the Lucene framework, the query object can be defined using the query abstract class, which supports association queries between conditions using and, or, and not. The Lucene search results are traversed through the top-level documents. A matching threshold must be set for the sensitive data identification results. A match is successful when the number of matches / the number of current data attributes exceeds the threshold, and the information can be marked as sensitive information.

[0107] S302: Determine, from the data governance solution group corresponding to the label type with the most label types in the multiple sensitive data, a number of pending data governance solutions that match the highest sensitivity level of the label level.

[0108] S303: Determine the business process step information for preprocessing the data to be governed through the acquired data governance process configuration information, match the business management nodes connected to the data management network organization according to the business process step information, and select an initial data governance plan from several pending data governance plans.

[0109] Exemplarily, eight groups of sensitive data are generated through step S302, and the label attribute of each group of sensitive data is A 12 , A 32 , A 24 , A 31 , A 22 , A 13 , A 26 , A 25 ; Among them, the number of times the label type attribute is 1 is 2, the number of times the label type attribute is 2 is 4, and the number of times the label type attribute is 3 is 2, so the label type attribute is determined to be 2. Therefore, the corresponding data governance solution is found from the grouping of data governance solutions whose label type is 2. Further, the data whose label type is 2 are A 24 , A 22 , A 26 , A 25, the highest sensitivity level is 6, so it is necessary to find a data governance solution with a sensitivity level of 6 from the data governance solution group with label type 2, and label the sensitive attribute as A 26 The corresponding data governance plan is determined as the initial data governance plan.

[0110] In one embodiment, in step S2, the data governance method based on big data further includes:

[0111] S21: In the preset data quality specification management model, the initial data to be governed is preprocessed for data quality management to obtain preprocessed data to be governed; data quality management includes data cleaning, data correction, data anomaly verification and detection.

[0112] In this embodiment, in the data quality standard management model, an intelligent data quality management method is used to perform data cleaning, data correction, data anomaly verification and detection on the initial data to be governed, so as to efficiently achieve quality management of enterprise data governance.

[0113] S22: The preset data quality specification management model is built based on the random forest algorithm, isolation forest algorithm and data error correction algorithm, and is obtained through deep machine learning and training.

[0114] Specifically, based on the random forest algorithm, random features of the initial data to be governed are extracted and predicted, multiple decision trees are constructed to classify the initial data to be governed, the classification results of each decision tree are integrated, and the category with the highest frequency in the classification results is selected as the prediction result; based on the isolation forest algorithm, abnormal data in the initial data to be governed are identified, and the initial data to be governed with low density and isolation in the feature space are detected by constructing a random binary search tree to obtain the initial data to be governed information; then, the anomaly score in the initial data to be governed information is calculated, and the anomaly score is compared with a preset anomaly score value. When it is judged that the anomaly score of the initial data to be governed information is greater than the preset anomaly score value, the initial data to be governed information is abnormal data information, otherwise it is normal data information; thereby achieving the purpose of data anomaly verification and detection of the initial data to be governed.

[0115] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0116] In one embodiment, a data governance system based on big data is provided, and the data governance system based on big data corresponds to a data governance method based on big data in the above-mentioned embodiment.

[0117] A data governance system based on big data includes a governance standard and management framework formulation module, a preprocessing and information extraction module, a data attribute association matching module, a data governance solution optimization module, and a data governance execution module. The detailed description of each functional module is as follows:

[0118] The governance standard and management framework formulation module is used to build enterprise data governance standards based on the data governance processes of various relevant business nodes of the enterprise's existing business system, and to build multiple business management nodes corresponding to the enterprise's existing business system. Multiple business management nodes form a data management network organization;

[0119] The preprocessing and information extraction module is used to obtain the initial data to be governed, perform standardized preprocessing on the initial data to be governed, and obtain preprocessed data to be governed; perform type and feature extraction on the preprocessed data to be governed, and obtain type feature information;

[0120] The data attribute association matching module is used to perform sensitive attribute verification on the pre-processed data to be governed, obtain the corresponding data sensitive attribute identification, and associate and match the business management nodes of the data management network organization based on the data sensitive attribute identification to determine the initial data governance plan;

[0121] The data governance scheme optimization module is used to match the corresponding data governance standards based on the type feature information of the pre-processed data to be governed and the data sensitive attribute identification, and optimize and adjust the initial data governance scheme to obtain the target data governance scheme;

[0122] The data governance execution module is used to perform data governance on the pre-processed data to be governed according to the target data governance plan.

[0123] For the specific definition of a data governance system based on big data, please refer to the definition of a data governance method based on big data in the above text, which will not be repeated here; each module in the above-mentioned data governance system based on big data can be implemented in whole or in part through software, hardware and a combination thereof; each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0124] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pre-processed data to be managed, target data management solutions, data transmission compression intervals and data quality specification management models, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data management method based on big data is implemented.

[0125] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, steps of a data governance method based on big data are implemented.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a data governance method based on big data are implemented.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0128] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0129] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the features thereof may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data governance method based on big data, characterized in that: include: Establish enterprise data governance standards based on the data governance processes of various relevant business nodes of the enterprise's existing business system, and establish multiple business management nodes corresponding to the enterprise's existing business system, and the multiple business management nodes form a data management network organization; Acquire initial data to be managed, and perform standardized preprocessing on the initial data to be managed to obtain preprocessed data to be managed; Extracting types and features of the pre-processed data to be managed to obtain type feature information; Perform sensitive attribute verification on the pre-processed data to be governed to obtain corresponding data sensitive attribute identifiers, and associate and match the business management nodes of the data management network organization based on the data sensitive attribute identifiers to determine an initial data governance plan; Matching corresponding data governance standards according to the type feature information of the pre-processed data to be governed and the data sensitive attribute identifier, and optimizing and adjusting the initial data governance plan to obtain a target data governance plan; Performing data governance on the pre-processed data to be governed according to the target data governance solution; The data management network mechanism is configured with different levels for the plurality of business management nodes, and the plurality of business management nodes of different levels form a business flow corresponding to the existing business system of the enterprise; After the sensitive attribute verification is performed on the pre-processed data to be governed to obtain the corresponding data sensitive attribute identifier, and the business management node of the data management network organization is associated and matched based on the data sensitive attribute identifier, and an initial data governance solution is determined, the method includes: Obtaining the data acquisition time in the data information of the pre-processed data to be managed; within a preset time period, creating multiple data transmission compression intervals according to the data acquisition time of multiple groups of the pre-processed data to be managed and the corresponding data sensitive attribute identifiers of the service management nodes of the same level of the current processing level; According to the plurality of data transmission compression intervals, the plurality of groups of the pre-processed data to be managed are combined and compressed to obtain a plurality of compression sets of the pre-processed data to be managed; Delivering the plurality of compressed sets of pre-processed data to be managed to a service management node at the next level; The service management node includes a basic-level service management node; when the data sensitivity identifier of the basic-level service management node meets a preset optimization processing identifier condition; Within the preset time period, multiple data transmission compression intervals are created according to the data acquisition time of multiple groups of the pre-processed data to be managed and the corresponding data sensitive attribute identifiers of the service management nodes of the same level of the current processing level, including: According to the data acquisition time of multiple groups of the pre-processed data to be managed by the service management nodes of the same level of the current processing level, the pre-processed data to be managed currently being processed and the created data transmission compression interval are determined; the created data transmission compression intervals are all associated with the data to be compressed packages; the created data transmission compression intervals include expandable data transmission compression intervals; Determine whether the pre-processed data to be managed currently is within the created data transmission compression interval; If the data acquisition time of the currently processed pre-processed data to be managed is within the created data transmission compression interval, the currently processed pre-processed data to be managed is merged into the corresponding data to be compressed package; If the data acquisition time of the currently processed pre-processed data to be managed is not within the created data transmission compression interval, but within the expandable interval of the created data transmission compression interval, the compression time interval range and the expandable interval time range of the created data transmission compression interval are updated based on the data acquisition time of the currently processed pre-processed data to be managed, and the currently processed pre-processed data to be managed is merged into the updated data transmission compression interval; If the data acquisition time of the preprocessed data to be managed currently being processed is not within the created data transmission compression interval and is not within the expandable interval of the created data transmission compression interval, a new data transmission compression interval is created based on the data acquisition time of the preprocessed data to be managed currently being processed, and the preprocessed data to be managed currently being processed is merged into the new data transmission compression interval.

2. According to the data governance method based on big data according to claim 1, it is characterized in that: The step of determining whether the currently processed pre-processed data to be managed is within the created data transmission compression interval includes: The expandable interval of the data transmission compression interval expands the data transmission compression interval into an increase-time transmission compression interval or a decrease-time transmission compression interval according to a preset dynamic compression duration, wherein the time range of the increase-time transmission compression interval is calculated based on the data acquisition time of the next pre-processed data to be processed and managed; the time range of the decrease-time transmission compression interval is calculated based on the data acquisition time of the next pre-processed data to be processed and managed; According to the data acquisition time of the next pre-processed data to be processed, the corresponding data transmission compression interval is determined to be an increase-time transmission compression interval or a decrease-time transmission compression interval; the maximum value of the maximum compression time of the increase-time transmission compression interval and the decrease-time transmission compression interval satisfies the sum of the minimum compression time and the dynamic compression duration; When it is determined that the creation time of a certain data transmission compression interval meets the preset timeout period, the data transmission compression interval is compressed and output.

3. According to the data governance method based on big data in claim 1, it is characterized in that: The sensitive attribute verification of the pre-processed data to be governed is performed to obtain a corresponding data sensitive attribute identifier, and based on the data sensitive attribute identifier, the business management node of the data management network organization is associated and matched to determine an initial data governance solution, including: Creating and associating a data index for the pre-processed data to be governed, matching a sensitive data index with a preset sensitive attribute identification rule based on the data index, and generating multiple sensitive data based on the successfully matched sensitive data index, and obtaining a sensitive attribute label for each sensitive data, wherein the sensitive attribute label includes a label type for characterizing a data type and a label level for characterizing a sensitivity level; Determine, from the data governance solution groups corresponding to the most label types in the plurality of sensitive data, a plurality of pending data governance solutions that match the highest sensitivity level of the label level; The business process step information of the pre-processed data to be governed is determined by the acquired data governance process configuration information, and the business management nodes connected to the data management network organization are matched according to the business process step information, and an initial data governance plan is obtained by screening from the several pending data governance plans.

4. According to the data governance method based on big data in claim 1, it is characterized in that: The step of obtaining the initial data to be governed and performing standardized preprocessing on the initial data to be governed to obtain preprocessed data to be governed includes: In a preset data quality standard management model, the initial data to be managed is preprocessed for data quality management to obtain preprocessed data to be managed; the data quality management includes data cleaning, data correction, data anomaly verification and detection; The preset data quality specification management model is constructed based on the random forest algorithm, the isolation forest algorithm and the data error correction algorithm, and is obtained through deep machine learning and training.

5. A data governance system based on big data, characterized in that: A data governance method based on big data as described in any one of claims 1 to 4, wherein the system comprises: A governance standard and management framework formulation module is used to construct enterprise data governance standards according to the data governance processes of various relevant business nodes of the enterprise's existing business system, and to construct multiple business management nodes corresponding to the enterprise's existing business system, and the multiple business management nodes form a data management network organization; The preprocessing and information extraction module is used to obtain initial data to be managed, perform standardized preprocessing on the initial data to be managed, and obtain preprocessed data to be managed; perform type and feature extraction on the preprocessed data to be managed, and obtain type feature information; A data attribute association and matching module is used to perform sensitive attribute verification on the pre-processed data to be governed, obtain corresponding data sensitive attribute identifiers, and associate and match the business management nodes of the data management network organization based on the data sensitive attribute identifiers to determine an initial data governance solution; A data governance scheme optimization module, used to match the corresponding data governance standards according to the type feature information of the pre-processed data to be governed and the data sensitive attribute identifier, and optimize and adjust the initial data governance scheme to obtain a target data governance scheme; A data governance execution module, used for performing data governance on the pre-processed data to be governed according to the target data governance solution; After the sensitive attribute verification is performed on the pre-processed data to be governed to obtain the corresponding data sensitive attribute identifier, and the business management node of the data management network organization is associated and matched based on the data sensitive attribute identifier, and an initial data governance solution is determined, the method includes: Obtaining the data acquisition time in the data information of the pre-processed data to be managed; within a preset time period, creating multiple data transmission compression intervals according to the data acquisition time of multiple groups of the pre-processed data to be managed and the corresponding data sensitive attribute identifiers of the service management nodes of the same level of the current processing level; According to the plurality of data transmission compression intervals, the plurality of groups of the pre-processed data to be managed are combined and compressed to obtain a plurality of compression sets of the pre-processed data to be managed; Delivering the plurality of compressed sets of pre-processed data to be managed to a service management node at the next level; The service management node includes a basic-level service management node; when the data sensitivity identifier of the basic-level service management node meets a preset optimization processing identifier condition; Within the preset time period, multiple data transmission compression intervals are created according to the data acquisition time of multiple groups of the pre-processed data to be managed and the corresponding data sensitive attribute identifiers of the service management nodes of the same level of the current processing level, including: According to the data acquisition time of multiple groups of the pre-processed data to be managed by the service management nodes of the same level of the current processing level, the pre-processed data to be managed currently being processed and the created data transmission compression interval are determined; the created data transmission compression intervals are all associated with the data to be compressed packages; the created data transmission compression intervals include expandable data transmission compression intervals; Determine whether the pre-processed data to be managed currently is within the created data transmission compression interval; If the data acquisition time of the currently processed pre-processed data to be managed is within the created data transmission compression interval, the currently processed pre-processed data to be managed is merged into the corresponding data to be compressed package; If the data acquisition time of the currently processed pre-processed data to be managed is not within the created data transmission compression interval, but within the expandable interval of the created data transmission compression interval, the compression time interval range and the expandable interval time range of the created data transmission compression interval are updated based on the data acquisition time of the currently processed pre-processed data to be managed, and the currently processed pre-processed data to be managed is merged into the updated data transmission compression interval; If the data acquisition time of the preprocessed data to be managed currently being processed is not within the created data transmission compression interval and is not within the expandable interval of the created data transmission compression interval, a new data transmission compression interval is created based on the data acquisition time of the preprocessed data to be managed currently being processed, and the preprocessed data to be managed currently being processed is merged into the new data transmission compression interval.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of a data governance method based on big data as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a data governance method based on big data as described in any one of claims 1 to 4 are implemented.

Images