A big data management system and method

Abstract

The present invention relates to the field of big data technology, and discloses a big data governance system and method. The modules of the present invention realize collaboration based on natural language interaction through artificial intelligence technology, improve the intelligence level of multiple links from data control to data processing, change the traditional mode of relying on manual setting of a large number of parameters and rules, and reduce the time cost of manual operation. Therefore, by implementing the present invention, by integrating data access, storage, control, processing and intelligent auxiliary modules into one system, the automation and integration of the data governance process are realized.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a big data governance system and method. Background Art

[0002] A general data governance system is a comprehensive system that provides data management (development) personnel with a visual operating interface, supports functions such as data collection, processing, task scheduling, and data governance control. Data management (development) personnel carry out various data governance activities through the coordinated use of the functions of each module.

[0003] However, although the existing data governance system provides data governance personnel with an operating console to perform process management such as standard management, data modeling, and data quality exploration, in actual management and control work, the governance system only serves as a relatively convenient tool for managing these activities. The main process still relies on people to complete, which is very time-consuming and labor-intensive, further making it difficult to rationally reduce data governance costs. Summary of the invention

[0004] In view of this, the present invention provides a big data governance system and method to solve the problem that in actual management and control work, the existing data governance system is only a relatively convenient tool for managing these activities, and the main process still relies on people to complete, which is very time-consuming and labor-intensive, further making it difficult to reasonably reduce data governance costs.

[0005] In a first aspect, the present invention provides a big data governance system, the system comprising: a data access module, a data storage module, a data management module, a data processing module and an intelligent auxiliary module;

[0006] The data access module is used to obtain multi-source data sets and send the multi-source data sets to the data storage module for storage; the data control module is used to obtain the metadata set, and call the intelligent auxiliary module based on the metadata set and generate the control standards; the data control module is also used to send data call instructions to the data processing module based on the control standards; the data processing module is used to call the target data set from the multi-source data set of the data storage module based on the data call instructions, and send the target data set to the data control module; the data control module is also used to determine the big data governance results based on the control standards and the target data set.

[0007] The big data governance system provided by the present invention realizes collaboration based on natural language interaction between modules through artificial intelligence technology, improves the intelligence level of multiple links from data management to data processing, changes the traditional mode of relying on manual setting of a large number of parameters and rules, and reduces the time cost of manual operation. Therefore, by implementing the present invention, by integrating data access, storage, management, processing and intelligent auxiliary modules into one system, the automation and integration of the data governance process is realized.

[0008] In an optional embodiment, the management and control standards are one or more of data management standards, data quality detection standards and intelligent standard models; the data management and control module includes: an intelligent data standard management unit, an intelligent data quality detection unit and an intelligent sub-warehouse modeling management unit; the intelligent data standard management unit is used to call the intelligent auxiliary module to generate data management standards based on the metadata set; the intelligent data quality detection unit is used to call the intelligent auxiliary module to generate data quality detection standards based on the metadata set; the intelligent sub-warehouse modeling management unit is used to call the intelligent auxiliary module to generate an intelligent standard model based on the metadata set.

[0009] The big data governance system provided by the present invention refines the control standards into data management standards, data quality exploration standards and intelligent standard models, which can correspond to data governance needs at different levels. Furthermore, based on the metadata data set, the intelligent auxiliary module is called to generate these standards, which utilizes the advantages of the NLP big model in understanding and parsing natural language input, so that the massive and complex emergency data has unified and detailed standards to follow, avoiding problems such as data confusion and inconsistent formats, and improving the overall standardization. Furthermore, with the help of the intelligent auxiliary module, multi-dimensional control standards are quickly generated through natural language interaction, which greatly reduces the time of manual input, reduces the manpower and time costs, and enhances the efficiency of control.

[0010] In an optional implementation, the data processing module is specifically used for:

[0011] Based on the data calling instruction, the intelligent auxiliary module is called and a target script is generated, and the target data set is called from the multi-source data sets in the data storage module by using the target script, and the target data set is sent to the data control module.

[0012] The big data governance system provided by the present invention realizes automated operation by generating target scripts based on data call instructions, and combines NLP technology to understand the connotation of instructions, so as to accurately locate the target data set in multi-source data, get rid of the problem of tedious manual comparison and screening, and improve the flexibility of data processing module to process complex and diverse data. Furthermore, by automatically generating scripts, manual writing and debugging errors are reduced and the preparation cycle is shortened.

[0013] In an optional implementation, the system further includes: a task management and control module for managing and controlling target scripts in the data processing module.

[0014] The big data governance system provided by the present invention can control the operation of the target script in the data processing module through the task control module, and provide support for the acquisition of subsequent target data sets.

[0015] In an optional implementation, the task control module includes: a task management unit, a task flow management unit and a task monitoring unit;

[0016] The task management unit is used to manage the operation of the target script when there is one target script in the data processing module; the task flow management unit is used to schedule and manage the operation of multiple target scripts when there are multiple target scripts in the data processing module; the task monitoring unit is used to monitor the operation of the target script.

[0017] The big data governance system provided by the present invention can realize the full life cycle control of the operation of a single target script in the data processing module through the task management unit. Further, the task flow management unit focuses on the multi-target script scenario, and arranges and schedules the operation of multiple target scripts according to the logical sequence and data dependency between scripts, ensuring the orderly connection of multiple links and smooth data flow, avoiding data misuse and process blockage caused by chaotic dependencies. Furthermore, the task monitoring unit can track the operation of the target script in all directions and feedback the progress in real time, which helps operation and maintenance personnel to grasp the real-time status of the script, predict bottlenecks and failures in advance, and expand resources or optimize script logic in time to ensure that data processing is not interrupted and promoted efficiently.

[0018] In an optional implementation, the data control module further includes:

[0019] The metadata management unit is used to record and manage descriptive information of data; the data asset map unit is used to display the results of big data governance.

[0020] The big data governance system provided by the present invention can sort out and record information such as data source, structure, and meaning through the metadata management unit. Furthermore, the big data governance results can be intuitively displayed through the data asset map unit, so that users can gain insight into the scale and growth trend of data assets at a glance. Furthermore, it is convenient for business personnel to locate the required data, which improves the convenience and coordination of data use.

[0021] In an optional implementation, the system further includes: a data security module, the data security module including a rights management unit and a unified authentication unit;

[0022] The authority management unit is used to receive the multi-source data set sent by the data access module and manage the user access rights based on the multi-source data set; the unified authentication unit is used to perform unified authentication on the user access information.

[0023] The big data governance system provided by the present invention can manage user access rights through the rights management unit, thereby ensuring that sensitive data is only open to authorized users. Furthermore, the user access information can be uniformly authenticated through the unified authentication unit, realizing a unified authentication service, simplifying the user login process, saving time and improving the user experience, reducing the problem of work delays caused by cumbersome authentication, and ensuring the timeliness and consistency of emergency data interaction.

[0024] In an optional implementation, the data security module further includes: a data classification unit, a data encryption and decryption unit, and a data desensitization unit;

[0025] A data security module is used to receive a multi-source data set sent by a data access module or a big data governance result sent by a data control module; a data classification unit is used to perform classification processing on the multi-source data set or to perform classification processing on multiple data corresponding to the big data governance result; a data encryption and decryption unit is used to perform encryption / decryption processing on the multi-source data set or to perform encryption / decryption processing on multiple data corresponding to the big data governance result; a data desensitization unit is used to perform desensitization processing on the multi-source data set or to perform desensitization processing on multiple data corresponding to the big data governance result.

[0026] The big data governance system provided by the present invention can classify and mark data of different importance through the data classification unit, so as to facilitate the implementation of targeted protection measures. Furthermore, the data managed in the system can be encrypted and desensitized through the data encryption and decryption unit, which can effectively prevent the risk of leakage of sensitive information when it is shared or publicly used.

[0027] In an optional implementation, the data processing module includes:

[0028] The data export unit is used to call the multi-source data set from the data storage module and export the multi-source data set to the target database.

[0029] The big data governance system provided by the present invention can export multi-source data sets to other third-party target databases through a data export unit, breaking the system's "data islands" and achieving efficient data sharing while maintaining data asset security.

[0030] In a second aspect, the present invention provides a big data governance method, which is used in the big data governance system of the first aspect or any corresponding embodiment thereof, and the method includes:

[0031] Obtain multi-source data sets and metadata sets; generate management and control standards based on the metadata sets; determine the target data set in the multi-source data sets based on the management and control standards; determine the big data governance results based on the management and control standards and the target data set.

[0032] The big data governance method provided by the present invention reduces operational differences and confusion, improves the standardization level of global emergency data governance, and realizes the automation and integration of data governance processes by executing the data governance process in the big data governance system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 is a structural block diagram of a big data governance system according to an embodiment of the present invention;

[0035] Figure 2 It is a structural block diagram of a big data governance system based on smart emergency response according to an embodiment of the present invention;

[0036] Figure 3 is a flowchart of a big data governance method according to an embodiment of the present invention;

[0037] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0039] An embodiment of the present invention provides a big data governance system, which integrates data access, storage, management, processing and intelligent auxiliary modules into one system to achieve automation and integration of the data governance process.

[0040] In this embodiment, a big data management system is provided. Figure 1As shown, the big data governance system 1 includes: data access module 11, data storage module 12, data control module 13, data processing module 14, intelligent auxiliary module 15, task control module 16 and data security module 17. Among them, the intelligent auxiliary module 15 is a background support module that is invisible to system users.

[0041] Optionally, the data access module 11 is used to obtain a multi-source data set and send the multi-source data set to the data storage module 12 for storage.

[0042] The data access module 11 may be responsible for aggregating and integrating various data from different units and departments (ie, data sources), and then transferring the data to the data storage module 12 for subsequent storage and processing.

[0043] Specifically, the data access module 11 provides a visual operation interface, through which the relevant information of the data source can be viewed, and then the type of data source to be accessed can be identified. For example, the data source may be the DAMO database used by a cooperative unit, or real-time updated data provided by another department through an API interface, or historical emergency record documents in the form of local files, etc. These data source types are rich and varied, covering common forms such as relational databases (such as DAMO database, Mysql database, Oracle, MongoDB, Gauss200, etc.), data warehouses (Hive, Hbase), API interfaces, local files, kafka message middleware, etc.

[0044] Furthermore, after determining the data source type, the data access module 11 can encapsulate various data access operations into fixed components, and further, can select corresponding access components according to the identified data source type. For example, if the data source is an Oracle database, then select the access component for the Oracle database; if the data is obtained through an API interface, select the corresponding API access component.

[0045] Furthermore, after selecting the access component, you can configure key parameters to access data from the source end to the target end (i.e. from various units and departments to the Emergency Management Bureau). Taking the configuration of the Oracle database access component as an example, you need to fill in the database connection address, port number, user name, password and other key parameter information to ensure that you can establish a correct connection with the data source and successfully obtain data.

[0046] Furthermore, after receiving the multi-source data set sent by the data access module 11, the data storage module 12 can use the big data storage components (HDFS+Hive, Hbase) it adopts for data storage.

[0047] For example, some structured emergency material information data may be stored in a certain table structure through Hive to facilitate subsequent query and analysis; and some semi-structured or unstructured data (such as multimedia emergency materials such as on-site pictures and videos, and some log files, etc.) can be stored with the help of Hbase to cope with its flexible data format characteristics, ensuring that these multi-source data can be stored in the system in an orderly and secure manner, providing a reliable data foundation for subsequent emergency data governance work (such as data processing, management and control, etc.).

[0048] Optionally, the data control module 13 is used to obtain a metadata set, and based on the metadata set, call the intelligent assistance module 15 and generate a control standard.

[0049] The data control module 13 includes an intelligent data standard management unit 131 , an intelligent data quality detection unit 132 and an intelligent warehouse modeling management unit 133 .

[0050] Furthermore, the control standard may be one or more of a data management standard, a data quality detection standard, and an intelligent standard model, which may be determined based on actual control needs.

[0051] Furthermore, the metadata set may include descriptive information of various types of data in the big data governance system 1, such as data source, structure, meaning, etc.

[0052] Furthermore, the intelligent auxiliary module 15 has the following capabilities:

[0053] (1) Assisted script development capability. It provides the ability to intelligently generate data processing scripts, supports inputting natural language to describe the logic of the data processing script, the table name and field name to be processed by the script, and then automatically generates the script. This capability is integrated with the data processing module, allowing developers to quickly develop data processing scripts.

[0054] (2) Assisting data management and control capabilities. Providing auxiliary capabilities that can assist developers in improving efficiency in important governance and control activities.

[0055] Optionally, the intelligent data standard management unit 131 is used to call the intelligent auxiliary module 15 to generate a data management standard based on the metadata set.

[0056] Specifically, the intelligent data standard management unit 131 may call upon the capability of the intelligent auxiliary module 15 to assist in data standard sorting.

[0057] Among them, the intelligent auxiliary module 15's ability to assist in data standard sorting supports parsing of input documents (such as existing ministerial emergency data standard files, provincial and ministerial emergency data standard files, municipal data standard files, municipal emergency management bureau data standard files, data source unit data interpretation files, etc.).

[0058] Furthermore, through this capability, after the relevant emergency data standard files are uploaded to the file upload interface provided by the intelligent assistance module 15, the file contents can be parsed and the various data standard regulations can be broken down. These regulations can cover the Chinese name and English name of the data table, the Chinese name and English name of the data field, the data type, the field length, the constraints, the Chinese pinyin abbreviations, the definition description, the field code enumeration value and other detailed contents.

[0059] Furthermore, by calling the intelligent auxiliary module 15, the intelligent data standard management unit 131 can also compare the published standards with the metadata managed by the system to filter out fields for which no standards have been established.

[0060] For example, if there are 5,000 fields in the data governance system, and the standard management module has published 2,000 field standards, by comparison, it is possible to screen out the 3,000 fields that have no standards, and further improve the comprehensiveness of data management standards.

[0061] Finally, based on the above analysis of the data standard provisions and possible comparison results, the intelligent data standard management unit 131 can generate corresponding data management standards. Furthermore, the standard clarifies the data specification requirements in various aspects, provides clear guidelines for the subsequent data management, and can also perform manual standard formulation, publication, update and other operations through the visual maintenance interface provided for standard formulation, publication, update and maintenance, thereby realizing flexible control of data management standards.

[0062] Optionally, the intelligent data quality detection unit 132 is used to call the intelligent auxiliary module 15 to generate a data quality detection standard based on the metadata set.

[0063] Specifically, the intelligent data standard management unit 131 may call upon the capability of the intelligent assistance module 15 to assist in data quality auditing.

[0064] Furthermore, developers can input a natural language description of the data quality problem through the intelligent assistance interface provided by the intelligent assistance module 15, such as a description such as "check whether there are negative numbers in the quantity field in the emergency material inventory data".

[0065] Furthermore, the intelligent assistance module 15 can design quality audit rules according to the input description, and generate corresponding scripts according to the rules. It also supports testing and manual modification of the generated scripts to ensure the accuracy and effectiveness of the rules and scripts.

[0066] Furthermore, based on the generated quality audit rules and related scripts, the intelligent data standard management unit 131 can generate corresponding quality exploration standards. This standard clarifies the rules and methods that need to be followed in data quality inspection. Developers can also manually design data quality inspection rules and configure data quality audit tasks through the visual operation interface provided by itself, and combine them with the standards generated by intelligent assistance to achieve more comprehensive data quality exploration standards that are more in line with actual needs.

[0067] Optionally, the intelligent warehouse modeling management unit 133 is used to call the intelligent auxiliary module 15 to generate an intelligent standard model based on the metadata set.

[0068] Specifically, the intelligent data standard management unit 131 may call upon the capability of the intelligent auxiliary module 15 to assist in data mode design.

[0069] Furthermore, developers can input modeling requirements through the intelligent assistance interface provided by the intelligent assistance module 15, which is generally a description of the conceptual model (for example, "constructing a hierarchical storage model for emergency command center data, divided according to event type and time dimension") or images and other content.

[0070] Furthermore, the intelligent auxiliary module 15 can automatically perform logical model design and physical model design according to the input description or image, and generate corresponding model content.

[0071] Furthermore, the generated logical model and physical model will be matched with the published data standards to ensure that the generated table model meets the existing standard requirements, ensure the standardization of the model and its compatibility with the entire data governance system.

[0072] Furthermore, through the above process, the intelligent warehouse modeling management unit 133 can generate a corresponding intelligent standard model, which can be used to guide the architectural design of data in storage and other aspects, and developers can also perform manual model creation, management, storage, version control and publishing operations through the visual operation interface provided by it, thereby further improving the data warehouse modeling management work.

[0073] Therefore, through the intelligent data standard management unit 131, the intelligent data quality exploration unit 132 and the intelligent warehouse modeling management unit 133, the data control module 13 can generate control standards covering different dimensions such as data management standards, data quality exploration standards and intelligent standard models based on the metadata set with the help of the capabilities of the intelligent auxiliary module 15, meeting the diverse data control needs in the emergency big data governance system. At the same time, further, with the help of the intelligent auxiliary module, multi-dimensional control standards can be quickly generated through natural language interaction, which greatly reduces the time of manual input, reduces the manpower and time costs, and enhances the control efficiency.

[0074] Optionally, the data control module 13 is also used to send data call instructions to the data processing module 14 based on the control standards.

[0075] Specifically, the corresponding data requirements can be determined according to the control standards:

[0076] (1) The data management standard specifies in detail the requirements for data tables and data fields. For example, it specifies that in the emergency material data, the material name field must adopt a unified Chinese naming format, and the data type of the material quantity field should be an integer. Based on these standard requirements, when it is necessary to further process the emergency material data that meets specific management standards (such as counting the total amount of different types of materials, etc.), the data control module 13 can determine which data tables and corresponding fields to call based on these standard details, and sort out the corresponding data requirements list.

[0077] (2) The data quality detection standard defines the rules and requirements for data quality inspection. For example, it is stipulated that when checking the information data of emergency rescue personnel, it is necessary to ensure that the value of the age field is within a reasonable range (such as 18-60 years old), and the telephone number field must comply with specific format specifications. If it is found that some data may not meet the quality standards, it is necessary to process these data with quality questions (such as correcting format errors, supplementing missing values, etc.). The data control module 13 can determine the data set that needs to be called based on the data range and problem type involved in the quality detection standard, so that the data processing module 14 can carry out targeted quality repair and other processing operations.

[0078] (3) The intelligent standard model plans the storage architecture and logical relationships of the data. For example, a data sub-warehouse model is constructed according to the type of emergency event (such as fire, earthquake, etc.) and the time dimension. The data of different types of emergency events are stored in different logical areas and have corresponding associations. When the data is to be processed and analyzed based on this model (such as analyzing the changes in the frequency of occurrence of various emergency events in different time periods, etc.), the data management and control module 13 clarifies which sub-warehouse areas need to call which related data based on the data storage location, association relationship and other information defined in the intelligent standard model, thereby determining the accurate data call range for the data processing module 14.

[0079] Furthermore, according to the data requirements specified from the perspectives of the above-mentioned different control standards, the data control module 13 can construct specific data call instructions.

[0080] The instruction content may include key elements such as the name of the data table to be called, specific data field information (which can be a single field or a combination of multiple fields), data screening conditions (such as time range limitations, specific category limitations, etc.), and the source location of the data (such as which specific table space in the Hive database is stored).

[0081] Taking the emergency material inventory data call as an example, the instruction may be "call the 'Material Name', 'Material Quantity' and 'Entry Time' fields from the 'Material Inventory Table' stored in the Hive database, and filter the data whose entry time is within the past month and whose material category is 'Protective Equipment'."

[0082] Further, the constructed data calling instruction is sent to the corresponding data processing module 14.

[0083] Optionally, the data processing module 14 is used to call a target data set from the multi-source data sets of the data storage module 12 based on the data calling instruction, and send the target data set to the data management and control module 13 .

[0084] Among them, according to the number of management and control standards, the number of data call instructions can be one or more. Further, the number of target scripts corresponds to the number of data call instructions, which can be one or more.

[0085] Specifically, the data processing module 14 can call the intelligent assistance module 15 according to the received data calling instruction and generate a target script, and use the target script to call the target data set from the multi-source data sets of the data storage module 12, and send the target data set to the data management module 13.

[0086] First, after receiving the data call instruction, the data processing module 14 can parse the data call instruction and extract the key information contained therein, such as the name of the data table to be called, specific data fields, data screening conditions, and data source location, so as to clarify the scope of subsequent data processing tasks.

[0087] Furthermore, the parsed data call instruction content can be converted into a script development requirement description that meets the input requirements of the intelligent auxiliary module 15. For example, if the data call instruction requires "obtaining the name, quantity and corresponding warehouse number field data of protective equipment materials that have been put into storage within the past month from the emergency material table of the Hive database", the corresponding script development requirement description may be "writing a query script, using the Hive language (assuming that the system uses Hive by default), filtering out records with a time field (storage time) within the past month and a material category of protective equipment from the emergency material table, and extracting data from the three fields of material name, material quantity and warehouse number".

[0088] Secondly, the script development capability of the intelligent auxiliary module integrated in the data processing module 14 can be used to input the above-determined script development requirement description content in the script intelligent auxiliary development interface provided by it.

[0089] Furthermore, the intelligent assistance module 15 can understand and analyze the input requirement description based on its built-in natural language understanding and script generation algorithm (using NLP big model technology, etc.), and then automatically generate a corresponding target script.

[0090] Further, the generated target script can be tested and run in the test environment provided by the data processing module 14 to check whether the script can be executed correctly and obtain the expected data. If it is found during the test that the script has syntax errors, logical errors, or the data obtained does not meet the requirements, the developer can use the manual modification function provided by the script intelligent auxiliary development interface to adjust and improve the script. For example, if it is found that the date format in the screening condition is inconsistent with the format stored in the actual table, resulting in inaccurate data screening, the date processing part in the script can be manually modified so that it can accurately filter out data within the past month.

[0091] Then, the data processing module 14 can establish a connection with the data storage module 12 through the corresponding database connection driver (such as the JDBC driver for the Hive database, etc.) based on the data source location specified in the target script (such as database name, table space and other information), to ensure access to the multi-source data sets stored therein.

[0092] Furthermore, the generated and tested (or modified and improved) target script can be used to perform corresponding data query and extraction operations on the connection with the data storage module 12 to obtain the target data set that meets the script screening conditions and field requirements from the multi-source data set. For example, after executing the Hive SQL script generated above, the specific data content such as the name, quantity and warehouse number of the relevant materials that meet the material category and storage time conditions will be obtained from the storage area corresponding to the emergency material table to form the target data set.

[0093] Furthermore, the data processing module 14 may send the obtained target data set to the data management and control module 13 .

[0094] In an optional implementation, before sending the target data set, the data processing module 14 may also organize and package the acquired target data set according to the requirements of the data control module 13 or the data transmission format agreed upon by the system. For example, if the target data set is in the form of multiple records, it may be organized into a specific data structure (such as a JSON format array, etc.) to facilitate subsequent transmission and parsing by the data control module 13.

[0095] By generating target scripts based on data call instructions to achieve automated operations and combining NLP technology to understand the meaning of instructions, the target data set in the multi-source data set can be accurately located, getting rid of the problem of tedious manual comparison and screening, and improving the flexibility of the data processing module to handle complex and diverse data. Furthermore, by automatically generating scripts, manual writing and debugging errors are reduced and the preparation cycle is shortened.

[0096] Optionally, the task control module 16 is used to control the target script in the data processing module 14 .

[0097] The task control module 16 may include: a task management unit 161 , a task flow management unit 162 and a task monitoring unit 163 .

[0098] Specifically, there may be one or more target scripts in the data processing module 14 , and therefore, the task management and control module 16 manages different numbers of target scripts differently.

[0099] Furthermore, the task control module 16 provides developers with a visual operation interface, through which developers can clearly and intuitively conduct all-round control over the operation of various tasks (target scripts) in the data processing module 14, to ensure that the script can accurately and efficiently obtain the required target data set from the data storage module 12. Moreover, the task control module 16 supports a variety of flexible scheduling methods such as timed scheduling and event triggering. Developers can choose appropriate scheduling strategies based on actual business needs and data processing priorities, so as to ensure that the target scripts in the data processing module 14 can be efficiently executed, thereby ensuring the orderly operation of the entire emergency big data governance system.

[0100] Optionally, the task management unit 161 is used to manage the execution of a target script when there is a target script in the data processing module.

[0101] Specifically, when there is a target script in the data processing module 14, the task management unit 161 is first responsible for the startup management of the script.

[0102] Furthermore, developers can use the visual operation interface of the task management module 16 to find the corresponding data processing task (i.e., the task related to the target script), click the start button (there is an operation button clearly marked on the interface), or follow the pre-set scheduled time (for example, set the script to start at 9 am every day to perform specific data processing operations).

[0103] Furthermore, after receiving the start instruction, the task management unit 161 can interact with the data processing module 14 and notify it to start executing the target script.

[0104] Optionally, the task flow management unit 162 is used to schedule and manage the execution of multiple target scripts when there are multiple target scripts in the data processing module.

[0105] Specifically, when there are multiple target scripts in the data processing module 14, the task flow management unit 162 first determines the function and role of each target script.

[0106] For example, some scripts are responsible for extracting raw data from different data sources (multi-source data sets in the data storage module 12), some scripts clean and convert the extracted data, and some scripts perform aggregation analysis and other operations based on the cleaned data. By analyzing the input and output requirements and business logic of these scripts, the task flow management unit 162 can sort out the dependencies between the scripts, clarify which scripts must be started after other scripts are executed, which scripts can be executed in parallel, and so on.

[0107] For example, if the input data of a data cleaning script depends on the output of a data extraction script, then the data extraction script is a prerequisite script of the data cleaning script, and there is a sequence relationship between them. For cleaning scripts of different types of data (such as emergency material data and emergency personnel data), if they have no data dependencies on each other, they can be executed in parallel to improve overall execution efficiency.

[0108] Furthermore, based on the sorted dependency relationships, the task flow management unit 162 can schedule and manage the execution of multiple target scripts.

[0109] Optionally, the task monitoring unit 163 is used to monitor the running status of the target script.

[0110] Specifically, the task monitoring unit 163 can collect various key data information in real time during the process of the target script calling the target data from the multi-source data set.

[0111] Furthermore, these data information can cover multiple aspects, such as script execution progress data (including the number of data rows read, the proportion of target data acquired, how long it is expected to complete the acquisition of all target data, etc.), system resource usage (such as the CPU usage, memory usage, disk I / O read and write speed, and network bandwidth usage occupied by the current script running, which are used to measure the degree of system resource consumption by the script), data quality related information (for example, whether there are format errors in the read data, whether the data integrity meets the requirements, whether there is duplicate data, etc., to ensure the quality of the acquired target data is reliable) and data interaction between the script and the data storage module 12 (for example, whether the data reading rate is stable, whether there are abnormal situations such as connection interruption or timeout, to ensure that the data can be smoothly transferred from the storage module to the script execution environment) and other contents.

[0112] In an optional implementation, during the process of continuously collecting monitoring data, the task monitoring unit 163 may also use preset rules and algorithms to perform real-time analysis on the data to detect whether there are any abnormal situations.

[0113] For example, a threshold is set to determine anomalies. When the CPU usage of the script continues to exceed 90% and is maintained for a certain period of time (this may mean that the script has an infinite loop caused by complex calculations or unreasonable resource allocation, resulting in excessive consumption of CPU resources), or the data reading rate suddenly drops to almost zero (it may be due to a database connection failure, an anomaly at the data source, or a network problem affecting data transmission), or the proportion of format error data in the read data exceeds a pre-set reasonable range (indicating that there are serious problems with data quality, which may affect subsequent data processing and analysis), etc., the task monitoring unit 163 can determine that the corresponding script execution has an abnormality.

[0114] Furthermore, once an abnormal situation is detected, the task monitoring unit 163 can immediately pop up a striking warning prompt box on the visual operation interface, clearly displaying the specific information of the abnormality, such as the name of the script where the abnormality occurs, the type of abnormality (whether it is a resource abnormality, data quality abnormality or data interaction abnormality, etc.), the time when the abnormality occurred and the scope of possible impact, so that developers can quickly locate the problem.

[0115] Optionally, the data management module 13 also includes a metadata management unit 134 and a data asset map unit 135 .

[0116] Specifically, the metadata management unit 134 can be used to record and manage descriptive information of the data, such as data source, structure, meaning, etc.:

[0117] (1) Data source information record:

[0118] Data is gathered from multiple units and departments (data sources), accessed through the data access module 11 and stored in the data storage module 12. Whenever new data is accessed, the metadata management unit 134 starts working to obtain relevant information about its source. For example, for a batch of fire accident scene data from the fire department, the metadata management unit 134 can record that it is provided by the "XX City Fire Rescue Detachment", accessed through the "Fire Business Database Interface" in the data access module, and the access time is "XX year XX month XX day XX hour XX minute" and other detailed source information to ensure that the source of each piece of data is clear and traceable. For subsequent updated or replaced data sources (for example, the fire department changes the data reporting system and the data source interface changes), the metadata management unit 134 can follow up and update the corresponding data source records in a timely manner to ensure the timeliness and accuracy of the information.

[0119] (2) Data structure information management:

[0120] For various types of data stored in the data storage module 12 (whether structured data stored in relational databases such as MySQL and Oracle, or semi-structured data such as log files, and unstructured data such as pictures and videos), the metadata management unit 134 can analyze their structural characteristics.

[0121] Taking the material data table in the emergency material management system as an example, the metadata management unit 134 can record the various fields contained in the table, such as the "material number" field (its data type is a string, the length is set to 10 bits, and it uniquely identifies each type of material), "material name" (the data type is a string, the length is set according to the actual longest name, etc.), "inventory quantity" (the data type is an integer), etc. At the same time, it will also record the association between the fields (such as "material number" and other related data tables through foreign key associations, etc.), forming a complete description of the data structure. In addition, as the business develops, if the structure of the material data table changes (such as adding a "material shelf life" field), the metadata management unit 134 can perceive and update the corresponding structural information record in real time, so that the system always has the accurate structure of the data.

[0122] (3) Information sorting of data meaning:

[0123] For each item of data, the metadata management unit 134 can clarify its specific meaning in the emergency management business scenario. For example, in the emergency event record data table, the "location of the incident" field does not only mean a simple description of the geographical location, but also includes the administrative division, specific coordinates, and other information that are of great guiding significance to emergency rescue, all of which will be recorded in detail. At the same time, if the meaning of certain data is redefined during the adjustment of the emergency management business process (for example, the classification standard of the "emergency response level" is changed), the metadata management unit 134 can coordinate the relevant modules, uniformly update the interpretation of the data meaning, and make synchronous modifications in its own records to ensure consistency in the understanding of the data meaning throughout the system.

[0124] Furthermore, the metadata management unit 134 may also provide a visual operation interface for metadata query, browsing, editing, and blood relationship construction and display.

[0125] Furthermore, the data asset map unit 135 can be used to display big data governance results.

[0126] Specifically, the data asset map unit 135 may be responsible for collecting and organizing key indicator information about system management data resources, which may specifically cover information such as the total number of tables, the total number of fields, the total number of data items, and the total data storage capacity.

[0127] Furthermore, the data asset map unit 135 can also compare the data statistics of the previous day, identify the newly added data tables, fields, and data records, and separately count the newly added data volume (number of newly added data items, newly added storage volume, etc.) of the day. The newly added situation of the day is also presented in an intuitive way (such as using different colors to distinguish the newly added data indicators of the day from the overall data indicators on the interface, or listing the newly added situation section separately for display), so that users can grasp the dynamic changes of data resources in a timely manner.

[0128] Furthermore, the data asset map unit 135 also supports functions such as data retrieval, data viewing / preview, and data lineage.

[0129] Optionally, the data security module 17 includes a permission management unit 171 , a unified authentication unit 172 , a data classification unit 173 , a data encryption and decryption unit 174 and a data desensitization unit 175 .

[0130] Optionally, the permission management unit 171 is used to receive the multi-source data set sent by the data access module 11, and manage the user access rights based on the multi-source data set.

[0131] Specifically, the rights management unit 171 may analyze the nature, source, potential sensitivity, and other information of the received multi-source data set.

[0132] Furthermore, the authority division principles can be determined based on factors such as emergency management business needs and the sensitivity of the data.

[0133] Furthermore, according to the determined permission division principle, the permission management unit 171 can allocate corresponding access rights to different user roles (such as administrators, ordinary staff, external cooperation personnel, the public, etc.) through a visual permission configuration interface (the interface can be operated by the system administrator). During the specific configuration, for each type of data (distinguished from the multi-source data set), the permissions of different levels such as read, write, modify, and delete are clearly set and associated with the corresponding user role or specific user account.

[0134] For example, on the permission configuration interface, select "Emergency Material Inventory Data Table", configure read-write permissions for the "Warehouse Manager" role, and configure only read-only permissions for the "General Query Personnel" role. Through such detailed configuration operations, it is ensured that different users can only access the corresponding data according to their granted permissions, thereby ensuring that sensitive data is only open to authorized users, preventing data leakage and unauthorized access.

[0135] Furthermore, the permission management unit 171 can also dynamically update and synchronize user access rights. For example, when a new data category is added to the multi-source data set (such as the addition of monitoring data of a special emergency device), the corresponding access rights need to be re-evaluated and configured; if a staff member is transferred from an ordinary executive position to a management position, the permission management unit 171 can increase the scope of his or her access rights accordingly, so that he or she can access more sensitive data. Through this dynamic management mechanism, the effectiveness and accuracy of permission management can always be maintained.

[0136] Optionally, the unified authentication unit 172 is used to perform unified authentication on user access information.

[0137] Specifically, when a user attempts to log in to the emergency big data governance system or access data resources within the system, he or she first needs to submit access information to the unified authentication unit 172, which may include information such as user name and password, and may also include some other identity authentication elements (such as dynamic verification codes, digital certificates, etc., depending on the security requirements and configuration of the system).

[0138] Furthermore, the unified authentication unit 172 supports docking with existing network authentication systems (such as government network authentication systems, enterprise internal single sign-on systems, etc.). Therefore, through standard interface protocols (such as OAuth, SAML and other common authentication protocols), a communication link with the existing network authentication system can be established to achieve collaborative work of data interaction and identity authentication.

[0139] Furthermore, after the integration is completed, users only need to enter their existing account and password and other regular authentication information in the existing network authentication system when accessing. The unified authentication unit 172 can pass this information to the corresponding existing network authentication system for verification, which greatly simplifies the user login process and improves the user experience. At the same time, it also leverages the security mechanism of the existing network authentication system to enhance the overall security of the system.

[0140] Optionally, the data security module is used to receive the multi-source data set sent by the data access module 11 or the big data governance results sent by the data management and control module 13.

[0141] Optionally, the data grading unit 173 is used to perform grading processing on multi-source data sets or to perform grading processing on multiple data corresponding to the big data governance results.

[0142] Specifically, the data classification unit 173 may formulate detailed data classification standards according to the importance and sensitivity of the data in the emergency management business and the degree of impact on personal privacy.

[0143] Furthermore, according to the established classification standards, the data classification unit 173 can classify and mark each data item in the received multi-source data set or big data governance result. For example, by adding corresponding classification labels to the metadata information of the data (such as adding a "data level" field in the field attributes of the database table, with values ​​such as "top secret", "confidential", "public", etc.), or setting a special mark in the logical structure of data storage to reflect the level of the data, it is ensured that each piece of data has a clear classification mark.

[0144] Optionally, the data encryption / decryption unit 174 is used to perform encryption / decryption processing on multi-source data sets, or to perform encryption / decryption processing on multiple data corresponding to big data governance results.

[0145] Specifically, the data encryption and decryption unit 174 can select a suitable encryption algorithm according to the characteristics and security requirements of the data: for massive emergency big data, if the encryption speed is required to be high and the data sensitivity is relatively not extremely high, a symmetric encryption algorithm (such as the AES algorithm) can be selected, which has fast encryption and decryption speeds and relatively simple key management; and for data involving important confidential information (such as emergency command keys, etc.), an asymmetric encryption algorithm (such as the RSA algorithm) can be used to ensure the confidentiality of the data by taking advantage of its high security and public key publicly distributable characteristics. After selecting the encryption algorithm, configure the corresponding algorithm parameters (such as key length, encryption mode, etc.) to ensure the effectiveness and security of the encryption process.

[0146] Furthermore, for data that needs to be encrypted, the data can be encrypted bit by bit according to the selected encryption algorithm and configured parameters. For example, if the AES algorithm is used to encrypt a data of an emergency material allocation plan, the data will be processed in a specified grouping method (such as 128 bits per group), and the original plaintext data will be converted into ciphertext data through specific mathematical operations on the key and data. The ciphertext data appears to be a string of irregular character sequences. In this way, even if the data is intercepted during transmission, the attacker cannot directly obtain the original meaningful information, which effectively protects the confidentiality of the data.

[0147] Further, when an authorized user accesses encrypted data, the data encryption and decryption unit 174 may perform a decryption operation.

[0148] First, the user's access rights are verified (by cooperating with the permission management unit 171 to confirm whether the user has the right to access the encrypted data). If the permission verification is passed, the ciphertext data can be reversed by mathematical operations based on the corresponding data encryption algorithm and the pre-saved key (for symmetric encryption, the same key is used; for asymmetric encryption, the corresponding private key is used) to restore it to the original plaintext data, so that the user can view and use the data normally, ensuring that the availability of the data is not affected while ensuring data security.

[0149] Optionally, the data desensitizing unit 175 is used to perform desensitizing processing on multi-source data sets, or to perform desensitizing processing on multiple data corresponding to big data governance results.

[0150] Specifically, specific desensitization rules can be determined according to different types of data and corresponding sensitivity levels, and then the data can be desensitized according to the desensitization rules.

[0151] Optionally, the data processing module 14 includes: a data export unit 141, which is used to call the multi-source data set from the data storage module and export the multi-source data set to the target database.

[0152] The target database is any third-party database.

[0153] The data export unit 141 can be used to export multi-source data sets to other third-party target databases, breaking the system's "data islands" and achieving efficient data sharing while maintaining data asset security.

[0154] The big data governance system provided in this embodiment realizes collaboration between modules based on natural language interaction through artificial intelligence technology, improves the intelligence level of multiple links from data management to data processing, changes the traditional mode of relying on manual setting of a large number of parameters and rules, and reduces the time cost of manual operation. Therefore, by implementing the present invention, by integrating data access, storage, management, processing and intelligent auxiliary modules into one system, the automation and integration of the data governance process is realized.

[0155] In one example, if Figure 2 As shown, a big data governance system based on smart emergency response is provided, including:

[0156] 1. Data access module: The data access module aggregates the data provided by other units and departments (data source end) to the Emergency Management Bureau (target end). The data access module provides a visual operation interface. The module encapsulates various data access operations into fixed components. The corresponding access component is selected according to the data source type, and key parameters are configured to achieve data access from the source end to the target end. The data source type supports common relational databases (DaMeng database, Mysql database, Oracle, MongoDB, Gauss200, etc.), data warehouses (Hive, Hbase), API interfaces, local files, and Kafka message middleware.

[0157] 2. Data storage module: The storage module is used to store data gathered from various data sources (other units and departments) to the target end (Emergency Management Bureau). The storage module of this system uses big data storage components (HDFS+Hive, Hbase) for data storage.

[0158] 3. Intelligent auxiliary module (backend support module is not visible to system users):

[0159] (1) Assisted script development capability. It provides the ability to intelligently generate data processing scripts, supports inputting natural language to describe the logic of the data processing script, the table name and field name to be processed by the script, and then automatically generates the script. This capability is integrated with the data processing module, allowing developers to quickly develop data processing scripts.

[0160] (2) Assisting data management and control capabilities. Providing auxiliary capabilities that can assist developers in improving efficiency in important governance and control activities.

[0161] —Assist in sorting out data standards: support parsing of input documents (such as existing ministerial emergency data standard files, provincial and departmental emergency data standard files, municipal data standard files, Municipal Emergency Management Bureau data standard files, data source unit data interpretation files, etc.), and disassemble various data standard regulations. The data standard regulations include the Chinese name and English name of the data table, the Chinese name and English name of the data field, data type, field length, constraints, Chinese pinyin abbreviations, definition instructions, field code enumeration values, etc.

[0162] —Assist in data quality auditing: Support natural language description of data quality issues, design quality auditing rules based on the description, and generate corresponding scripts based on the rules.

[0163] —Assist in data model design: Supports input of natural language description or image of conceptual model, and automatically performs logical model design and physical model design based on the description or image.

[0164] 4. Data processing module:

[0165] —The visual interface provided by the data processing module is used for the creation, deletion, modification, editing, and publishing of data processing scripts. Developers can customize the script type to be developed, including SQL, HQL, Shell, Spark, etc.; at the same time, the intelligent auxiliary module script development capability is integrated, and the script intelligent auxiliary development interface is provided. The script development requirement description can be entered to automatically generate the script, and script testing and manual modification are supported. It is used for developers to design and manage complex data processing logic, so as to complete data cleaning, conversion, aggregation and other processing.

[0166] —Data source management supports adding, deleting, modifying and checking the database information that can be connected to this system.

[0167] —Data export supports exporting data in the data storage module of this system to a specified third-party database.

[0168] 5.Data control module:

[0169] ——Intelligent data standards management: Provides a visual maintenance interface for standard formulation, publication, updating and maintenance, and supports manual formulation and management of data standards, including the addition, deletion, modification and query of Chinese and English names of data tables, Chinese and English names of data fields, data types, field lengths, constraints, Chinese pinyin abbreviations, definition descriptions, field code enumeration values, etc. At the same time, it integrates the standard combing capability of the intelligent auxiliary module, provides a file upload interface, and automatically combs and disassembles data standards according to the content of the uploaded file. Supports comparison of the published standards in the system with the metadata managed by this system, and filters out fields for which no standards have been established. (Note: If there are 5,000 fields in the data governance system, and the standard management module has published 2,000 field standards, you can filter out the 3,000 fields that do not have standards)

[0170] —Intelligent data quality exploration: Provides a visual operation interface for developers to manually design data quality inspection rules and configure data quality audit tasks. At the same time, it integrates the quality audit capability of the intelligent auxiliary module, provides an intelligent auxiliary interface, and automatically generates data problem audit strategies, audit rules, and scripts corresponding to audit rules after entering the data problem description, and supports script testing and manual modification.

[0171] —Metadata management: Metadata management. The metadata management module is responsible for recording and managing the descriptive information of all data in the data governance system, such as data source, structure, meaning, etc. It provides a visual operation interface for metadata query, browsing, editing, and kinship relationship building and display.

[0172] —Data asset map: displays the overall situation of data resources managed by the system and the new additions on the day. Specific indicators include the total number of tables, the total number of fields, the total number of data items, and the total data storage volume. It supports functions such as data retrieval, data viewing / preview, and data lineage.

[0173] —Intelligent warehouse modeling management: Provides a visual operation interface for developers to manually create and manage data models, and is used for manual model design including logical models, physical models, etc., and supports model design storage, version control and publishing functions. At the same time, it integrates the data model design capabilities of the intelligent auxiliary module, provides an intelligent auxiliary model design interface, supports input modeling requirements (generally a description or image of a conceptual model), and automatically generates logical models and logical models based on the input content, and matches the published data standards to ensure that the generated table model meets the existing standard requirements.

[0174] 6. Task management and control module: Supports visual operation interface for developers to manage and schedule various tasks in the data management system. Task types include data collection, processing, management, etc. Supports multiple scheduling methods such as timed scheduling and event triggering to ensure efficient execution of tasks.

[0175] 7. Data security module: It integrates functions such as permission management, unified authentication, data classification, data encryption and decryption, and data desensitization. By configuring the user's access rights to data, it can ensure that sensitive data is only open to authorized users. It supports the integration of the existing network's authentication system to achieve unified authentication services to simplify the user login process. The data classification function supports the classification and labeling of data of different importance, which is convenient for the implementation of targeted protection measures. It supports encryption and desensitization of the data managed in this system, which can effectively prevent the risk of leakage of sensitive information when it is shared or used publicly.

[0176] The big data governance system based on smart emergency response provided in this example has the following beneficial effects:

[0177] (1) Deep integration of artificial intelligence technology: The present invention closely combines artificial intelligence technology, especially NLP big model technology, with the emergency data governance system, realizes the intelligent generation of data processing scripts and intelligent assistance of data management activities, and improves the intelligent level of data governance.

[0178] (2) Application of natural language interaction: Through NLP technology, the system can understand and parse the data processing logic, data standards, quality audit rules, and data model design requirements described in natural language, thereby automatically generating corresponding scripts and models, greatly simplifying the operations of developers.

[0179] (3) Intelligent management of data standards and quality: The system can automatically parse and sort out various data standard documents, compare them with existing data, and quickly identify and fill in the gaps in data standards. At the same time, through the design of intelligent quality audit rules, the system can automatically discover data quality problems and further promote the improvement of data accuracy and consistency.

[0180] (4) Intelligent data modeling: The system supports intelligent design of data models through natural language or image description, automatically generates logical models and physical models, and matches them with published data standards to ensure the compliance and efficiency of model design.

[0181] Therefore, by implementing the present invention, through the intelligent generation of data processing scripts and intelligent auxiliary data management and control activities, the time cost and technical requirements for manual script writing and data management are significantly reduced, and the overall efficiency of data governance is improved. Furthermore, the introduction of intelligent quality auditing and repair functions enables data quality problems to be discovered and effectively handled in a timely manner, thereby improving the accuracy and reliability of the data and providing more solid data support for emergency management decisions. Furthermore, the intelligent data governance system reduces dependence on manual labor, reduces the manpower cost and time cost of data governance, and makes data governance work more economical and efficient.

[0182] According to an embodiment of the present invention, an embodiment of a big data governance method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0183] In this embodiment, a big data management method is provided, which can be used in the big data management system 1 provided in the above embodiment of the present invention. Figure 3 is a flowchart of a big data governance method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0184] Step S301, obtaining a multi-source data set and a metadata set.

[0185] For the specific process, please refer to the functional description of the data access module 11 and the data control module 13 in the big data governance system 1 provided in the above embodiment of the present invention, which will not be repeated here.

[0186] Step S302: Generate control standards based on the metadata set.

[0187] Among them, the control standards are one or more of data management standards, data quality detection standards and intelligent standard models.

[0188] For the specific process, please refer to the functional description of the data management module 13 in the big data governance system 1 provided in the above embodiment of the present invention, which will not be repeated here.

[0189] Step S303: determining a target data set from multiple source data sets based on the control standard.

[0190] For the specific process, please refer to the interaction process description and functional description of the data management module 13, data processing module 14, intelligent assistance module 15 and task management module 16 in the big data governance system 1 provided in the above-mentioned embodiment of the present invention, which will not be repeated here.

[0191] Step S304, determine the big data governance results based on the management and control standards and the target data set.

[0192] For the specific process, please refer to the functional description of the data management module 13 in the big data governance system 1 provided in the above embodiment of the present invention, which will not be repeated here.

[0193] The big data governance method provided in this embodiment reduces operational differences and confusion, improves the standardization level of global emergency data governance, and realizes the automation and integration of data governance processes by executing the data governance process in the big data governance system.

[0194] The embodiment of the present invention also provides a computer device for executing the above Figure 3 The big data governance approach shown.

[0195] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0196] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0197] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0198] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0199] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0200] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0201] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0202] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0203] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A big data governance system, characterized in that: The system includes: a data access module, a data storage module, a data management module, a data processing module and an intelligent auxiliary module; The data access module is used to obtain a multi-source data set and send the multi-source data set to the data storage module for storage; The data control module is used to obtain a metadata set, and based on the metadata set, call the intelligent auxiliary module and generate a control standard; The data control module is further used to send a data call instruction to the data processing module based on the control standard; The data processing module is used to call a target data set from the multi-source data sets in the data storage module based on the data calling instruction, and send the target data set to the data control module; The data control module is also used to determine the big data governance results based on the control standards and the target data set.

2. The system according to claim 1, characterized in that The control standard is one or more of a data management standard, a data quality detection standard and an intelligent standard model; the data control module includes: an intelligent data standard management unit, an intelligent data quality detection unit and an intelligent warehouse modeling management unit; The intelligent data standard management unit is used to call the intelligent auxiliary module to generate the data management standard based on the metadata set; The intelligent data quality detection unit is used to call the intelligent auxiliary module to generate the data quality detection standard based on the metadata set; The intelligent warehouse modeling management unit is used to call the intelligent auxiliary module to generate the intelligent standard model based on the metadata set.

3. The system according to claim 1, characterized in that The data processing module is specifically used for: The intelligent auxiliary module is called based on the data calling instruction and a target script is generated. The target script is used to call the target data set from the multi-source data sets of the data storage module, and the target data set is sent to the data control module.

4. The system according to claim 3, characterized in that The system further comprises: The task control module is used to control the target script in the data processing module.

5. The system according to claim 4, characterized in that The task control module includes: a task management unit, a task flow management unit and a task monitoring unit; The task management unit is used to manage the operation of the target script when there is a target script in the data processing module; The task flow management unit is used to schedule and manage the execution of multiple target scripts when there are multiple target scripts in the data processing module; The task monitoring unit is used to monitor the running status of the target script.

6. The system according to claim 2, characterized in that The data control module also includes: Metadata management unit, used to record and manage descriptive information of data; The data asset map unit is used to display the big data governance results.

7. The system according to claim 1, characterized in that The system further comprises: a data security module, the data security module comprising a rights management unit and a unified authentication unit; The permission management unit is used to receive the multi-source data set sent by the data access module, and manage user access rights based on the multi-source data set; The unified authentication unit is used to perform unified authentication on user access information.

8. The system according to claim 7, characterized in that The data security module also includes: a data classification unit, a data encryption and decryption unit, and a data desensitization unit; The data security module is used to receive the multi-source data set sent by the data access module or the big data governance result sent by the data control module; The data classification unit is used to perform classification processing on the multi-source data set or to perform classification processing on multiple data corresponding to the big data governance result; The data encryption / decryption unit is used to perform encryption / decryption processing on the multi-source data set, or to perform encryption / decryption processing on multiple data corresponding to the big data governance result; The data desensitizing unit is used to perform desensitizing processing on the multi-source data set, or to perform desensitizing processing on multiple data corresponding to the big data governance results.

9. The system according to claim 1, characterized in that The data processing module includes: The data export unit is used to call the multi-source data set from the data storage module and export the multi-source data set to a target database.

10. A big data governance method, characterized in that: Used in a big data governance system as claimed in any one of claims 1 to 9; the method comprising: Acquire multi-source datasets and metadata sets; Generate a control standard based on the metadata set, where the control standard is one or more of a data management standard, a data quality detection standard, and an intelligent standard model; Based on the control criteria, determining a target data set from the multi-source data sets; Based on the control standards and the target data set, determine the big data governance results.

Images