A data warehouse construction method, device, equipment and computer storage medium
By integrating data management modules during the construction of data warehouses and automatically processing and managing data, the high operation and maintenance cost problems caused by the separation of data warehouse construction and management are solved, and data warehouse construction and management are realized, reducing labor consumption and operation costs.
Patent Information
- Application Number
- CN202210168707.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-23
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-02-23
AI Technical Summary
During the construction of data warehouses, data warehouses and data management are carried out separately, which leads to the need for a lot of manpower and time to manage data in the later stage, increasing operation and maintenance costs and workload.
By integrating data management during the construction of the data warehouse, using data acquisition module, data rule management module, metadata management module, data model management module, data permission module and data life cycle management module, automated processing and management data, and generating data reports And perform permissions and life cycle management.
It realizes the construction and management of data warehouses while building, reducing human resources consumption, reducing construction and operation costs, and allowing developers to concentrate on designing excellent data warehouses and data governance architectures.
Smart Images

Figure CN114547173B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data management, and in particular to a data warehouse construction method, device, equipment and computer storage medium. Background Art
[0002] In the process of building a data warehouse, building a data model for the data warehouse is the foundation, and the ETL processing of data is the core. Generally, the traditional process of building a data warehouse is to first build a multidimensional data model through a data modeling tool. In the process of building the model, a large number of dimension tables and fact tables need to be created manually, which consumes a lot of manpower and is difficult to use standard specifications. Constraints, resulting in non-standard and non-uniform data models, bring unnecessary workload and hidden dangers to subsequent data processing and data applications.
[0003] In the process of building the data warehouse model, metadata management, master data management, data traceability and other governance were not performed on the data, which required later developers to manage the data in the data warehouse so that users can efficiently use and share the data in the data warehouse.
[0004] Therefore, how to combine the data warehouse construction process with data management to save the operation and maintenance costs of data warehouse and data management is an issue that technical personnel are increasingly concerned about. Summary of the invention
[0005] The embodiments of the present application provide a data warehouse construction method, apparatus, device and computer storage medium, which solve the problem that the data warehouse construction and data management are carried out separately, resulting in the later developers consuming a lot of human resources and time resources to manage the data in the data warehouse.
[0006] In a first aspect, an embodiment of the present application provides a data warehouse construction method, the method comprising: extracting first target source data from a business database; adding a first tag to the first target source data to obtain second target source data; the first tag is identification information of the business database corresponding to the first target source data; reading first data indication information in the target business database and second data indication information in the data warehouse construction system; the target business database is the business database corresponding to the second target source data, the first data indication information comprises a mapping relationship between each character data of the second target source data and a character data indication object, and the second data indication information comprises a mapping relationship between each character data of the second target source data and a character data indication object; in the case where the first data indication information is inconsistent with the second data indication information, modifying the character data in the second target source data based on the second data indication information to obtain third target source data; the indication object of the character data in the third target source data is consistent with the second data indication information; performing metadata management on the third target source data to obtain fourth target source data; constructing a data report based on a preset data model and the fourth target source data, and constructing a data warehouse based on the data report.
[0007] In the above embodiment, by integrating the construction of the data warehouse and data management, the goal of building and managing the data warehouse at the same time is achieved, and data scripts or codes are automatically generated as much as possible, which greatly liberates manpower and reduces the construction cost and operation cost of the data warehouse, allowing developers to shift their focus to designing excellent data warehouses and data governance architectures.
[0008] In combination with the first aspect, in a possible implementation, the character data in the second target source data is modified based on the second data indication information to obtain the third target source data, including: reading the first field; the first field is used to represent the data type of the character data in the first storage area of the first target source data; reading the indication object of each character data in the first storage area in the first data indication information based on the first field; based on the indication object, reading the target character data corresponding to the indication object in the second indication information; if the target character data of the indication object is inconsistent with the character data corresponding to the indication object in the first storage area, modifying the character data corresponding to the indication object in the first storage area to the target character data to obtain the third target source data. In this way, by unifying the meaning of the character data in the second target source data, the working efficiency of the data warehouse can be improved.
[0009] In combination with the first aspect, in one possible implementation, metadata management is performed on the third target source data, specifically including: reading a first field, the first field being used to characterize the data type of character data in a first storage area of the first target source data; obtaining a target attribute label in a metadata list based on the first field; the metadata list including attribute labels of character data in each storage area in the third target source data; and adding the target attribute label in a second field in the first storage area.
[0010] In combination with the first aspect, in a possible implementation method, after building a data report based on a preset data model and the fourth target source data and building a data warehouse based on the data report, it also includes: performing authority management on the data report.
[0011] In combination with the first aspect, in one possible implementation method, permission management is performed on data reports, including: receiving user information sent by a query system, the user information including a user name and an identifier of a target data report; querying a permission list based on the user information, the permission list including a permission mapping relationship between the user information and the data report; retrieving the target data report when it is found that the user corresponding to the user information has the permission to access the target data report; and sending the target data report to the query system.
[0012] In combination with the first aspect, in one possible implementation method, a data report is constructed based on a preset data model and a fourth target source data, and after a data warehouse is constructed based on the data report: the data report retrieval log is read regularly; the data report retrieval log includes identification information of the retrieved data report and the user name of the user corresponding to the retrieved data report; when a non-target user name exists in the log, an early warning message is sent to the query system; the non-target user name is a user name that does not exist in the user list, and the user list is a list stored in the data warehouse construction system.
[0013] In combination with the first aspect, in a possible implementation method, after building a data report based on a preset data model and a fourth target source data, and building a data warehouse based on the data report, it also includes: managing the data life cycle of the data report; wherein, managing the data life cycle of the data report includes: stamping the fourth target source data with a first timestamp, the first timestamp being the generation time of the data report; periodically stamping the fourth target source data with a second timestamp; calculating the difference between the first timestamp and the second timestamp; and deleting the fourth target source data when the difference is greater than or equal to a first threshold.
[0014] In a second aspect, an embodiment of the present application provides a data warehouse construction device, the device comprising: a data acquisition module, a first label adding module, a data rule management module, a metadata management module, and a data model management module, wherein:
[0015] A data acquisition module, used to extract first target source data from a business database;
[0016] A data information reading module, used to read the first data indication information in the target business database and the second data indication information in the data warehouse construction system;
[0017] A data rule management module is used to modify the character data in the second target source data based on the second data indication information to obtain third target source data when the first data indication information is inconsistent with the second data indication information; the indication object of the character data in the third target source data is consistent with the second data indication information.
[0018] A metadata management module, used to perform metadata management on the third target source data to obtain fourth target source data;
[0019] The data model management module is used to construct a data report based on a preset data model and the fourth target source data, and to construct a data warehouse based on the data report.
[0020] In conjunction with the second aspect, in a possible implementation manner, the data rule management module includes:
[0021] A field reading unit, used for reading a first field;
[0022] A first reading unit, configured to read, in the first data indication information, an indication object of each character data in the first storage area based on the first field;
[0023] A second reading unit is used to read the target character data corresponding to the indicated object in the second indication information based on the indicated object; a modification unit is used to modify the character data corresponding to the indicated object in the first storage area to the target character data when the target character data of the indicated object is inconsistent with the character data corresponding to the indicated object in the first storage area, so as to obtain third target source data.
[0024] In conjunction with the second aspect, in a possible implementation, the metadata management module includes:
[0025] A first field reading unit, used for reading a first field;
[0026] A label acquisition unit, used to acquire a target attribute label in the metadata list based on the first field;
[0027] The label adding unit is used to add the target attribute label in the second field in the first storage area.
[0028] In combination with the second aspect, in one possible implementation, the device further includes a data permission module, and the data permission module is used to perform permission management on data reports.
[0029] In conjunction with the second aspect, in one possible implementation, the data authority module includes:
[0030] A user information receiving unit, used to receive user information sent by the query system;
[0031] A permission query unit, used to query the permission list based on user information;
[0032] A retrieving unit, used to retrieve the target data report when it is found that the user corresponding to the user information has the authority to access the target data report;
[0033] The data report sending unit is used to send the target data report to the query system.
[0034] In conjunction with the second aspect, in a possible implementation, the device further includes a data supervision module, which is used to perform data supervision on the data report; wherein the data supervision module includes:
[0035] Log reading unit, used to read data reports and retrieve logs at regular intervals;
[0036] The warning information sending unit is used for the data supervision module to send warning information to the query system when there is a non-target user name in the log.
[0037] In conjunction with the second aspect, in a possible implementation, the device further includes a data lifecycle management module, the data lifecycle management module is used to perform data lifecycle management on the data report, and the data lifecycle management module includes:
[0038] A first timestamp adding unit, used for adding a first timestamp to the fourth target source data;
[0039] A second timestamp adding unit, used for periodically adding a second timestamp to the fourth target source data;
[0040] A difference calculation unit, used to calculate the difference between the first timestamp and the second timestamp;
[0041] The deleting unit is used to delete the fourth target source data when the difference is greater than or equal to the first threshold.
[0042] In a third aspect, an embodiment of the present application provides a data warehouse construction device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the following steps: extracting first target source data from a business database; adding a first tag to the first target source data to obtain second target source data; the first tag is identification information of the business database corresponding to the first target source data; reading first data indication information in the target business database and second data indication information in the data warehouse construction system; the target business database is the business corresponding to the second target source data Database, the first data indication information includes a mapping relationship between each character data of the second target source data and the character data indication object, and the second data indication information includes a mapping relationship between each character data of the second target source data and the character data indication object; in the case where the first data indication information is inconsistent with the second data indication information, the character data in the second target source data is modified based on the second data indication information to obtain third target source data; the indication object of the character data in the third target source data is consistent with the second data indication information; metadata management is performed on the third target source data to obtain fourth target source data; based on a preset data model and the fourth target source data, a data report is constructed, and a data warehouse is constructed based on the data report.
[0043] In combination with the third aspect, in one possible implementation, the one or more processors call the computer instructions to enable the electronic device to execute: modify the character data in the second target source data based on the preset data writing rules to obtain the third target source data, including: reading the first field, the first field being used to characterize the data type of the character data in the first storage area of the first target source data; reading the indication object of each character data in the first storage area in the first data indication information based on the first field; based on the indication object, reading the target character data corresponding to the indication object in the second indication information; if the target character data of the indication object is inconsistent with the character data corresponding to the indication object in the first storage area, modifying the character data corresponding to the indication object in the first storage area to the target character data to obtain the third target source data.
[0044] In combination with the third aspect, in one possible implementation, the one or more processors call the computer instructions to cause the electronic device to execute: read the first field, the first field being used to characterize the data type of the character data in the first storage area of the first target source data; obtaining a target attribute label in a metadata list based on the first field; the metadata list including attribute labels of the character data in each storage area in the third target source data; and adding the target attribute label in the second field in the first storage area.
[0045] In conjunction with the third aspect, in one possible implementation, the one or more processors call the computer instructions to cause the electronic device to execute: performing rights management on the data report.
[0046] In combination with the third aspect, in one possible implementation, the one or more processors call the computer instructions to cause the electronic device to execute: performing permission management on the data report, including: receiving user information sent by a query system, the user information including a user name and an identifier of a target data report; querying a permission list based on the user information, the permission list including a permission mapping relationship between the user information and the data report; retrieving the target data report when it is found that the user corresponding to the user information has the authority to access the target data report; and sending the target data report to the query system.
[0047] In combination with the third aspect, in one possible implementation, the one or more processors call the computer instructions to cause the electronic device to execute: regularly reading the data report retrieval log; the data report retrieval log includes identification information of the retrieved data report and the user name of the user corresponding to the data report; if there is a non-target user name in the log, sending an early warning message to the query system; the non-target user name is a user name that does not exist in the user list, and the user list is a list stored in the data warehouse construction system.
[0048] In combination with the third aspect, in one possible implementation, the one or more processors call the computer instructions to cause the electronic device to execute: performing data lifecycle management on the data report; wherein the data lifecycle management on the data report includes: marking the fourth target source data with a first timestamp, wherein the first timestamp is the generation time of the data report; periodically marking the fourth target source data with a second timestamp; calculating the difference between the first timestamp and the second timestamp; and deleting the fourth target source data when the difference is greater than or equal to a first threshold.
[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a data warehouse construction device, causes the device to execute the method described in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A flowchart of the interaction between the functional modules in a data warehouse construction system provided in an embodiment of the present application;
[0051] Figure 2 A metadata schematic diagram is provided for an embodiment of the present application;
[0052] Figure 3 A schematic diagram of the structure of a data warehouse construction device provided in this application;
[0053] Figure 4 A schematic diagram of the structure of a data warehouse construction device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0054] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Mentioning "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present embodiment application. The appearance of this phrase in various positions in the specification is not necessarily the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It can be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0055] The terms "first", "second", "third", etc. in the specification and claims of the present application and the drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a series of steps or units are included, or optionally, steps or units not listed are included, or optionally, other steps or units inherent to these processes, methods, products or devices are included.
[0056] Only the part relevant to the present application but not all content is shown in the accompanying drawings.Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processing or methods described as flow charts.Although flow charts describe various operations (or steps) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously.In addition, the order of various operations can be rearranged.Described processing can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings.Described processing can correspond to method, function, procedure, subroutine, subprogram etc.
[0057] As used in this specification, the terms "component", "module", "system", "unit", etc. are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a unit can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or distributed between two or more computers. In addition, these units can be executed from various computer-readable media having various data structures stored thereon. Units can communicate through local and / or remote processes, for example, based on signals having one or more data packets (e.g., data from a second unit interacting with another unit in a local system, a distributed system, and / or a network. For example, the Internet interacts with other systems via signals).
[0058] A data warehouse is a strategic collection of all types of data that supports decision-making processes at all levels of an enterprise. This collection is a single data store created by an enterprise to obtain analytical reports and decision support. By integrating data distributed in various places into a unified data warehouse, the business departments of the enterprise can process and analyze the data stored in the data warehouse, thereby guiding business process improvements and monitoring time, cost, quality and control.
[0059] Data management includes: data resource combing, data collection and cleaning, metadata management, lineage tracking, etc. Since the data in the data warehouse is often redundant and complex, and the quantity is huge, this means that data management of the data in the data warehouse is a huge project that requires a huge period of time and huge manpower costs.
[0060] At present, in the process of building a data warehouse, the source data in multiple data service libraries are only collected into the data warehouse through extraction, transformation and loading, and the data in the data warehouse is not managed. That is, the construction and data management of the current data warehouse are carried out separately, which makes it impossible for users to efficiently use the data in the data warehouse for analysis. Therefore, in order to efficiently use the data warehouse for data analysis, it is often necessary to manage the data in the data warehouse after the data warehouse is built. Due to the inconsistency of data types and different attributes such as naming rules of the data in the data warehouse, it is necessary to manage the data of the data warehouse after the data warehouse is established, which consumes a lot of time and human resources.
[0061] In order to solve the problem that a large amount of human resources and time resources are consumed to manage the data in the data warehouse after the data warehouse is built because the data warehouse construction and data management are carried out separately, the embodiment of the present application provides a data warehouse construction method, which is applied to the data warehouse construction system, and the method includes: in the initial stage of data warehouse construction, business and data resources are sorted to obtain a classified data resource list. Then, the data is extracted (extracted), transformed (transformed) and loaded (loaded) from the business database through a visual ETL tool (for example, DataX, Pentaho Data Integration), and centrally stored. Then, the database table structure is designed based on certain principles, and the data collection and cleaning process is designed according to the data resource list, and the neat and clean data is stored in the data warehouse, and the data item attributes in the basic library and the theme library are managed. At the same time, the business meaning of the data item is associated with the data item, so that the business personnel can also understand the meaning of the data field in the database, and the data is linked to its data service library to establish a lineage tracking, so that when the data is used by the business scenario, the data error is found, and the data governance team can quickly locate the data source and fix the data error. Finally, relevant topic models are established and data reports are generated to enable data sharing within or between enterprises.
[0062] Below, the interaction process of each functional module in the data warehouse construction system is explained. The data warehouse construction system mainly works through the configuration engine, which includes: data acquisition module, data rule management module, metadata management module, data model management module, data permission module, data supervision module and data life cycle management module. Among them, the data acquisition module is used to extract and import data from the business database; the data rule management module is used to unify the data rules of the source data extracted from different business databases; the metadata management module is used to perform metadata management on the source data extracted by the data acquisition module; the data model management module is used to establish a data model and extract relevant data according to the data model to form different data reports; the data permission module is used to perform permission management on the generated data reports, and the data permission module does not provide relevant data reports for users who do not meet the permissions; the data supervision module is used to supervise the data reports in the data warehouse, and in the case of a network attack on the data warehouse, timely issue an early warning to the administrator to avoid data leakage; the data life cycle management module is used to perform life cycle management on the business data reports in the data warehouse, and clear the data reports that exceed the life cycle to avoid data redundancy in the data warehouse.
[0063] Next, combine Figure 1 , the interaction process of each functional module in the data warehouse construction system is explained. The specific process is as follows:
[0064] Step S101: The data acquisition module extracts first target source data from multiple business databases.
[0065] Specifically, the business database is a database designed by the business department according to its business needs. The business database is a collection of all business data of the business department. These business data are usually stored in the form of tables. For example, the human resources department usually has data tables such as employee address data tables, employee information data tables, and employee attendance data tables. The address data table usually includes data such as the employee's name, employee gender, and employee address information. The employee information data table usually includes information such as the employee's name, employee age, employee gender, and employee work experience. The employee attendance data table usually includes information such as the employee's name, employee work number, employee punch-in times, and employee punch-in time.
[0066] Before extracting the business database, the data acquisition module will prioritize the data tables in the business database and mark the data tables with high priority as the data tables to be extracted, which are the first target source data.
[0067] Exemplarily, the data acquisition module can calculate the priority of each data table in the business database based on factors such as the business matching degree between the data table in the business database and the business department. For example, the KPI assessment business of the personnel department for employees is mainly evaluated based on the two aspects of employee attendance and work quality. The employee's attendance data is mainly recorded in the employee attendance data table, and the employee's work quality and other related data are recorded in the workload data table. The workload data table mainly includes data such as employee name, work number, and the amount of work completed in the month. For the KPI assessment business, since the employee attendance data table and the workload data table are highly related to the business, the business matching degree of these two data tables with the KPI assessment business is very high. Based on factors such as business matching degree, the data acquisition module determines that the employee attendance data table and the workload data are high-priority data tables, marks them as the first target source data, and extracts the first target source data.
[0068] In this way, the data acquisition module can screen the useful data in the business database without having to extract all the data in the business database, thereby saving a lot of resources in the construction of the data warehouse and avoiding the appearance of a large amount of redundant data in the data warehouse, which in turn causes the problem of high operation and maintenance costs of the data warehouse in the later stage.
[0069] Step S102: The data acquisition module adds a first tag to the first target source data to obtain second target source data.
[0070] Specifically, in the process of extracting the first target source data from the business database by the data acquisition module, a first tag can be added to the field of the first target source data, and the first tag is the unique identification information of the business database corresponding to the first target source data. By adding the first tag to the first target source data, the first target source data can be associated with its corresponding business database. For example, the first target source data extracted from database 1 is data table 1, the first target source data extracted from database 2 is data table 2, the first target source data extracted from database 3 is data table 3, and the first target source data extracted from database 4 is data table 4. The identification information of database 1 is a, the identification information of database 2 is b, the identification information of database 3 is c, and the identification information of database 4 is d. When the data acquisition module extracts data 1 from database 1, field a will be inserted into data table 1, field b will be inserted into data table 2, field c will be inserted into data table 3, and field d will be inserted into data table 4, and 4 second target source data will be obtained respectively. In this way, it is possible to determine which business database the second target source data corresponds to through field information such as a, b, c, and d.
[0071] By adding the first tag to the first target source data, each second target source data can have the information of its corresponding business database, and the second target source data can be associated with its corresponding business database to trace the second target source data. This can solve the problem that when the data is used in business scenarios, if an error occurs in the data, the data management team cannot quickly locate the source of the data and repair the data error in time. For example, during the use of the data warehouse, if a data error is found, it is often necessary to trace the source. If the first tag is included in the data, the data can be traced according to the first tag in the data, so as to quickly locate the business database of the data, so that the data management team can analyze the relevant data in the business database, so that the data management team can quickly repair the erroneous data.
[0072] Step S103: the data acquisition module imports the second target source data into the data buffer layer (STAGE layer).
[0073] Step S104: the data processing module sends a first notification message to the data rule management module.
[0074] Specifically, after the data acquisition module imports the second target source data into the data buffer layer, the data acquisition module may send a first notification message to the data rule management module, where the first notification message is used to instruct the data rule management module to perform a first process on the second target source data.
[0075] Step S105: The data rule management module performs a first process on the second target source data to obtain third target source data.
[0076] Specifically, the first processing is to unify the meaning of the character data in the second target source data. Since the second target source data imported into the data buffer layer is extracted from multiple business databases, the data in different business databases may have different writing rules for character data, resulting in multiple expressions for one type of data. For example, in the database of department 1, the technicians use characters "0" and "1" to represent undergraduate and graduate degrees in academic qualifications, in the database of department 2, the technicians use characters "A" and "B" to represent undergraduate and graduate degrees in academic qualifications, and in the database of department 3, the technicians use characters "00" and "01" to represent undergraduate and graduate degrees in academic qualifications. Therefore, there may be three or more representations for character data of the "education level" type in the data warehouse. When the data in the data warehouse increases, if the education level type character data called is inconsistent with the system preset writing rules, it is necessary to analyze the business database of the data table corresponding to the education level type field and query the writing rules for the education level character data in the business database, which undoubtedly increases the workload of the system and reduces the work efficiency of the system. Therefore, in order to improve the work efficiency of the data warehouse, it is necessary to unify the meaning of the character data in the second target source data.
[0077] The specific steps of the data rule management module performing the first processing on the second target source data are:
[0078] 1. Read the first data indication information in the target business database and the second data indication information in the data warehouse construction system; the target business database is the business database corresponding to the second target source data, the first data indication information includes the mapping relationship between each character data of the second target source data and the character data indication object, and the second data indication information includes the mapping relationship between each character data of the second target source data and the character data indication object.
[0079] 2. Read the first field; the first field is used to characterize the data type of the character data in the first storage area of the first target source data; for example, the data type stored in the first storage area may be "education level", "age", etc.
[0080] 3. Read the indication object of each character data in the first storage area in the first data indication information based on the first field; the first data indication information includes the mapping relationship between each character data of the second target source data and the character data indication object, and the second data indication information includes the mapping relationship between each character data of the second target source data and the character data indication object.
[0081] 4. Based on the indicated object, read the target character data corresponding to the indicated object in the second indication information;
[0082] 5. If the target character data of the indicated object is inconsistent with the character data in the first storage area corresponding to the indicated object, the character data in the first storage area corresponding to the indicated object is modified to the target character data to obtain the third target source data. For example, the data type stored in the first storage area of the second target source data is academic qualifications. After the data rule management module reads that the data type of the first field identifier is "academic qualifications", the data rule management module queries the first data indication information of the business database for academic qualifications according to the first identifier in the second target source data. If the first data indication information represents that the character "A" represents an undergraduate degree and the character "B" represents a graduate degree, the second data indication information stored in the data warehouse construction system represents an undergraduate degree with the character "0" and a graduate degree with the character "1". If the character in the first storage area is A, the data rule management module changes the character A to 0, so that the object indicated by the character in the first storage area is consistent with the object represented by the second data indication information.
[0083] Step S106: The data acquisition module extracts the third target source data into the historical data layer.
[0084] Specifically, after the data rule management module performs the first processing on the second target source data to obtain the third target source data, the data acquisition module extracts the third target source data to the historical data layer.
[0085] Step S107: the data acquisition module sends a second notification message to the metadata management module.
[0086] Specifically, after the data acquisition module extracts the third target source data into the historical data layer, the data acquisition module sends a second notification message to the metadata management module, where the second notification message is used to instruct the metadata management module to perform metadata management on the third target source data.
[0087] Step S108: The metadata management module performs metadata management on the third target source data to obtain fourth target source data.
[0088] Specifically, metadata is data used to describe data, and metadata is used to indicate the attributes of a data type. The method by which the metadata management module manages metadata for data is as follows: the metadata management module first reads the first field in each storage area in the third target source data to obtain the data type of the character data in each storage area, and then obtains the target attribute label of the character data in the metadata list according to the first field. The metadata list is a list pre-stored by the data warehouse construction system, and the list includes the correspondence between the character data and its attributes in each storage area of the third target source data. Finally, the metadata management module adds the target attribute label to the second field in the first storage area. For example, Figure 2The data type of one of the storage areas of the data table shown is "Name", which includes the names of five employees, namely "Li Hua", "Lin Yi", "Zhang San", "Wang Li" and "Liu Dan". If Li Hua and Lin Yi are R&D personnel, Zhang San and Wang Li are administrative personnel, and Liu Dan is a technical personnel, the name fields of Li Hua and Lin Yi can be added with R&D labels, the name fields of Zhang San and Wang Li can be marked with administrative labels, and the name field of Liu Dan can be marked with technical labels. Among them, the R&D label, administrative label, and technical label are attribute labels of the name data, and the field to which the attribute label is added is the second field.
[0089] Step S109: The metadata management module sends a third notification message to the data model management module.
[0090] Specifically, after the metadata management module performs metadata management on the third target source data in the historical data layer and obtains the fourth target source data, the metadata management module sends a third notification message to the data model management module. The fourth notification message is used to instruct the data model management module to collect the fourth target source data required for the data model based on its preset data model and build a data warehouse.
[0091] Step S110: The data model management module collects the fourth target source data required by the data model based on the preset data model and builds a data warehouse.
[0092] Specifically, after receiving the notification message sent by the metadata management module, the data model management module collects the fourth target source data required for the model according to the preset data model, and generates corresponding report data based on the data model, classifies and organizes the report data, and stores it in the data warehouse layer, thereby completing the establishment of the data warehouse.
[0093] Below, we take the data report established by the data model management module as an example of a financial analysis report. The company's financial analysis report requires financial expenditures and financial income; among them, financial expenditures include: employee salary expenditures, R&D expenses, bank repayments, product costs, etc.; financial income includes: business income, etc. Therefore, the financial analysis business model includes at least the above five elements, and the data model management module extracts the relevant fourth target source data in the historical data layer through the data acquisition module.
[0094] Exemplarily, the data acquisition module can extract corresponding data from the historical data layer according to the attribute tag. For example, the data acquisition module can extract all fourth target source data about salary in the historical data layer based on the attribute tag "salary". After the data acquisition module extracts the data required to generate the financial analysis report, the data model management module screens and integrates the extracted data according to the relevant algorithms in the financial analysis business model, and finally forms the financial analysis report.
[0095] Step S111: The data authority module performs authority management on the generated data report.
[0096] Specifically, after the data model management module generates a data report, the data model management module sends a first indication message to the data permission module, and the first indication message is used to instruct the data permission module to perform permission management on different business reports based on the preset permission rules. After receiving the first indication message, the data permission module performs permission management on the generated data report. Exemplarily, when a company employee logs into the system, the background will send the user name entered by the employee to the system, and send the user name of the employee to the permission management module, and the permission management module will read the preset permission table, which includes the mapping relationship between the user name and each data report. When the user wants to retrieve the data report in the data warehouse through the data system, if the data permission module finds in the permission table that there is no mapping relationship between the user name and the data report, the data permission module does not provide the data report that the employee wants to retrieve, and returns a prompt message to prompt the user that he has no permission to access the data report. If the data permission module finds in the permission table that there is a mapping relationship between the user name and the data report, the data permission module sends the data report to the system account corresponding to the user name for the user to view.
[0097] Step S112: The data supervision module performs data supervision on the generated data report.
[0098] Specifically, after the data model management module generates a data report, the data model management module sends a second indication message to the data supervision module, and the second indication message is used to instruct the data supervision module to perform data supervision on the data reports in the data warehouse. When a network attack or the like occurs, the data supervision module can promptly send an early warning message to the administrator to prevent data leakage in the data warehouse, thereby causing data security risk problems. The specific way in which the data supervision module performs data supervision on the data report is as follows: the data supervision module periodically reads the data report retrieval log; the data report retrieval log includes all information about the user's retrieval of report data in the data warehouse within a period of time, including the identification information of each retrieved data report and the user name of the user who retrieved the report data. In the case where there is a non-target user name in the log, the data supervision module sends an early warning message to the query system; the non-target user name is a user name that does not exist in the user list.
[0099] Step S113: The data lifecycle management module manages the data lifecycle of the data report.
[0100] Specifically, after the data model management module generates a data report, the data model management module sends a third indication information to the data lifecycle management module, and the third indication information is used to instruct the data lifecycle management module to manage the data lifecycle of the data report, specifically including: the data lifecycle management module adds a first timestamp to the fourth target source data, and the first timestamp is the time corresponding to the data lifecycle management module receiving the third indication information; the data lifecycle management module periodically adds a second timestamp to the fourth target source data; the data lifecycle management module calculates the difference between the first timestamp and the second timestamp; if the difference is greater than or equal to the first threshold, the data lifecycle management module deletes the fourth target source data. The data lifecycle management module adds a first timestamp to the fourth target source data, and the first timestamp is the time corresponding to the data lifecycle management module receiving the third indication information; the data lifecycle management module periodically adds a second timestamp to the fourth target source data; the data lifecycle management module calculates the difference between the first timestamp and the second timestamp; if the difference is greater than or equal to the first threshold, the data lifecycle management module deletes the fourth target source data.
[0101] Step S111 to step S113 are for further managing the data in the data warehouse after the construction of the data warehouse is completed. Therefore, after the data model management module generates a business data report, it will send instruction information to the data permission module, the data supervision module and the data lifecycle management module respectively, so as to activate the above three modules to manage the data reports in the data warehouse. Step S111 to step S113 are executed after step S110, and may be executed simultaneously or not. The embodiment of the present application does not limit the execution order of step S111, step S112 and step S113.
[0102] The data warehouse construction method provided in the embodiment of the present application achieves the goal of building and managing the data warehouse simultaneously by integrating the construction of the data warehouse and data management. Data scripts or codes are automatically generated as much as possible, which greatly liberates manpower and reduces the construction cost and operation cost of the data warehouse, allowing developers to shift their focus to designing excellent data warehouses and data governance architectures.
[0103] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a data warehouse construction device provided in this application. It should be noted that: Figure 3 The data warehouse construction device shown is used to execute this application Figure 1 For the convenience of explanation, only the part related to the embodiment of the present application is shown, and the specific technical details are not disclosed. Please refer to the present application for details. Figure 1 The data warehouse construction device 30 may include: a data acquisition module 301, a first tag adding module 302, a data reading module 303, a data rule management module 304, a metadata management module 305, and a data model management module 306; wherein:
[0104] The data acquisition module 301 is used to extract the first target source data from the business database;
[0105] A first label adding module 302, configured to add a first label to the first target source data to obtain second target source data;
[0106] A data information reading module 303, used to read the first data indication information in the target business database and the second data indication information in the data warehouse construction system;
[0107] A data rule management module 304 is used to modify the character data in the second target source data based on the second data indication information to obtain third target source data when the first data indication information is inconsistent with the second data indication information; the indication object of the character data in the third target source data is consistent with the second data indication information;
[0108] The metadata management module 305 is used to perform metadata management on the third target source data to obtain fourth target source data;
[0109] The data model management module 306 is used to construct a data report based on the preset data model and the fourth target source data, and to construct a data warehouse based on the data report.
[0110] In one possible implementation, the data rule management module 304 includes:
[0111] A field reading unit, used for reading a first field;
[0112] A first reading unit, configured to read, in the first data indication information, an indication object of each character data in the first storage area based on the first field;
[0113] A second reading unit, configured to read target character data corresponding to the indicated object in the second indication information based on the indicated object;
[0114] The modifying unit is used to modify the character data corresponding to the indicated object in the first storage area to the target character data to obtain third target source data when the target character data of the indicated object is inconsistent with the character data corresponding to the indicated object in the first storage area.
[0115] In a possible implementation, the metadata management module 305 includes:
[0116] A first field reading unit, used for reading a first field;
[0117] A label acquisition unit, used to acquire a target attribute label in the metadata list based on the first field;
[0118] The label adding unit is used to add the target attribute label in the second field in the first storage area.
[0119] In a possible implementation, the device 30 further includes a data permission module, which is used to perform permission management on data reports.
[0120] In one possible implementation, the data permission module includes:
[0121] A user information receiving unit, used to receive user information sent by the query system;
[0122] A permission query unit, used to query the permission list based on user information;
[0123] A retrieving unit, used to retrieve the target data report when it is found that the user corresponding to the user information has the authority to access the target data report;
[0124] The data report sending unit is used to send the target data report to the query system.
[0125] In a possible implementation, the device 30 further includes a data supervision module, which is used to perform data supervision on the data report; wherein the data supervision module includes:
[0126] Log reading unit, used to read data reports and retrieve logs at regular intervals;
[0127] The warning information sending unit is used for the data supervision module to send warning information to the query system when there is a non-target user name in the log.
[0128] In a possible implementation, the device 30 further includes a data lifecycle management module, which is used to perform data lifecycle management on the data report, and the data lifecycle management module includes:
[0129] A first timestamp adding unit, used for adding a first timestamp to the fourth target source data;
[0130] A second timestamp adding unit, used for periodically adding a second timestamp to the fourth target source data;
[0131] A difference calculation unit, used to calculate the difference between the first timestamp and the second timestamp;
[0132] The deleting unit is used to delete the fourth target source data when the difference is greater than or equal to the first threshold.
[0133] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of a data warehouse construction device provided in an embodiment of the present application. Figure 4 As shown, the data warehouse construction device 40 includes: at least one processor 401 and a memory 402. Optionally, the data warehouse construction device may also include a network interface 403, and the processor 401, the network interface 403 and the memory 402 are interconnected via an internal bus 404. The processor 401, the memory 402 and the network interface 403 can exchange data, the network interface 403 is controlled by the processor 401 to send and receive messages, the memory 402 is used to store computer programs, the computer programs include program instructions, and the processor 401 is used to execute the program instructions stored in the memory 402. The processor 401 is configured to call the program instructions to execute the above-mentioned data warehouse construction method.
[0134] The memory 402 may include a volatile memory, such as a random-access memory (RAM); the memory 402 may also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the memory 402 may also include a combination of the above-mentioned types of memory.
[0135] The processor 401 may be a central processing unit (CPU). In one embodiment, the processor 401 may also be a graphics processing unit (GPU). The processor 401 may also be a combination of a CPU and a GPU.
[0136] The memory 402 is used to store program instructions. The processor 401 calls the computer instructions to enable the device to perform the following steps: extract the first target source data from the business database; add a first tag to the first target source data to obtain the second target source data; the first tag is the identification information of the business database corresponding to the first target source data; read the first data indication information in the target business database and the second data indication information in the data warehouse construction system; the target business database is the business database corresponding to the second target source data, the first data indication information includes the mapping relationship between each character data of the second target source data and the character data indication object, and the second data indication information includes the mapping relationship between each character data of the second target source data and the character data indication object; in the case where the first data indication information is inconsistent with the second data indication information, modify the character data in the second target source data based on the second data indication information to obtain the third target source data; the indication object of the character data in the third target source data is consistent with the second data indication information; perform metadata management on the third target source data to obtain the fourth target source data; construct a data report based on the preset data model and the fourth target source data, and construct a data warehouse based on the data report.
[0137] In one possible implementation, the processor 401 calls the computer instruction to enable the electronic device to execute: modifying the character data in the second target source data based on the preset data writing rule to obtain the third target source data, including: reading the first field, the first field being used to characterize the data type of the character data in the first storage area of the first target source data; reading the indication object of each character data in the first storage area in the first data indication information based on the first field; based on the indication object, reading the target character data corresponding to the indication object in the second indication information; if the target character data of the indication object is inconsistent with the character data corresponding to the indication object in the first storage area, modifying the character data corresponding to the indication object in the first storage area to the target character data to obtain the third target source data.
[0138] In one possible implementation, the processor 401 calls the computer instruction to cause the electronic device to execute: read the first field, the first field being used to characterize the data type of the character data in the first storage area of the first target source data; obtaining a target attribute label in a metadata list based on the first field; the metadata list including attribute labels of the character data in each storage area in the third target source data; and adding the target attribute label in the second field in the first storage area.
[0139] In a possible implementation, the processor 401 calls the computer instruction to enable the electronic device to execute: performing rights management on the data report.
[0140] In one possible implementation, the processor 401 calls the computer instruction to cause the electronic device to execute: performing permission management on the data report, including: receiving user information sent by a query system, the user information including a user name and an identifier of a target data report; querying a permission list based on the user information, the permission list including a permission mapping relationship between the user information and the data report; retrieving the target data report when it is found that the user corresponding to the user information has the authority to access the target data report; and sending the target data report to the query system.
[0141] In one possible implementation, the processor 401 calls the computer instruction to cause the electronic device to execute: periodically reading the data report retrieval log; the data report retrieval log includes identification information of the retrieved data report and the user name of the user corresponding to the data report; if there is a non-target user name in the log, sending an early warning message to the query system; the non-target user name is a user name that does not exist in the user list, and the user list is a list stored in the data warehouse construction system.
[0142] In one possible implementation, the processor 401 calls the computer instruction to cause the electronic device to execute: performing data lifecycle management on the data report; wherein the data lifecycle management on the data report includes: stamping the fourth target source data with a first timestamp, wherein the first timestamp is the generation time of the data report; periodically stamping the fourth target source data with a second timestamp; calculating the difference between the first timestamp and the second timestamp; and deleting the fourth target source data when the difference is greater than or equal to a first threshold.
[0143] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.
[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by computer programs to instruct related hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.
[0145] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for constructing a data warehouse, characterized in that: Applied to a data warehouse construction system, the method includes: Extracting first target source data from a business database; Adding a first tag to the first target source data to obtain second target source data; the first tag is identification information of a business database corresponding to the first target source data; Reading first data indication information in a target business database and second data indication information in the data warehouse construction system; the target business database is a business database corresponding to the second target source data, the first data indication information includes a mapping relationship between each character data of the second target source data and the character data indication object, and the second data indication information includes a mapping relationship between each character data of the second target source data and the character data indication object; In the case where the first data indication information is inconsistent with the second data indication information, modifying the character data in the second target source data based on the second data indication information to obtain third target source data; the indication object of the character data in the third target source data is consistent with the second data indication information; Performing metadata management on the third target source data to obtain fourth target source data; Building a data report based on the preset data model and the fourth target source data, and building a data warehouse based on the data report; The step of modifying the character data in the second target source data based on the second data indication information to obtain the third target source data includes: reading a first field; the first field is used to represent the data type of the character data in the first storage area of the first target source data; reading the indication object of each character data in the first storage area in the first data indication information based on the first field; reading the target character data corresponding to the indication object in the second data indication information based on the indication object; if the target character data corresponding to the indication object is inconsistent with the character data corresponding to the indication object in the first storage area, modifying the character data corresponding to the indication object in the first storage area to the target character data to obtain the third target source data; The metadata management of the third target source data specifically includes: reading a first field, the first field being used to characterize a data type of character data in a first storage area of the first target source data; obtaining a target attribute label in a metadata list based on the first field; the metadata list including attribute labels of character data in each storage area in the third target source data; and adding the target attribute label in a second field in the first storage area.
2. The method according to claim 1, characterized in that After building a data report based on the preset data model and the fourth target source data, and building a data warehouse based on the data report, the method further includes: Perform authority management on the data report.
3. The method according to claim 2, characterized in that The performing rights management on the data report includes: Receiving user information sent by the query system, the user information including a user name and an identifier of a target data report; Based on the user information, query the permission list, wherein the permission list includes a permission mapping relationship between the user information and the data report; In the case where it is found that the user corresponding to the user information has the authority to access the target data report, retrieving the target data report; The target data report is sent to the query system.
4. The method according to claim 1, characterized in that After building a data report based on the preset data model and the fourth target source data, and building a data warehouse based on the data report, the method further includes: Reading a data report retrieval log at regular intervals; the data report retrieval log includes identification information of the retrieved data report and a user name of a user corresponding to the retrieved data report; When there is a non-target user name in the log, an early warning message is sent to the query system; the non-target user name is a user name that does not exist in the user list, and the user list is a list stored in the data warehouse construction system.
5. The method according to claim 1, characterized in that After building a data report based on the preset data model and the fourth target source data, and building a data warehouse based on the data report, the method further includes: Performing data lifecycle management on the data report; The management of the data life cycle of the data report includes: Adding a first timestamp to the fourth target source data, where the first timestamp is the generation time of the data report; Regularly stamping the fourth target source data with a second timestamp; Calculating a difference between the first timestamp and the second timestamp; When the difference is greater than or equal to the first threshold, the fourth target source data is deleted.
6. A data warehouse construction device, the device being used to execute the method according to any one of claims 1 to 5, characterized in that: The device comprises: A data acquisition module, used to extract first target source data from a business database; A first label adding module, used for adding a first label to the first target source data to obtain second target source data; A data information reading module, used to read the first data indication information in the target business database and the second data indication information in the data warehouse construction system; a data rule management module, configured to modify the character data in the second target source data based on the second data indication information to obtain third target source data when the first data indication information is inconsistent with the second data indication information; the indication object of the character data in the third target source data is consistent with the second data indication information; A metadata management module, used to perform metadata management on the third target source data to obtain fourth target source data; The data model management module is used to construct a data report based on a preset data model and the fourth target source data, and to construct a data warehouse based on the data report.
7. A data warehouse construction device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
