Dimensional data processing method, device, equipment and storage medium

Abstract

The embodiments of the present application provide a dimensional data processing method, apparatus, device and storage medium. In a first aspect, the dimensional data processing method includes: obtaining application system source database data; loading the application system source database data into the operational data storage ODS layer; adding the dimension version field to the ODS layer to obtain target dimension data; importing the target dimension data into the Hive layer; the embodiments of the present application solve the problem of poor performance of large data warehouses in processing slowly changing dimensional data in the prior art.

Description

Technical Field

[0001] The present application belongs to the field of big data warehouse data processing, and in particular, relates to a dimensional data processing method, device, equipment and storage medium. Background Art

[0002] A data warehouse is a system that is created to further mine data resources and meet decision-making needs when a large number of databases already exist. It is not a so-called "large database". The purpose of building a data warehouse solution is to serve as a basis for front-end query and analysis. Due to the large redundancy, the required storage is also large. In order to better serve front-end applications, data warehouses often have the following characteristics:

[0003] First, data retrieval is simple and has good performance. The data in a data warehouse is centralized and summarized, unlike the scattered data in a database in many tables. Therefore, the query is simpler and does not require many table joins, so it has better performance.

[0004] Second, the data quality is high. The various information provided by the data warehouse is obtained through the process of data extraction, cleaning, conversion, loading, query, and display. Compared with the source database with data inconsistency and dirty data, it has higher quality assurance.

[0005] Third, good scalability. The reason why some large data warehouse system architectures are complex is that they take into account the scalability of the next 3-5 years. In this way, it will not cost too much to rebuild the data warehouse system in the future, and it can run stably. This is mainly reflected in the rationality of data modeling and data stratification.

[0006] From the characteristics of data warehouses, we can see that data warehouse technology can awaken the data accumulated by enterprises over the years, not only to help enterprises manage these massive data, but also to explore the potential value of the data.

[0007] In the era of big data, data warehouses are more important than ever before. The "big" and "dirty" characteristics of data require a good construction of the underlying data model. At present, the big data warehouse platform is mainly built based on Hive under the Hadoop system of Haidup, and the modeling technology still uses dimensional modeling technology based on fact tables and dimension tables.

[0008] In a big data warehouse, there are some dimension tables whose attributes change over time, and a type of data called slowly changing dimension data needs to be statistically analyzed for its historical status and latest status. In the prior art, a big data warehouse processes slowly changing dimension data at a slow speed. Summary of the invention

[0009] The embodiments of the present application provide a dimensional data processing method, apparatus, device and storage medium, which can solve the problem of poor performance of large data warehouses in processing slowly changing dimensional data in the prior art.

[0010] In a first aspect, an embodiment of the present application provides a dimensional data processing method, the method comprising:

[0011] Obtain data from the source database of the application system;

[0012] Load the application system source database data into the operational data storage ODS layer;

[0013] Add the dimension version field to the ODS layer to obtain the target dimension data;

[0014] Import the target dimension data into the Hive layer.

[0015] Furthermore, in one embodiment, the application system source database data includes: slowly changing dimension values ​​and slowly changing dimension-related business data.

[0016] Furthermore, in one embodiment, loading the application system source database data into the operational data storage ODS layer includes:

[0017] Load the slowly changing dimension values ​​into the slowly changing dimension source table at the ODS layer;

[0018] Load the business data related to the slowly changing dimension into the business table related to the slowly changing dimension in the ODS layer.

[0019] Furthermore, in one embodiment, a dimension version field is added to the ODS layer to obtain target dimension data, including:

[0020] Add the dimension version field to the slowly changing dimension source table of the ODS layer and the related business table of the slowly changing dimension of the ODS layer to obtain the target dimension data.

[0021] Furthermore, in one embodiment, the method further comprises:

[0022] Update according to the changes in the slowly changing dimension values: the dimension version field of the slowly changing dimension source table and the dimension version field of the related business table.

[0023] Furthermore, in one embodiment, importing the target dimension data into the Hive layer includes:

[0024] Import the data stored in the slowly changing dimension source table of the ODS layer into the dimension table in the Hive layer, and import the data stored in the related business table of the slowly changing dimension of the ODS layer into the fact table in the Hive layer.

[0025] In a second aspect, an embodiment of the present application provides a dimensional data processing device, the device comprising:

[0026] The acquisition module is used to obtain the source database data of the application system;

[0027] The loading module is used to load the application system source database data into the operational data storage ODS layer;

[0028] Add module to add dimension version field to ODS layer to obtain target dimension data;

[0029] The import module is used to import the target dimension data into the Hive layer.

[0030] Furthermore, in one embodiment, the application system source database data includes: slowly changing dimension values ​​and slowly changing dimension-related business data.

[0031] In a third aspect, an embodiment of the present application provides a computer device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program implements the above-mentioned dimensional data processing method when executed by the processor.

[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which is stored an implementation program for information transmission, and when the program is executed by a processor, the above-mentioned dimensional data processing method is implemented.

[0033] The dimensional data processing method, apparatus, device and storage medium of the embodiments of the present application first pre-process the dimensional version of the acquired application system source database data on the ODS layer, and then synchronize the pre-processed application system source database data to the hive layer, which can give full play to the advantages of the ODS layer and Hive, avoid a large amount of data processing on the Hive layer, and thus speed up the data processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 This is a schematic diagram of a hierarchical architecture of a Hive-based big data warehouse provided by an embodiment of the present application;

[0036] Figure 2 It is a flowchart of a dimensional data processing method provided by an embodiment of the present application;

[0037] Figure 3is a structural diagram of a dimensional data processing device provided by an embodiment of the present application;

[0038] Figure 4 It is a schematic diagram of the structure of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present application and are not configured to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0040] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0041] The existing methods for processing slowly changing dimensions are as follows:

[0042] (1) Using surrogate key processing

[0043] A surrogate key is a self-increasing integer value used as the primary key in a dimension table to connect the dimension table to the fact table. Surrogate keys are a traditional way of dealing with slowly changing dimensions. Data is saved to the dimension table by adding new rows using surrogate keys. The following is an example to illustrate the role of surrogate keys in slowly changing dimensions.

[0044]

[0045]

[0046] Table 1 Employee dimension table

[0047]

[0048] Table 2 Order fact table

[0049] As shown in Table 1, the employee dimension table, the employee ID is the surrogate key, which is used to uniquely identify the employee whose responsible area has changed. In Table 2, the order fact table, the employee ID is saved and used to connect with the employee dimension table. When it is necessary to count the order amount by year and region, the corresponding information can be accurately counted.

[0050] The disadvantages of using surrogate key processing are as follows:

[0051] 1) The maintenance cost of surrogate keys is very high, especially during data loading, which has a great impact on the fact table. The impact is particularly serious in the construction of a large data warehouse based on Hive. Since Hive has great limitations in supporting updates and non-equivalent joins, the ETL process such as surrogate key generation and loading of fact table association keys will be very complicated.

[0052] 2) Since the surrogate key is a self-increasing sequence with no business meaning, once the dimension table data is lost, it is difficult to regenerate a surrogate key that is consistent with the original dimension table data. The only option is to regenerate all of it. At the same time, the associated keys of the fact table also need to be reprocessed, which is a huge workload.

[0053] (2) Using snapshot dimension table processing

[0054] In a big data warehouse, snapshot dimension tables are often used to solve the problem of slowly changing dimensions. The so-called snapshot dimension table means that the dimension table data is saved as a snapshot on a daily basis, and the changes in the dimension are marked by the snapshot date. The following example illustrates the role of snapshot dimension tables in slowly changing dimensions.

[0055] Employee Name Responsible area Snapshot Date A Region 1 2020-1-1 B Area 2 2020-1-1 A Region 1 2020-1-2 B Area 2 2020-1-2 …… …… …… A Area 3 2020-7-1 B Area 2 2020-7-1 …… …… …… A Area 3 2020-12-31 B Area 2 2020-12-31

[0056] Table 3 Employee dimension table

[0057]

[0058] Table 4 Order fact table

[0059] As shown in Table 3, the employee dimension table stores the data of each day by snapshot date. When the responsible area occurs, the data of each day thereafter is the changed data. In Table 4, the order fact table, the employee name is saved. When the order amount needs to be counted by year and region, the employee name is connected with the employee name in Table 3, and the transaction date is connected with the snapshot date in Table 3, so that the corresponding information can be accurately counted.

[0060] Using snapshot dimension tables for processing has the following disadvantages:

[0061] 1) The snapshot dimension table needs to store a large amount of duplicate snapshot data, resulting in a certain waste of storage space. In addition, the data of the snapshot dimension table needs to be frequently maintained.

[0062] 2) As time goes by, the snapshot dimension table will become very large, and the connection with the fact table will become a problem of connecting large tables to large tables. Once there is a problem of data skew, performance tuning will become extremely difficult.

[0063] In order to solve the problems of the prior art, the embodiment of the present application provides a hierarchical architecture of a big data warehouse based on Hive. Figure 1 As shown in the figure, the source database is the business system database, the operational data store (ODS) layer (using open source relational databases such as postgresql or mysql) collects data from multiple source databases, and its data model is basically consistent with the source database. The data warehouse layer is built based on hive and uses a star model established by dimensional modeling. The top layer is the application layer data, which is used to provide data for applications such as reporting systems and data analysis.

[0064] Based on the above-mentioned layered architecture of the Hive-based big data warehouse, the embodiment of the present application provides a dimensional data processing method, device, equipment and storage medium. The embodiment of the present application first pre-processes the dimension version of the acquired application system source database data on the ODS layer, and then synchronizes the pre-processed application system source database data to the hive layer, which can give full play to the advantages of the ODS layer and Hive, avoid a large amount of data processing in the Hive layer, and thus speed up the data processing speed. The following first introduces the dimensional data processing method provided in the embodiment of the present application.

[0065] Figure 2 FIG. 1 is a flow chart of a dimension data processing method provided by an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0066] S200, obtaining application system source database data.

[0067] In one embodiment, the application system source database data includes: slowly changing dimension values ​​and slowly changing dimension-related business data.

[0068] The ODS layer uses an open source relational application system source database (such as postgresql or mysql). First, the table structure of the data table that needs to be synchronized is established in the ODS layer. The table structure is kept consistent with the application system source database. The table name can be prefixed with different prefixes to distinguish it according to the source system. Then, the source database data is extracted and loaded into the ODS layer.

[0069] S202, loading the application system source database data into the operational data storage ODS layer.

[0070] In one embodiment, S202 may include:

[0071] Load the slowly changing dimension values ​​into the slowly changing dimension source table of the ODS layer; load the slowly changing dimension related business data into the related business table of the slowly changing dimension of the ODS layer.

[0072] S204, adding the dimension version field to the ODS layer to obtain target dimension data.

[0073] In one embodiment, S204 may include:

[0074] Add the dimension version field to the slowly changing dimension source table of the ODS layer and the related business table of the slowly changing dimension of the ODS layer to obtain the target dimension data.

[0075] Add a dimension version field to the slowly changing dimension source table, and keep the other fields unchanged; add a dimension version field to the related business table, and keep the other fields unchanged. If the related business table involves multiple slowly changing dimensions, it is necessary to add a dimension version field for each slowly changing dimension and distinguish them in the field naming.

[0076] S206, importing the target dimension data into the Hive layer.

[0077] In one embodiment, S206 may include:

[0078] Import the data stored in the slowly changing dimension source table of the ODS layer into the dimension table in the Hive layer, and import the data stored in the related business table of the slowly changing dimension of the ODS layer into the fact table in the Hive layer.

[0079] The data stored in the slowly changing dimension source table includes: slowly changing dimension value and dimension version field; the data stored in the related business table of the slowly changing dimension includes: slowly changing dimension related business data and dimension version field.

[0080] Before importing the target dimension data into the Hive layer, model and generate the slowly changing dimension table and the slowly changing dimension-related fact table at the Hive layer.

[0081] After completing the modeling of the Hive layer, extract the ODS layer data and load it into the dimension table and fact table of the Hive layer. The dimension version field has been processed in the ODS layer and is loaded into the Hive layer without any modification. Import the slowly changing dimension value and dimension version field into the dimension table so that the dimension value and dimension version field uniquely determine a row of information in the dimension table. After that, you can connect the dimension table with the fact table in the Hive layer through the dimension value and dimension version field.

[0082] In one embodiment, the method further comprises:

[0083] S208, updating the dimension version field of the slowly changing dimension source table and the dimension version field of the related business table according to the change of the slowly changing dimension value.

[0084] The dimension version fields of the slowly changing dimension source table are updated at the ODS layer according to the changes in the slowly changing dimension values, including:

[0085] The version field of the slowly changing dimension source table generates a number sequence from 1 to n in the dimension version field according to the order in which the slowly changing dimension value of each slowly changing dimension is generated. If a slowly changing dimension value has not changed, just fill in 1 in its dimension version field. If it changes once, fill in 2 in its dimension version field, and so on. This update logic can be flexibly handled by using technologies such as analytical functions or stored procedures of relational databases.

[0086] The dimension version fields of the related business tables that are updated at the ODS layer based on the changes in the slowly changing dimension values ​​include:

[0087] When the dimension version field of the slowly changing dimension source table has been updated, the version field of the related business table can be updated through the association relationship between the related business table and the slowly changing dimension source table.

[0088] To help you understand, the following example illustrates the role of the dimension version field in a slowly changing dimension:

[0089] Employee Name Employee dimension version field Responsible area Start Date End Date A 1 Region 1 2020-1-1 2020-6-30 B 1 Area 2 2020-1-1 9999-12-31 A 2 Area 3 2020-7-1 9999-12-31

[0090] Table 5 Employee dimension table

[0091]

[0092] Table 6 Order fact table

[0093] As shown in Table 5, the employee dimension table, the employee name plus the employee dimension version field uniquely identifies the employee whose responsible area has changed. In Table 6, the order fact table, the employee name and employee dimension version fields are saved for connection with the employee dimension table. When it is necessary to count the order amount by year and region, the corresponding information can be accurately counted.

[0094] The dimension data processing method provided in the embodiment of the present application redesigns the dimension table model: adds a dimension version field corresponding to the dimension value to the dimension table, and replans the data loading process: directly imports the dimension version field into the Hive layer after generating it in the ODS layer; at the same time, the complexity of the data loading data warehouse (Extract-Transform-Load, ETL) process of the surrogate key and the performance problem of connecting the large table of the snapshot dimension table to the large table are solved; compared with the method of processing dimension data by using the surrogate key and processing dimension data by using the snapshot dimension table in the prior art, it has the following advantages:

[0095] 1) Use surrogate keys to process slowly changing dimensions. Since surrogate keys are auto-incrementing sequences that span all dimension values, they are not conducive to incremental data processing. In addition, the generation of auto-incrementing sequences is not convenient for cross-database instances. Therefore, surrogate keys must be processed at the Hive layer. Processing surrogate keys at the Hive layer, especially loading fact tables, is a huge workload.

[0096] The embodiment of the present application uses the dimension version field to process slowly changing dimensions. Since the dimension version field is independently generated based on each dimension value, it can be placed in the ODS layer for flexible incremental processing, which greatly reduces the workload of Hive data processing.

[0097] 2) Use surrogate keys to process slowly changing dimensions. Once the dimension table data is damaged or lost, it is difficult to regenerate a surrogate key that is consistent with the original data. The entire data can only be reprocessed, and the fact table must also be reloaded.

[0098] The embodiment of the present application uses a dimension version field to process slowly changing dimensions. Even if the dimension table data is damaged or lost, the dimension version field can be easily regenerated according to its generation logic and can be associated with the fact table.

[0099] 3) Using snapshot dimension tables to process slowly changing dimensions requires generating snapshots of dimension tables every day, greatly increasing the frequency of data processing.

[0100] The embodiment of the present application uses the dimension version field to process the slowly changing dimension, and processes only when the dimension changes, and the processing frequency is low.

[0101] 4) Use snapshot dimension tables to process slowly changing dimensions. The amount of data in the dimension tables will grow too fast, and when using them, you will face the problem of connecting large tables. Once data skew occurs, it will be very difficult to optimize performance.

[0102] The embodiment of the present application uses the dimension version field to process the slowly changing dimension. The dimension table changes slowly, which greatly reduces its usage difficulty and the performance tuning difficulty is relatively low.

[0103] Figure 1-2The dimensional data processing method is described below. Figure 3 and attached Figure 4 Describe the device provided in the embodiment of the present application.

[0104] Figure 3 A schematic diagram of the structure of a dimensional data processing device provided by an embodiment of the present application is shown. Figure 3 Each module in the device shown has the function of realizing Figure 2 The functions of each step in the process can achieve the corresponding technical effects. Figure 3 As shown, the device may include:

[0105] The acquisition module 300 is used to acquire data from the source database of the application system.

[0106] In one embodiment, the application system source database data includes: slowly changing dimension values ​​and slowly changing dimension-related business data.

[0107] The ODS layer uses an open source relational application system source database (such as postgresql or mysql). First, the table structure of the data table that needs to be synchronized is established in the ODS layer. The table structure is kept consistent with the application system source database. The table name can be prefixed with different prefixes to distinguish it according to the source system. Then, the source database data is extracted and loaded into the ODS layer.

[0108] The loading module 302 is used to load the application system source database data into the operational data storage ODS layer.

[0109] In one embodiment, the loading module 302 may be specifically used for:

[0110] Load the slowly changing dimension values ​​into the slowly changing dimension source table of the ODS layer; load the slowly changing dimension related business data into the related business table of the slowly changing dimension of the ODS layer.

[0111] The adding module 304 is used to add the dimension version field to the ODS layer to obtain the target dimension data.

[0112] In one embodiment, the adding module 304 may be specifically used to:

[0113] Add the dimension version field to the slowly changing dimension source table of the ODS layer and the related business table of the slowly changing dimension of the ODS layer to obtain the target dimension data.

[0114] Add a dimension version field to the slowly changing dimension source table, and keep the other fields unchanged; add a dimension version field to the related business table, and keep the other fields unchanged. If the related business table involves multiple slowly changing dimensions, it is necessary to add a dimension version field for each slowly changing dimension and distinguish them in the field naming.

[0115] The import module 306 is used to import the target dimension data into the Hive layer.

[0116] In one embodiment, the import module 306 may be specifically used to:

[0117] Import the data stored in the slowly changing dimension source table of the ODS layer into the dimension table in the Hive layer, and import the data stored in the related business table of the slowly changing dimension of the ODS layer into the fact table in the Hive layer.

[0118] The data stored in the slowly changing dimension source table includes: slowly changing dimension value and dimension version field; the data stored in the related business table of the slowly changing dimension includes: slowly changing dimension related business data and dimension version field.

[0119] Before importing the target dimension data into the Hive layer, model and generate the slowly changing dimension table and the slowly changing dimension-related fact table at the Hive layer.

[0120] After completing the modeling of the Hive layer, extract the ODS layer data and load it into the dimension table and fact table of the Hive layer. The dimension version field has been processed in the ODS layer and is loaded into the Hive layer without any modification. Import the slowly changing dimension value and dimension version field into the dimension table so that the dimension value and dimension version field uniquely determine a row of information in the dimension table. After that, you can connect the dimension table with the fact table in the Hive layer through the dimension value and dimension version field.

[0121] In one embodiment, the apparatus further comprises:

[0122] The updating module 308 is used to update the dimension version field of the slowly changing dimension source table and the dimension version field of the related business table according to the change of the slowly changing dimension value.

[0123] The dimension version fields of the slowly changing dimension source table are updated at the ODS layer according to the changes in the slowly changing dimension values, including:

[0124] The version field of the slowly changing dimension source table generates a number sequence from 1 to n in the dimension version field according to the order in which the slowly changing dimension value of each slowly changing dimension is generated. If a slowly changing dimension value has not changed, just fill in 1 in its dimension version field. If it changes once, fill in 2 in its dimension version field, and so on. This update logic can be flexibly handled by using technologies such as analytical functions or stored procedures of relational databases.

[0125] The dimension version fields of the related business tables that are updated at the ODS layer based on the changes in the slowly changing dimension values ​​include:

[0126] When the dimension version field of the slowly changing dimension source table has been updated, the version field of the related business table can be updated through the association relationship between the related business table and the slowly changing dimension source table.

[0127] To help you understand, the following example illustrates the role of the dimension version field in a slowly changing dimension:

[0128] Employee Name Employee dimension version field Responsible area Start Date End Date A 1 Region 1 2020-1-1 2020-6-30 B 1 Area 2 2020-1-1 9999-12-31 A 2 Area 3 2020-7-1 9999-12-31

[0129] Table 7 Employee dimension table

[0130]

[0131] Table 8 Order fact table

[0132] As shown in Table 7, the employee dimension table, the employee name plus the employee dimension version field uniquely identifies the employee whose responsible area has changed. In Table 8, the order fact table, the employee name and employee dimension version fields are saved for connection with the employee dimension table. When it is necessary to count the order amount by year and region, the corresponding information can be accurately counted.

[0133] The dimension data processing device provided in the embodiment of the present application redesigns the dimension table model: adds a dimension version field corresponding to the dimension value to the dimension table, and replans the data loading process: directly imports the dimension version field into the Hive layer after generating it in the ODS layer; at the same time, the complexity of the data loading data warehouse (Extract-Transform-Load, ETL) process of the surrogate key and the performance problem of connecting the large table of the snapshot dimension table to the large table are solved; compared with the method of processing dimension data by using the surrogate key and processing dimension data by using the snapshot dimension table in the prior art, it has the following advantages:

[0134] 1) Use surrogate keys to process slowly changing dimensions. Since surrogate keys are auto-incrementing sequences that span all dimension values, they are not conducive to incremental data processing. In addition, the generation of auto-incrementing sequences is not convenient for cross-database instances. Therefore, surrogate keys must be processed at the Hive layer. Processing surrogate keys at the Hive layer, especially loading fact tables, is a huge workload.

[0135] The embodiment of the present application uses the dimension version field to process slowly changing dimensions. Since the dimension version field is independently generated based on each dimension value, it can be placed in the ODS layer for flexible incremental processing, which greatly reduces the workload of Hive data processing.

[0136] 2) Use surrogate keys to process slowly changing dimensions. Once the dimension table data is damaged or lost, it is difficult to regenerate a surrogate key that is consistent with the original data. The entire data can only be reprocessed, and the fact table must also be reloaded.

[0137] The embodiment of the present application uses a dimension version field to process slowly changing dimensions. Even if the dimension table data is damaged or lost, the dimension version field can be easily regenerated according to its generation logic and can be associated with the fact table.

[0138] 3) Using snapshot dimension tables to process slowly changing dimensions requires generating snapshots of dimension tables every day, greatly increasing the frequency of data processing.

[0139] The embodiment of the present application uses the dimension version field to process the slowly changing dimension, and processes only when the dimension changes, and the processing frequency is low.

[0140] 4) Use snapshot dimension tables to process slowly changing dimensions. The amount of data in the dimension tables will grow too fast, and when using them, you will face the problem of connecting large tables. Once data skew occurs, it will be very difficult to optimize performance.

[0141] The embodiment of the present application uses the dimension version field to process the slowly changing dimension. The dimension table changes slowly, which greatly reduces its usage difficulty and the performance tuning difficulty is relatively low.

[0142] Figure 4 FIG. 1 is a schematic diagram showing the structure of a computer device provided by an embodiment of the present application. Figure 4 As shown, the device may include a processor 401 and a memory 402 storing computer program instructions.

[0143] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0144] The memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In one example, the memory 402 may include a removable or non-removable (or fixed) medium, or the memory 402 is a non-volatile solid-state memory. The memory 402 may be inside or outside the integrated gateway disaster recovery device.

[0145] In one example, the memory 402 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0146] The processor 401 reads and executes the computer program instructions stored in the memory 402 to implement Figure 2 The method in the embodiment shown in the figure is achieved Figure 2 The corresponding technical effects achieved by executing the method in the example shown are not repeated here for the sake of brevity.

[0147] In one example, the computer device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0148] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0149] Bus 410 includes hardware, software or both, and the components of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (Accelerated Graphics Port, AGP) or other graphics bus, enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) bus, front side bus (Front Side Bus, FSB), Hyper Transport (Hyper Transport, HT) interconnection, industry standard architecture (Industry Standard Architecture, ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0150] The computer device can execute the dimensional data processing method in the embodiment of the present application, thereby achieving Figure 2The corresponding technical effects of the described dimensional data processing method.

[0151] In addition, in combination with the dimensional data processing method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the dimensional data processing methods in the above embodiment is implemented.

[0152] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0153] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0154] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0155] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0156] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A dimensional data processing method, characterized in that: include: Acquire application system source database data, wherein the application system source database data includes: slowly changing dimension values ​​and slowly changing dimension-related business data; Loading the application system source database data into the operational data storage ODS layer; Add the dimension version field to the ODS layer to obtain the target dimension data; Importing the target dimension data into the Hive layer, and connecting the slowly changing dimension table of the Hive layer with the slowly changing dimension-related fact table through the dimension value and the dimension version field in the Hive layer; The step of adding the dimension version field to the ODS layer to obtain target dimension data includes: Adding the dimension version field to the slowly changing dimension source table of the ODS layer and the related business table of the slowly changing dimension of the ODS layer respectively, to obtain the target dimension data; The method further includes: updating, according to the change of the slowly changing dimension value: the dimension version field of the slowly changing dimension source table and the dimension version field of the related business table.

2. The dimensional data processing method according to claim 1, characterized in that: The step of loading the application system source database data into the operational data storage ODS layer includes: Loading the slowly changing dimension value into the slowly changing dimension source table of the ODS layer; The business data related to the slowly changing dimension is loaded into the business table related to the slowly changing dimension of the ODS layer.

3. The dimensional data processing method according to claim 1, characterized in that: The step of importing the target dimension data into the Hive layer includes: The data stored in the slowly changing dimension source table of the ODS layer is imported into the slowly changing dimension table in the Hive layer, and the data stored in the business table related to the slowly changing dimension of the ODS layer is imported into the fact table related to the slowly changing dimension in the Hive layer.

4. A dimensional data processing device, characterized in that: include: An acquisition module, used to acquire application system source database data, wherein the application system source database data includes: slowly changing dimension values ​​and slowly changing dimension-related business data; A loading module, used for loading the application system source database data into the operational data storage ODS layer; Add module to add dimension version field to ODS layer to obtain target dimension data; An import module, used for importing the target dimension data into a Hive layer, and connecting the slowly changing dimension table of the Hive layer with a slowly changing dimension-related fact table through the dimension value and the dimension version field in the Hive layer; The adding module is further used to add the dimension version field to the slowly changing dimension source table of the ODS layer and the related business table of the slowly changing dimension of the ODS layer respectively, to obtain the target dimension data; An updating module is used to update, according to the change of the slowly changing dimension value: the dimension version field of the slowly changing dimension source table and the dimension version field of the related business table.

5. A computer device, characterized in that: The computer device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the dimensional data processing method according to any one of claims 1 to 3 when executed by the processor.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the dimensional data processing method according to any one of claims 1 to 3 is implemented.

Images