Data warehouse model optimization method and related equipment thereof

Abstract

The invention relates to the technical field of big data, and discloses a data warehouse model optimization method and related equipment thereof, and the method comprises the steps: obtaining a data scheduling task based on a big data platform, and determining whether the data scheduling task is a published task or not; when the data scheduling task is a published task, obtaining blood relationship data of the published task, and analyzing the blood relationship data based on a big data platform to obtain first data information; determining whether a data table which is repeatedly used exists in the first data information, and when it is determined that the data table exists, tracing the data table to obtain a source layer of the data table in the data warehouse model; and obtaining a target processing task according to the source layer, and performing optimization processing on a first data layer table corresponding to the summary data layer, a second data layer table corresponding to the detail data layer and a third data layer table corresponding to the dimension data layer in the data warehouse model through the target processing task. According to the method, accurate optimization of the data warehouse model is realized, repeated data construction is avoided, and data redundancy is reduced.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data warehouse model optimization method and related equipment. Background Art

[0002] A data warehouse is a structured data environment for decision support systems and online analytical application data sources. Through a data warehouse, the problem of obtaining information from a database can be studied and solved.

[0003] As the complexity of business increases, the modeling of data warehouses has gradually increased, but the accuracy of data warehouse modeling is low. In historical data warehouses, there is often repeated modeling of a large amount of data. A large amount of repeated data will lead to data redundancy in the data warehouse and inconsistent data caliber; in addition, the maintenance and management costs of the data warehouse will also increase, occupying a large amount of computing resources, and ultimately leading to the problem of inefficient data management. Summary of the invention

[0004] The embodiments of the present application provide a data warehouse model optimization method and related equipment, which can solve the technical problem of low accuracy of data warehouse models.

[0005] The present application provides a data warehouse model optimization method, including:

[0006] Acquire a data scheduling task based on the big data platform, and determine whether the data scheduling task is a published task;

[0007] When the data scheduling task is the published task, obtaining the lineage data of the published task, parsing the lineage data based on the big data platform, and obtaining first data information;

[0008] Determine whether there is a reused data table in the first data information, and when it is determined that the data table exists, trace the source of the data table to obtain the source layer of the data table in the data warehouse model;

[0009] A target processing task is obtained according to the source layer, and a first data layer table corresponding to the summary data layer, a second data layer table corresponding to the detail data layer, and a third data layer table corresponding to the dimension data layer in the data warehouse model are optimized through the target processing task.

[0010] Further, a target processing task is obtained according to the source layer, and a first data layer table corresponding to the summary data layer, a second data layer table corresponding to the detail data layer, and a third data layer table corresponding to the dimension data layer in the data warehouse model are optimized through the target processing task, including:

[0011] When the source layer of the high-frequency table is a data mart layer or an original data layer, obtaining field information, and determining the field missing ranges corresponding to the first data layer table, the second data layer table, and the third data layer table respectively according to the field information;

[0012] Determine the type of the target processing task according to the missing field range, and optimize the first data layer table, the second data layer table, and the third data layer table respectively according to the task items corresponding to the type;

[0013] When the source layer of the data table is the summary data layer or the detail data layer, the target processing task is determined to be a retention task, and the first data layer table, the second data layer table, and the third data layer table are retained.

[0014] Further, the type of the target processing task is determined according to the missing field range, and the first data layer table, the second data layer table, and the third data layer table are optimized respectively according to the task items corresponding to the type, including:

[0015] When the field missing range is the first range, determining that the type of the target processing task is a completion type, and completing the missing fields in the first data layer table, the second data layer table, and the third data layer table based on the task items corresponding to the completion type;

[0016] When the field missing range is the second range, the type of the target processing task is determined to be a newly created type, and based on the task items corresponding to the newly created type, layer tables corresponding to a new detailed data layer, a new summary data layer, and a new dimensional data layer are sequentially constructed from the original data layer of the data warehouse model, wherein the first range is smaller than the second range.

[0017] Furthermore, when the source layer of the data table is the data mart layer or the original data layer, obtaining the field information includes:

[0018] When the source layer of the data table is the data mart layer, tracing back the original data layer table corresponding to the data table through the first data information to obtain the field information;

[0019] When the source layer of the data table is the original data layer, the target field content of the original data layer table in the original data layer is read to obtain the field information.

[0020] Further, after optimizing the first data layer table corresponding to the summary data layer, the second data layer table corresponding to the detail data layer, and the third data layer table corresponding to the dimension data layer in the data warehouse model through the target processing task, the method includes:

[0021] Obtaining an optimized first data layer table, an optimized second data layer table, and an optimized third data layer table, and pushing the optimized first data layer table, the optimized second data layer table, and the optimized third data layer table to a target user terminal based on the big data platform;

[0022] The tasks repeatedly submitted by the target user terminal are replaced with the optimized first data layer table, the optimized second data layer table and the optimized third data layer table.

[0023] Further, after determining whether the data scheduling task is a published task, the following steps are performed:

[0024] When it is determined that the data scheduling task is a task being published, the task being published is analyzed in real time to obtain second data information of the task being published;

[0025] Determine, based on the second data information, the missing ranges of the fields corresponding to the first data layer table of the summary data layer table, the second data layer table of the detail data layer table, and the third data layer table of the dimension data layer table in the current data warehouse model;

[0026] Corresponding processing items are determined according to the field missing range, and the first data layer table, the second data layer table, and the third data layer table are optimized based on the processing items.

[0027] Further, optimizing the first data layer table, the second data layer table, and the third data layer table based on the processing items includes:

[0028] If the field missing range is the minimum range, determining that the processing item is an extraction item, extracting the first data layer table, the second data layer table and the third data layer table from the current data warehouse model to the target user terminal based on the extraction item;

[0029] If the field missing range is not the minimum range, the processing item is determined to be an optimization item, and the first data layer table, the second data layer table and the third data layer table are optimized.

[0030] Accordingly, an embodiment of the present application provides a data warehouse model optimization device, including:

[0031] A confirmation module is used to obtain a data scheduling task based on the big data platform and determine whether the data scheduling task is a published task;

[0032] A parsing module, used for obtaining the lineage data of the published task when the data scheduling task is the published task, parsing the lineage data based on the big data platform, and obtaining the first data information;

[0033] A source tracing module is used to determine whether there is a reused data table in the first data information, and when it is determined that the data table exists, trace the data table to obtain the source layer of the data table in the data warehouse model;

[0034] The optimization module is used to obtain the target processing task according to the source layer, and optimize the first data layer table corresponding to the summary data layer, the second data layer table corresponding to the detailed data layer, and the third data layer table corresponding to the dimension data layer in the data warehouse model through the target processing task.

[0035] In addition, an embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the data warehouse model optimization method provided in the embodiment of the present application.

[0036] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for loading by a processor to execute any data warehouse model optimization method provided in the embodiment of the present application.

[0037] In addition, an embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any data warehouse model optimization method provided in the embodiment of the present application.

[0038] In an embodiment of the present application, a data scheduling task is obtained based on a big data platform to determine whether the data scheduling task is a published task; when the data scheduling task is a published task, the lineage data of the published task is obtained, and the lineage data is parsed based on the big data platform to obtain first data information, so that the published task can be post-processed through the lineage data to reduce data redundancy; then, it is determined whether there is a reused data table in the first data information, and when it is determined that there is a data table, the data table is traced to obtain the source layer of the data table in the data warehouse model, so that the missing or repeated fields in the data warehouse model layer table can be determined through the source layer; then, the target processing task is obtained according to the source layer, and the first data layer table corresponding to the summary data layer, the second data layer table corresponding to the detail data layer, and the third data layer table corresponding to the dimension data layer in the data warehouse model are optimized through the target processing task, thereby realizing accurate optimization of the data warehouse model, avoiding duplicate construction of data, and reducing data redundancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 It is a flowchart of a data warehouse model optimization method provided in an embodiment of the present application;

[0041] Figure 2 It is a structural diagram of a data warehouse model optimization device provided in an embodiment of the present application;

[0042] Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0044] The embodiment of the present application provides a data warehouse model optimization method, device, electronic device and computer storage medium. The data warehouse model optimization device can be integrated in an electronic device, which can be a server or a terminal.

[0045] Among them, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0046] The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0047] In addition, the term "plurality" in the embodiments of the present application refers to two or more than two. The terms "first" and "second" in the embodiments of the present application are used to distinguish descriptions and should not be understood as implying relative importance.

[0048] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0049] See also Figure 1 , Figure 1 : is a flow chart of a data warehouse model optimization method provided by an embodiment of the present application. The data warehouse model optimization method may include:

[0050] S101. Obtain a data scheduling task based on a big data platform, and determine whether the data scheduling task is a published task.

[0051] In this embodiment, the big data platform is a big data processing platform, through which data can be processed and handled. Based on the big data platform, a data scheduling task is obtained, and it is determined whether the data scheduling task is a published task. Specifically, when the data scheduling task is obtained, the task status of the data scheduling task is obtained. If the task status is a completed state, the data scheduling task is determined to be a published task; if the task status is a published state, the data scheduling task is determined to be a publishing task.

[0052] S102. When the data scheduling task is the published task, obtain the lineage data of the published task, and parse the lineage data based on the big data platform to obtain first data information.

[0053] In this embodiment, when the data scheduling task is a published task, the lineage data corresponding to the published task is obtained. Specifically, based on the big data platform, the lineage data of the table, task, and field are collected in real time or at a scheduled time. When it is determined that the data scheduling task of the big data platform is a published task, the lineage data collected by the published task on the big data platform is obtained; the lineage data is the linked-in data generated by the table, task, and field during processing. Among them, the task is a task that is automatically scheduled by the big data platform according to the day, week, month, hour, minute, and second fields. The task mainly includes SQL (Structured Query Language) code, through which the SQL code can be used to logically process and produce related tables; the table is a hive table, through which the data on Hadoop can be managed and processed, and the structured data can be mapped to the Hadoop distributed file system or other data storage system, so that the data storage system can use SQL queries to access and process the structured data; the field represents the data and attributes of a column in the database table; the lineage is the linked-in data of the task and table during the processing process, and the table and task can be traced through the linked-in data. When obtaining the lineage data corresponding to the data scheduling task, the SQL code of the task in the lineage data is parsed based on the big data platform to obtain the first data information corresponding to the published task. The first data information specifically includes the source table, source field, filtering conditions, association conditions, target table and other information of the published task.

[0054] S103: Determine whether there is a reused data table in the first data information. When it is determined that the data table exists, trace the source of the data table to obtain the source layer of the data table in the data warehouse model.

[0055] In this embodiment, when the first data information is obtained, it is determined whether there are reused data tables in the first data information, such as tables that appear in pairs in the first data information. If there are reused data tables, it means that multiple organizations use the table. When it is determined that the data table exists in the first data information, the data table is traced to obtain the source layer of the data table in the data warehouse model. Among them, the source layer of the data warehouse model mainly includes the ODS layer (Operational Data Store, original data layer), DWD layer (Data Warehouse Detail, detailed data layer), DWS layer (Data Warehouse Summary, summary data layer), DM layer (Data Mart, data mart layer), DIM layer (Dimension, dimension data layer) and TMP layer (Temporary Data Layer, temporary data layer) according to the data flow. According to the first data information, the data table is traced to determine the source layer of the data table in the data warehouse model.

[0056] S104. Obtain a target processing task according to the source layer, and optimize the first data layer table corresponding to the summary data layer, the second data layer table corresponding to the detail data layer, and the third data layer table corresponding to the dimension data layer in the data warehouse model through the target processing task.

[0057] In this embodiment, when determining the source layer of the data table in the data warehouse model, the target processing task is obtained according to the source layer, and different source layers may correspond to different target processing tasks. Specifically, the target processing task can be divided into a maintenance task and an optimization task according to the task type. When the source layer of the data table is the summary data layer or the detailed data layer, the target processing task is determined to be a maintenance task; when the source layer of the data table is the data mart layer or the original data layer, the target processing task is determined to be an optimization task. Among them, the maintenance task is to maintain the original detailed data layer table, the original summary data layer table, and the original dimension data layer table in the current data warehouse model. The optimization task is to perform related optimization operations on the original detailed data layer table, the original summary data layer table, and the original dimension data layer table in the current data warehouse model, such as field completion and layer table creation. In this way, the optimization of the data warehouse model is finally achieved.

[0058] The data warehouse model optimization method proposed in this application realizes the precise optimization of the data warehouse model, avoids the duplication of data construction, and reduces data redundancy.

[0059] In some embodiments of the present application, the above-mentioned optimization processing of the first data layer table corresponding to the summary data layer, the second data layer table corresponding to the detail data layer, and the third data layer table corresponding to the dimension data layer in the data warehouse model through the target processing task includes:

[0060] When the source layer of the high-frequency table is a data mart layer or an original data layer, obtaining field information, and determining the field missing ranges corresponding to the first data layer table, the second data layer table, and the third data layer table respectively according to the field information;

[0061] Determine the type of the target processing task according to the missing field range, and optimize the first data layer table, the second data layer table, and the third data layer table respectively according to the task items corresponding to the type;

[0062] When the source layer of the data table is the summary data layer or the detail data layer, the target processing task is determined to be a retention task, and the first data layer table, the second data layer table, and the third data layer table are retained.

[0063] In this embodiment, the source layer of the data warehouse model includes a data mart layer, a summary data layer, a detailed data layer, an original data layer, and a dimensional data layer, wherein the detailed data layer has the same granularity as the original data layer, and the data is cleaned, converted, and completed through the detailed data layer; the summary data layer performs light aggregation of subject data based on the detailed fact data and model dimension data of the detailed data layer. The data mart layer is mainly used to construct subject data based on business themes, and the data in the data mart layer is usually highly aggregated or highly aggregated data; the original data layer follows data security specifications and standards, and is generally synchronized from the business system in an incremental or full manner, and historical data is maintained in a partitioned manner to achieve long-term data preservation.

[0064] When the source layer of the data table is the data mart layer or the original data layer, the preset field information and the field information corresponding to the layer table included in the current data warehouse model are obtained. Specifically, the preset field information is the field information of all layer tables included in the standard data warehouse model, and the field information corresponding to the layer table included in the current data warehouse model is the field information of the original data layer of the current data warehouse model. By comparing the preset field information corresponding to the summary data layer in the standard data warehouse model with the field information of the summary data layer of the current data warehouse model, the field missing range corresponding to the first data layer table can be determined; by comparing the preset field information corresponding to the detail data layer in the standard data warehouse model with the field information of the detail data layer of the current data warehouse model, the field missing range corresponding to the second data layer table can be determined; by comparing the preset field information corresponding to the dimension data layer in the standard data warehouse model with the field information of the dimension data layer of the current data warehouse model, the field missing range corresponding to the third data layer table can be determined. In addition, only the field information of the current data warehouse model can be obtained, and the field missing ranges corresponding to the first data layer table, the second data layer table and the third data layer table can be determined according to the field source type included in the field information. The type of target processing task is further determined according to the scope of field missing, and the first data layer table, the second data layer table and the third data layer table are optimized respectively through the task items corresponding to each type.

[0065] When the source layer of the data table is the summary data layer or the detailed data layer, it means that all the field contents corresponding to the data table have been modeled in the current data warehouse model, and there is no need to re-create the summary data layer or the detailed data layer. Therefore, when the source layer of the data table is the summary data layer or the detailed data layer, the target processing task is determined to be a retention task, and the first data layer table, the second data layer table, and the third data layer table in the current data warehouse model are retained.

[0066] This embodiment obtains different source layers of the data table in the historical modeling process to determine the corresponding target processing tasks, thereby avoiding repeated modeling of the data table and reducing data redundancy.

[0067] In some embodiments of the present application, the above-mentioned determining the type of the target processing task according to the field missing range, and optimizing the first data layer table, the second data layer table, and the third data layer table respectively according to the task items corresponding to the type include:

[0068] When the field missing range is the first range, determining that the type of the target processing task is a completion type, and completing the missing fields in the first data layer table, the second data layer table, and the third data layer table based on the task items corresponding to the completion type;

[0069] When the field missing range is the second range, the type of the target processing task is determined to be a newly created type, and based on the task items corresponding to the newly created type, layer tables corresponding to a new detailed data layer, a new summary data layer, and a new dimensional data layer are sequentially constructed from the original data layer of the data warehouse model, wherein the first range is smaller than the second range.

[0070] In this embodiment, the specific optimization content corresponding to the current target processing task can be determined through the field missing ranges of the first data layer table, the second data layer table, and the third data layer table in the data warehouse model. Specifically, the field missing range includes a first range and a second range, wherein the first range is smaller than the second range. If the source layer of the data table is the data mart layer or the original data layer, the data table is traced according to the first data information to determine whether the field content of the data table has been completely covered by the first data layer table, the second data layer table, and the third data layer table in the data warehouse model. If the field content corresponding to the data table only contains part of the field content of the first data layer table, the field missing range of the first data layer table is determined to be the first range; if the field content corresponding to the data table only contains part of the field content of the second data layer table, the field missing range of the second data layer table is determined to be the first range; if the field content corresponding to the data table only contains part of the field content of the third data layer table, the field missing range of the third data layer table is determined to be the first range. For example, if the data table corresponds to 3 field contents, and only two fields are covered in the first data layer table, that is, the field contents of the first data layer table only include two fields, it means that the first data layer table is not completely covered, and the field missing range of the first data layer table is the first range. If the field contents of the data table do not contain the field contents of the first data layer table at all, the field missing range of the first data layer table is determined to be the second range; if the field contents of the data table do not contain the field contents of the second data layer table at all, the field missing range of the second data layer table is determined to be the second range; if the field contents of the data table do not contain the field contents of the third data layer table at all, the field missing range of the third data layer table is determined to be the second range. The coverage of the data table in each layer table can be determined based on the acquired field information. When the field missing range of each data layer table is the first range, the target processing task of the data layer table is determined to be the completion task, and the missing fields in the corresponding data layer table are completed according to the completion task. When the missing range of the field is the second range, the optimization task is determined to be a new task, and layer tables corresponding to a new detailed data layer, a new summary data layer, and a new dimension data layer are constructed respectively from the original data layer according to the new task.

[0071] This embodiment determines the specific optimization task corresponding to the target processing task through the field missing range, so that the layer table in the data warehouse model can be accurately optimized through the specific optimization task, thereby improving the optimization accuracy of the data warehouse model.

[0072] In some embodiments of the present application, when the source layer of the data table is the data mart layer or the original data layer, obtaining the field information includes:

[0073] When the source layer of the data table is the data mart layer, tracing back the original data layer table corresponding to the data table through the first data information to obtain the field information;

[0074] When the source layer of the data table is the original data layer, the target field content of the original data layer table in the original data layer is read to obtain the field information.

[0075] In this embodiment, the field information is the field information of the original data layer, and the missing range of the fields corresponding to the data table in the first data layer table, the second data layer table and the third data layer table can be determined through the field information. When the source layer of the data table is the data mart layer, the field situation of the original data layer table corresponding to the data mart layer is traced back through the first data information to obtain the field information; when the source layer of the data table is the original data layer, the target field content of the original data layer table in the current original data layer is directly obtained to obtain the field information.

[0076] This embodiment achieves tracing of the field content of the data table by obtaining the field information of the original data layer table, so that the field information of the original data layer table can be used to accurately determine the missing fields of the current data table in each layer table in the data warehouse model, further improving the efficiency of data warehouse model optimization.

[0077] In some embodiments of the present application, after the first data layer table corresponding to the summary data layer, the second data layer table corresponding to the detail data layer, and the third data layer table corresponding to the dimension data layer in the data warehouse model are optimized by the target processing task, the following steps are included:

[0078] Obtaining an optimized first data layer table, an optimized second data layer table, and an optimized third data layer table, and pushing the optimized first data layer table, the optimized second data layer table, and the optimized third data layer table to a target user terminal based on the big data platform;

[0079] The tasks repeatedly submitted by the target user terminal are replaced with the optimized first data layer table, the optimized second data layer table and the optimized third data layer table.

[0080] In this embodiment, when the optimization processing of the first data layer table, the second data layer table and the third data layer table in the data warehouse model is completed, the optimized first data layer table, the optimized second data layer table and the optimized third data layer table are obtained, and the optimized first data layer table, the optimized second data layer table and the optimized third data layer table are pushed to the target user terminal based on the big data platform, and the tasks repeatedly submitted by the target user terminal are replaced with the optimized first data layer table, the optimized second data layer table and the optimized third data layer table. Among them, the repeatedly submitted tasks are the processing tasks corresponding to the data table repeatedly submitted by the target user terminal.

[0081] This embodiment avoids repeated modeling of tasks and improves the modeling efficiency of the data warehouse by replacing repeatedly submitted tasks with an optimized first data layer table, an optimized second data layer table, and an optimized third data layer table.

[0082] In some embodiments of the present application, after determining whether the data scheduling task is a published task, the following steps are included:

[0083] When it is determined that the data scheduling task is a task being published, the task being published is analyzed in real time to obtain second data information of the task being published;

[0084] According to the second data information, determining the missing ranges of the fields respectively corresponding to the first data layer table of the summary data layer table, the second data layer table of the detail data layer table, and the third data layer table of the dimension data layer table in the current data warehouse model;

[0085] Corresponding processing items are determined according to the field missing range, and the first data layer table, the second data layer table, and the third data layer table are optimized based on the processing items.

[0086] In this embodiment, in addition to optimizing the data warehouse modeling of published tasks, it is also possible to optimize the modeling of tasks in progress. Specifically, when a SQL (Structured Query Language) task submitted by a target user terminal is received, the SQL task is determined to be a task in progress. The task in progress is parsed in real time to obtain the second data information of the task in progress. The second data information includes information such as the source table, source field, filter conditions, association conditions, target table, and traceability table of the task in progress. According to the second data information, the task in progress is traced to determine the missing range of fields in the first data layer table, the second data layer table, and the third data layer table of the task in progress in the data warehouse model, and the processing items corresponding to the task in progress are determined according to the missing range of the field, and the first data layer table, the second data layer table, and the third data layer table are optimized based on the processing items.

[0087] This embodiment performs real-time monitoring of tasks in progress and determines whether there are any data layer tables that have been modeled in the data warehouse model based on the table coverage. This achieves efficient real-time monitoring of tasks in progress, avoids duplicate data construction, reduces computing resources, and improves the reuse of data warehouse models.

[0088] In some embodiments of the present application, the above-mentioned optimizing the first data layer table, the second data layer table and the third data layer table based on the processing items includes:

[0089] If the field missing range is the minimum range, determining that the processing item is an extraction item, extracting the first data layer table, the second data layer table and the third data layer table from the current data warehouse model to the target user terminal based on the extraction item;

[0090] If the field missing range is not the minimum range, the processing item is determined to be an optimization item, and the first data layer table, the second data layer table and the third data layer table are optimized.

[0091] In this embodiment, if the tasks in the release can be traced back to the field contents corresponding to the first data layer table, the second data layer table and the third data layer table respectively, then it is determined that the missing range of the fields corresponding to the first data layer table, the second data layer table and the third data layer table respectively is the minimum range, that is, the first data layer table, the second data layer table and the third data layer table are all covered. When the missing range of the fields corresponding to each layer table is the minimum range, it is determined that the current processing item is an extraction item, and based on the extraction item, the first data layer table, the second data layer table and the third data layer table corresponding to the task in the release are extracted from the current data warehouse model to the target user terminal. If the task in the release cannot obtain the complete field content corresponding to any of the first data layer table, the second data layer table and the third data layer table in the data warehouse model, it is determined that the missing range of the fields is not the minimum range, and the current processing item is determined to be an optimization item, and the layer tables with missing fields in the first data layer table, the second data layer table and the third data layer table are optimized based on the optimization item.

[0092] This embodiment corresponds to different processing items through different table coverage ranges, so that the first data layer table, the second data layer table and the third data layer table of the data warehouse model can be processed differently through different processing items, thereby improving the efficiency of data warehouse modeling for in-process tasks.

[0093] In order to better implement the data warehouse model optimization method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above data warehouse model optimization method. The meanings of the terms are the same as those in the above data warehouse model optimization method, and the specific implementation details can refer to the description in the method embodiment.

[0094] For example, Figure 2 As shown, the data warehouse model optimization device may include: a confirmation module 201, a parsing module 202, a tracing module 203 and an optimization module 204.

[0095] Confirmation module 201, used to obtain a data scheduling task based on the big data platform and determine whether the data scheduling task is a published task;

[0096] The parsing module 202 is used for obtaining the lineage data of the published task when the data scheduling task is the published task, and parsing the lineage data based on the big data platform to obtain the first data information;

[0097] A source tracing module 203 is used to determine whether there is a reused data table in the first data information, and when it is determined that the data table exists, trace the data table to obtain the source layer of the data table in the data warehouse model;

[0098] The optimization module 204 is used to obtain the target processing task according to the source layer, and optimize the first data layer table corresponding to the summary data layer, the second data layer table corresponding to the detail data layer, and the third data layer table corresponding to the dimension data layer in the data warehouse model through the target processing task.

[0099] In one embodiment of the present application, the optimization module 204 includes:

[0100] A first confirmation unit is used to obtain field information when the source layer of the high-frequency table is a data mart layer or an original data layer, and determine the field missing ranges corresponding to the first data layer table, the second data layer table, and the third data layer table respectively according to the field information;

[0101] A first optimization unit is used to determine the type of the target processing task according to the field missing range, and optimize the first data layer table, the second data layer table and the third data layer table respectively according to the task items corresponding to the type;

[0102] The second optimization unit is used to determine that the target processing task is a retention task when the source layer of the data table is the summary data layer or the detail data layer, and to retain the first data layer table, the second data layer table and the third data layer table.

[0103] In one embodiment of the present application, the first confirmation unit includes:

[0104] a tracing unit, configured to, when the source layer of the data table is the data mart layer, trace back the original data layer table corresponding to the data table through the first data information to obtain the field information;

[0105] The reading unit is used to read the target field content of the original data layer table in the original data layer to obtain the field information when the source layer of the data table is the original data layer.

[0106] In one embodiment of the present application, the first optimization unit includes:

[0107] a completion unit, configured to, when the missing range of the field is a first range, determine that the type of the target processing task is a completion type, and complete the missing fields in the first data layer table, the second data layer table, and the third data layer table respectively based on the task items corresponding to the completion type;

[0108] A construction unit is used to determine that the type of the target processing task is a newly created type when the field missing range is a second range, and to construct layer tables corresponding to a new detailed data layer, a new summary data layer, and a new dimensional data layer from the original data layer of the data warehouse model in sequence based on task items corresponding to the newly created type, wherein the first range is smaller than the second range.

[0109] In one embodiment of the present application, the data warehouse model optimization device further includes:

[0110] A first push module is used to obtain an optimized first data layer table, an optimized second data layer table, and an optimized third data layer table, and push the optimized first data layer table, the optimized second data layer table, and the optimized third data layer table to a target user terminal based on the big data platform;

[0111] The second push module is used to replace the tasks repeatedly submitted by the target user terminal with the optimized first data layer table, the optimized second data layer table and the optimized third data layer table.

[0112] In one embodiment of the present application, the data warehouse model optimization device further includes:

[0113] A parsing unit, configured to, when determining that the data scheduling task is a task being published, perform real-time parsing on the task being published to obtain second data information of the task being published;

[0114] A second confirmation unit is used to determine, according to the second data information, the missing ranges of the fields corresponding to the first data layer table of the summary data layer table, the second data layer table of the detail data layer table, and the third data layer table of the dimension data layer table in the current data warehouse model;

[0115] The third optimization unit is used to determine corresponding processing items according to the field missing range, and optimize the first data layer table, the second data layer table and the third data layer table based on the processing items.

[0116] In one embodiment of the present application, the third optimization unit includes:

[0117] an extraction unit, configured to determine that the processing item is an extraction item if the field missing range is the minimum range, and extract the first data layer table, the second data layer table and the third data layer table from the current data warehouse model to the target user terminal based on the extraction item;

[0118] The fourth optimization unit is used to determine that the processing item is an optimization item if the field missing range is not the minimum range, and optimize the first data layer table, the second data layer table and the third data layer table.

[0119] The data warehouse model optimization device proposed in this application realizes the accurate optimization of the data warehouse model, avoids the duplication of data construction, and reduces data redundancy.

[0120] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation methods and corresponding beneficial effects of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0121] The present application also provides an electronic device, which may be a server or a terminal, etc. Figure 3 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0122] The electronic device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will appreciate that Figure 3 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0123] The processor 601 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing computer programs and / or modules stored in the memory 602, and calling data stored in the memory 602. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.

[0124] The memory 602 can be used to store computer programs and modules. The processor 601 executes various functional applications and data processing by running the computer programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, a computer program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0125] The electronic device also includes a power supply 603 for supplying power to each component. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 603 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0126] The electronic device may further include an input unit 604, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0127] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 601 in the electronic device will load the executable files corresponding to the processes of one or more computer programs into the memory 602 according to the following instructions, and the processor 601 will run the computer programs stored in the memory 602, thereby realizing various functions, such as:

[0128] Acquire a data scheduling task based on the big data platform, and determine whether the data scheduling task is a published task;

[0129] When the data scheduling task is the published task, obtaining the lineage data of the published task, parsing the lineage data based on the big data platform, and obtaining first data information;

[0130] Determine whether there is a reused data table in the first data information, and when it is determined that the data table exists, trace the source of the data table to obtain the source layer of the data table in the data warehouse model;

[0131] A target processing task is obtained according to the source layer, and a first data layer table corresponding to the summary data layer, a second data layer table corresponding to the detail data layer, and a third data layer table corresponding to the dimension data layer in the data warehouse model are optimized through the target processing task.

[0132] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the detailed description of the data warehouse model optimization method above, which will not be elaborated here.

[0133] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0134] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and the computer program can be loaded by a processor to execute the steps in any data warehouse model optimization method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0135] Acquire a data scheduling task based on the big data platform, and determine whether the data scheduling task is a published task;

[0136] When the data scheduling task is the published task, obtaining the lineage data of the published task, parsing the lineage data based on the big data platform, and obtaining first data information;

[0137] Determine whether there is a reused data table in the first data information, and when it is determined that the data table exists, trace the source of the data table to obtain the source layer of the data table in the data warehouse model;

[0138] A target processing task is obtained according to the source layer, and a first data layer table corresponding to the summary data layer, a second data layer table corresponding to the detail data layer, and a third data layer table corresponding to the dimension data layer in the data warehouse model are optimized through the target processing task.

[0139] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the previous embodiments and will not be described in detail here.

[0140] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0141] Since the computer program stored in the computer-readable storage medium can execute the steps in any one of the data warehouse model optimization methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the data warehouse model optimization methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0142] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned data warehouse model optimization method.

[0143] The above is a detailed introduction to a data warehouse model optimization method, device, electronic device and computer storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data warehouse model optimization method, characterized in that: include: Acquire a data scheduling task based on the big data platform, and determine whether the data scheduling task is a published task; When the data scheduling task is the published task, obtaining the lineage data of the published task, parsing the lineage data based on the big data platform, and obtaining first data information; Determine whether there is a reused data table in the first data information, and when it is determined that the data table exists, trace the source of the data table to obtain the source layer of the data table in the data warehouse model; A target processing task is obtained according to the source layer, and a first data layer table corresponding to the summary data layer, a second data layer table corresponding to the detail data layer, and a third data layer table corresponding to the dimension data layer in the data warehouse model are optimized through the target processing task.

2. The data warehouse model optimization method according to claim 1, characterized in that: The acquiring target processing tasks according to the source layer and optimizing the first data layer table corresponding to the summary data layer, the second data layer table corresponding to the detail data layer, and the third data layer table corresponding to the dimension data layer in the data warehouse model through the target processing tasks include: When the source layer of the high-frequency table is a data mart layer or an original data layer, obtaining field information, and determining the field missing ranges corresponding to the first data layer table, the second data layer table, and the third data layer table respectively according to the field information; Determine the type of the target processing task according to the missing field range, and optimize the first data layer table, the second data layer table, and the third data layer table respectively according to the task items corresponding to the type; When the source layer of the data table is the summary data layer or the detail data layer, the target processing task is determined to be a retention task, and the first data layer table, the second data layer table, and the third data layer table are retained.

3. The data warehouse model optimization method according to claim 2, characterized in that: The determining the type of the target processing task according to the field missing range, and optimizing the first data layer table, the second data layer table, and the third data layer table respectively according to the task items corresponding to the type, includes: When the field missing range is the first range, determining that the type of the target processing task is a completion type, and completing the missing fields in the first data layer table, the second data layer table, and the third data layer table based on the task items corresponding to the completion type; When the field missing range is the second range, the type of the target processing task is determined to be a newly created type, and based on the task items corresponding to the newly created type, layer tables corresponding to a new detailed data layer, a new summary data layer, and a new dimensional data layer are sequentially constructed from the original data layer of the data warehouse model, wherein the first range is smaller than the second range.

4. The data warehouse model optimization method according to claim 2, characterized in that: When the source layer of the data table is the data mart layer or the original data layer, obtaining the field information includes: When the source layer of the data table is the data mart layer, tracing back the original data layer table corresponding to the data table through the first data information to obtain the field information; When the source layer of the data table is the original data layer, the target field content of the original data layer table in the original data layer is read to obtain the field information.

5. The data warehouse model optimization method according to claim 1, characterized in that: After the first data layer table corresponding to the summary data layer, the second data layer table corresponding to the detail data layer, and the third data layer table corresponding to the dimension data layer in the data warehouse model are optimized by the target processing task, the method further includes: Obtaining an optimized first data layer table, an optimized second data layer table, and an optimized third data layer table, and pushing the optimized first data layer table, the optimized second data layer table, and the optimized third data layer table to a target user terminal based on the big data platform; The tasks repeatedly submitted by the target user terminal are replaced with the optimized first data layer table, the optimized second data layer table and the optimized third data layer table.

6. The data warehouse model optimization method according to claim 1, characterized in that: After determining whether the data scheduling task is a published task, the method further comprises: When it is determined that the data scheduling task is a task being published, the task being published is analyzed in real time to obtain second data information of the task being published; According to the second data information, determining the missing ranges of the fields respectively corresponding to the first data layer table of the summary data layer table, the second data layer table of the detail data layer table, and the third data layer table of the dimension data layer table in the current data warehouse model; Corresponding processing items are determined according to the field missing range, and the first data layer table, the second data layer table, and the third data layer table are optimized based on the processing items.

7. The data warehouse model optimization method according to claim 6, characterized in that: The optimizing the first data layer table, the second data layer table and the third data layer table based on the processing items includes: If the field missing range is the minimum range, determining that the processing item is an extraction item, extracting the first data layer table, the second data layer table and the third data layer table from the current data warehouse model to the target user terminal based on the extraction item; If the field missing range is not the minimum range, the processing item is determined to be an optimization item, and the first data layer table, the second data layer table and the third data layer table are optimized.

8. A data warehouse model optimization device, characterized in that: include: A confirmation module is used to obtain a data scheduling task based on the big data platform and determine whether the data scheduling task is a published task; A parsing module, used for obtaining the lineage data of the published task when the data scheduling task is the published task, parsing the lineage data based on the big data platform, and obtaining the first data information; A source tracing module is used to determine whether there is a reused data table in the first data information, and when it is determined that the data table exists, trace the data table to obtain the source layer of the data table in the data warehouse model; The optimization module is used to obtain the target processing task according to the source layer, and optimize the first data layer table corresponding to the summary data layer, the second data layer table corresponding to the detailed data layer, and the third data layer table corresponding to the dimension data layer in the data warehouse model through the target processing task.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the data warehouse model optimization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the data warehouse model optimization method according to any one of claims 1 to 7.

Images