Data processing method and apparatus, storage medium, and electronic device

Abstract

Embodiments of the present application relate to the technical field of data warehouse, and more particularly, to a data processing method and device, a storage medium and an electronic device. The method comprises: performing hierarchical processing on a cluster of to-be-processed data tables to determine the hierarchical attribution of each to-be-processed data table in a data warehouse; performing blood relationship analysis on a DWD table corresponding to a data detail layer of the data warehouse, and performing filtering processing on the DWD table based on the blood relationship analysis result; and determining a substitution mapping model according to the blood relationship of each data table between a data operation layer model and a data detail layer model after filtering processing of the data warehouse. The scheme can solve the problem of unreasonable dependence on an ods table, promote the richness of the construction of the intermediate layer of the data warehouse, improve the recognizability of the data warehouse model, reduce the computing cost, and improve the data use efficiency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data warehouse technology, and more specifically, the embodiments of the present invention relate to data processing methods and apparatus, storage media and electronic devices. Background Technology

[0002] This section is intended to provide background or context for embodiments of the present invention, and the description herein is not intended to imply that it is prior art simply because it is included in this section.

[0003] A data warehouse (DW) can generally be divided into an application data service (ADS), a data service (DWS), data warehouse details (DWD), and an operation data store (ODS). When managing a data warehouse (DW), several metrics can be used to measure the governance effectiveness of the data warehouse model; the calculation of different metrics requires the use of data tables across the data warehouse layers. In some technical solutions, metrics can reflect the table entries referenced by the table; however, these metrics can only resolve relatively obvious referencing errors and non-standard practices. They cannot accurately determine whether the references to table entries are reasonable, thus leading to unreasonable table referencing. Summary of the Invention

[0004] Therefore, there is a great need for improved data processing methods and devices, storage media and electronic equipment that can provide a more reasonable solution for referencing table entries during the data warehouse model construction process.

[0005] In this context, embodiments of the present invention are intended to provide a data processing method and apparatus, a storage medium, and an electronic device.

[0006] According to one aspect of this disclosure, a data processing method is provided, the method comprising: performing hierarchical processing on a cluster of data tables to be processed to determine the hierarchical affiliation of each data table in a data warehouse; performing lineage analysis on the DWD tables corresponding to the data detail layer of the data warehouse, and filtering the DWD tables based on the lineage analysis results; and determining an alternative mapping model based on the lineage relationship between the data tables in the data operation layer model and the filtered data detail layer model of the data warehouse.

[0007] In one exemplary embodiment of this disclosure, the hierarchical processing of the cluster of data tables to be processed includes: determining the hierarchical affiliation in the data warehouse based on the table name and / or storage path corresponding to the data table to be processed; or, determining the hierarchical affiliation in the data warehouse in response to the mounting operation of the data table to be processed.

[0008] In one exemplary embodiment of this disclosure, the step of performing lineage analysis on the DWD tables corresponding to the data detail layer of the data warehouse and filtering the DWD tables based on the lineage analysis results includes: obtaining the lineage relationships of each DWD table to other levels, and filtering DWD tables whose lineage relationships belong to non-standard lineage relationships; wherein, the non-standard lineage relationships include any one of the following: a reference relationship between a DWD table and an ADS table, or a reference relationship between a DWD table and a DWS table.

[0009] In one exemplary embodiment of this disclosure, the step of calculating the alternative mapping model based on the lineage relationship between each data table in the data warehouse's data operation layer model and the filtered data detail layer model includes: determining the data operation layer model corresponding to each DWD table based on the reference relationship between each DWD table in the data detail layer model and the ODS table in the data operation layer model, so as to generate the alternative mapping model.

[0010] In one exemplary embodiment of this disclosure, the method further includes: obtaining the corresponding field lineage relationship based on the reference relationship between the DWD table and the ODS table; and determining the data operation layer model corresponding to each DWD table based on the field lineage relationship to generate the alternative mapping model.

[0011] In one exemplary embodiment of this disclosure, the method further includes: performing lineage analysis on each ADS table and each DWS table of the data application layer of the data warehouse to obtain the corresponding ODS table to be analyzed; matching the ODS table to be analyzed with the alternative mapping model, and marking the ADS table and / or DWS table corresponding to the successfully matched ODS table to be analyzed; and generating a corresponding reference governance optimization strategy for the marked ADS table and / or DWS table based on the successfully matched alternative mapping model.

[0012] In one exemplary embodiment of this disclosure, the method further includes: when the target DWD table in the successfully matched alternative mapping model is unavailable, expanding the lineage field of the target DWD table based on the lineage field corresponding to the ODS table to be analyzed.

[0013] In one exemplary embodiment of this disclosure, the method further includes: in response to a model analysis request triggered by a pending service, calculating an alternative mapping model corresponding to the pending service; when it is identified that an ODS table contained in the alternative mapping model is used, determining a replaceable DWD table corresponding to the used ODS table; and generating corresponding model optimization information based on the replaceable DWD table.

[0014] In one exemplary embodiment of this disclosure, the method further includes: in response to a model optimization request for a task of a target type, calculating an alternative mapping model corresponding to the task; performing optimization analysis on the data warehouse model corresponding to the task based on the alternative mapping model, and generating a corresponding model optimization governance strategy; wherein the model optimization governance strategy includes a substitution method for the reference relationships of data tables at each layer of the data warehouse.

[0015] According to one aspect of this disclosure, a data processing apparatus is provided, the apparatus comprising:

[0016] The pending data table hierarchy parsing module is used to perform hierarchical processing on the cluster of pending data tables in order to determine the hierarchy of each pending data table in the data warehouse.

[0017] The data table filtering module is used to perform lineage analysis on the DWD tables corresponding to the data detail layer of the data warehouse, and to filter the DWD tables based on the lineage analysis results.

[0018] The alternative mapping model calculation module is used to determine the alternative mapping model based on the lineage relationship between each data table in the data warehouse's data operation layer model and the filtered data detail layer model.

[0019] In one exemplary embodiment of this disclosure, the data table hierarchy resolution module is used to determine the hierarchy in the data warehouse based on the table name and / or storage path corresponding to the data table to be processed; or, in response to the mounting operation of the data table to be processed, to determine the hierarchy in the data warehouse.

[0020] In an exemplary embodiment of this disclosure, the data table filtering module is used to obtain the lineage relationships of each DWD table to other levels, and filter DWD tables whose lineage relationships belong to non-standard lineage relationships; wherein, the non-standard lineage relationships include any one of the following: a reference relationship between a DWD table and an ADS table, or a reference relationship between a DWD table and a DWS table.

[0021] In one exemplary embodiment of this disclosure, the alternative mapping model calculation module is used to determine the data operation layer model corresponding to each DWD table based on the reference relationship between each DWD table in the data detail layer model and the ODS table in the data operation layer model, so as to generate the alternative mapping model.

[0022] In one exemplary embodiment of this disclosure, the apparatus further includes: a field lineage analysis module, configured to obtain the corresponding field lineage relationship based on the reference relationship between the DWD table and the ODS table; and to determine the data operation layer model corresponding to each DWD table based on the field lineage relationship, so as to generate the alternative mapping model.

[0023] In one exemplary embodiment of this disclosure, the apparatus further includes: a reference governance optimization strategy generation module, configured to perform lineage analysis on each ADS table in the data application layer and each DWS table in the data service layer of the data warehouse to obtain the corresponding ODS table to be analyzed; match the ODS table to be analyzed with the alternative mapping model, and mark the ADS table and / or DWS table corresponding to the successfully matched ODS table to be analyzed; and generate a corresponding reference governance optimization strategy for the marked ADS table and / or DWS table based on the successfully matched alternative mapping model.

[0024] In one exemplary embodiment of this disclosure, the apparatus further includes: a lineage field expansion module, configured to expand the lineage field of the target DWD table based on the lineage field corresponding to the ODS table to be analyzed when the target DWD table in the successfully matched alternative mapping model is unavailable.

[0025] In one exemplary embodiment of this disclosure, the apparatus further includes: a model optimization information calculation module, configured to calculate an alternative mapping model corresponding to the pending service in response to a model analysis request triggered by the pending service; determine a replaceable DWD table corresponding to the used ODS table when it is identified that the ODS table contained in the alternative mapping model is used; and generate corresponding model optimization information based on the replaceable DWD table.

[0026] In one exemplary embodiment of this disclosure, the apparatus further includes: a request response module, configured to, in response to a model optimization request for a task to be processed of a target type, calculate an alternative mapping model corresponding to the task to be processed; perform optimization analysis on the data warehouse model corresponding to the task to be processed based on the alternative mapping model, and generate a corresponding model optimization governance strategy; wherein the model optimization governance strategy includes a substitution method for the reference relationships of data tables at each layer of the data warehouse.

[0027] According to one aspect of this disclosure, a storage medium is provided that stores a computer program thereon, wherein the computer program, when executed by a processor, performs the data processing method described above.

[0028] According to one aspect of this disclosure, an electronic device is provided, comprising:

[0029] Processor; and

[0030] Memory for storing the executable instructions of the processor;

[0031] The processor is configured to execute any of the above-described data processing methods by executing the executable instructions.

[0032] According to the data processing method of this invention, the hierarchical affiliation of each data table in the data warehouse is determined by performing hierarchical processing on the cluster of data tables to be processed. This allows for lineage analysis of the Data Detailed View (DWD) tables corresponding to the data warehouse's data detail layer, and filtering of the DWD tables based on the lineage analysis results. An alternative mapping model is calculated based on the lineage relationships between the data warehouse's data operation layer model and the filtered DWD layer model. This alternative mapping model allows for accurate calculation of the reference relationships between data tables, guiding downstream users to select more appropriate tables when referencing them, thereby resolving the problem of unreasonable table references. Attached Figure Description

[0033] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0034] Figure 1 A schematic diagram of a data processing method according to an embodiment of the present invention is shown.

[0035] Figure 2 A schematic diagram of a data warehouse layered structure according to an embodiment of the present invention is shown.

[0036] Figure 3 A schematic diagram illustrating a current kinship model according to an embodiment of the present invention is shown.

[0037] Figure 4 A schematic diagram of an optimized kinship model according to an embodiment of the present invention is shown.

[0038] Figure 5 A schematic diagram of an alternative relationship model according to an embodiment of the present invention is shown.

[0039] Figure 6 A block diagram of a data processing apparatus according to an embodiment of the present invention is shown schematically.

[0040] Figure 7 A block diagram of an electronic device according to an embodiment of the invention is schematically shown; and

[0041] Figure 8A schematic diagram of a storage medium according to an embodiment of the present invention is shown.

[0042] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0043] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0044] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0045] The data disclosed herein may be data authorized by the user or fully authorized by all parties. The collection, dissemination, and use of the data shall comply with the requirements of relevant national laws and regulations. The implementation methods / executives of this disclosure may be combined with each other.

[0046] According to embodiments of the present invention, a data processing method, a data processing apparatus, a storage medium, and an electronic device are provided.

[0047] In this document, any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0048] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments. Invention Overview

[0050] The inventors have discovered that in the prior art, reference Figure 2As shown, a data warehouse (DW) can generally be divided into an application data service (ADS), a data service layer (DWS), a data warehouse details layer (DWD), and an operation data store layer (ODS). In some music application data warehouses, metrics have been introduced to measure and guide the governance of the data warehouse in terms of model design. Specifically, the introduced metrics may include: Coverage rate, which is the ratio of the number of ODS tables directly referenced only by CDM (DWD / DWS / DIM) tables within the data warehouse management scope to the total number of ODS tables; Idle rate, which is the ratio of the number of unreferenced ODS tables to the total number of ODS tables; Reuse rate, which is the ratio of the number of CDM (DWD / DWS / DIM) tables referenced by two or more CDMs / ADSs to the total number of CDM (DWD / DWS / DIM) tables; Penetration rate, which is the ratio of the number of ADS tables that directly consume ODS to the total number of ADS tables; Asset normalization rate, which is the ratio of the number of normalized CDM (DWD / DWS / DIM) tables to the total number of CDM (DWD / DWS / DIM) tables; and Asset citation rate, which is the ratio of the number of normalized CDM tables to the total number of CDM tables among all CDM (DWD / DWS / DIM) tables referenced by ADSs. Regarding the aforementioned metrics, each metric reflects the table entries referenced by the table. In terms of addressing "reuse rate" and "penetration rate," they only resolve some obvious referencing errors and non-compliance issues. For example, if table C references table A, and table B references table A (table B inherits more content from table A), and table C can also reference table B to generate its own data, the current metrics cannot effectively determine whether referencing table B or referencing table A is more reasonable for table C. Another example is referencing... Figure 3 The data warehouse model shown currently exhibits low utilization of some DWD tables, with DWD tables D2 and D3 currently idle. Furthermore, DWS table S1 references ODS tables A5 and A6, and ADS table S2 references ODS tables A4 and A7, indicating an over-reliance on ODS tables. For deeper governance, more targeted metrics are needed to assess and measure table references, evaluating their rationality at a finer granular level.

[0051] In view of the above, the basic idea of ​​the present invention is as follows: According to the data processing method and apparatus of the present invention, the lineage relationship between data tables at each level is analyzed, and the results of the lineage relationship analysis are used to guide downstream users to select a more reasonable table when referencing the table, thereby solving the problem of unreasonable reliance on data tables.

[0052] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below.

[0053] Exemplary methods

[0054] The data processing method according to an exemplary embodiment of the present invention will now be described with reference to the accompanying drawings.

[0055] refer to Figure 1 The data processing method may include the following steps:

[0056] S1. Perform hierarchical processing on the cluster of data tables to be processed to determine the hierarchical affiliation of each data table in the data warehouse.

[0057] S2. Perform lineage analysis on the DWD tables corresponding to the data detail layer of the data warehouse, and filter the DWD tables based on the lineage analysis results;

[0058] S3. Determine the alternative mapping model based on the lineage relationship between the data tables in the data operation layer model and the filtered DWD layer model of the data warehouse.

[0059] The data processing method in this invention is applied to a data warehouse. It determines the hierarchical affiliation of each data table in the data warehouse by performing hierarchical processing on the cluster of data tables to be processed; it performs lineage analysis on the DWD tables corresponding to the DWD layer of the data warehouse, and filters the DWD tables based on the lineage analysis results; it calculates an alternative mapping model based on the lineage relationship of each data table between the ODS layer model of the data warehouse and the filtered DWD layer model, so that the reference relationship between data tables can be accurately calculated using the alternative mapping model.

[0060] In step S1, the cluster of data tables to be processed is processed in layers to determine the hierarchical affiliation of each data table in the data warehouse.

[0061] In the exemplary embodiments of this disclosure, the data processing method described above can be executed by a server or by a user-side terminal device with similar computing power. For example, the server may respond to a business processing request triggered by a user-side terminal device and read a cluster of data tables to be processed from a specified database. This cluster of data tables may include a certain number of tables, such as Hive tables; there may be certain reference relationships between the tables. After obtaining the cluster of data tables to be processed, the tables in the cluster can first be hierarchically classified according to their names, the databases where they reside, the reference relationships between them, or other attribute information. This determines the level of each table, adds corresponding hierarchical tags to each table, and creates a corresponding data warehouse model. The data warehouse model may include an ADS layer model, a DWS layer model, a DWD layer model, and an ODS layer model; wherein the ADS layer model may include ADS tables, the DWS layer model may include DWS tables, the DWD layer model may include DWD tables, and the ODS layer model may include ODS tables. If the hierarchical affiliation of a data table cannot be determined based on its attribute information, a corresponding data table hierarchy partitioning task can be generated for the remaining unprocessed data tables and pushed to designated user terminals. The hierarchical affiliation of each table can then be determined manually, and corresponding hierarchical tags can be added. (Reference) Figure 2 The data warehouse hierarchy shown can be layered into any one of the following: Data Application Layer (ADS), Data Service Layer (DWS), Data Detail Layer (DWD), and Data Operations Layer (ODS).

[0062] In step S2, lineage analysis is performed on the DWD table corresponding to the data detail layer of the data warehouse, and the DWD table is filtered based on the lineage analysis results.

[0063] In an exemplary embodiment of this disclosure, step S2 may specifically include: obtaining the lineage relationships of each DWD table to other levels, and filtering DWD tables whose lineage relationships belong to non-standard lineage relationships; wherein, the non-standard lineage relationships include any one of the following: a reference relationship between a DWD table and an ADS table, or a reference relationship between a DWD table and a DWS table.

[0064] Specifically, for the Data Detailed (DWD) layer model, a lineage analysis can be performed on each DWD table in the DWD layer model to obtain the reference relationships between each DWD table and data tables in other layers. The results of the DWD table lineage analysis can include: reference relationships between DWD tables and ADS tables, reference relationships between DWD tables and DWS tables, and reference relationships between DWD tables and ODS tables.

[0065] After obtaining the lineage analysis results for the DWD tables, non-standard DWD table lineages and their corresponding data tables can be filtered according to pre-configured rules to obtain a standardized DWD model. This standardized DWD model can include standardized DWD tables and their corresponding table entry reference relationships. The aforementioned non-standard lineage relationships include: references from DWD tables to ADS tables, and references from DWD tables to DWS tables (reverse references); additionally, it can also include references from DWD tables to ADS tables (reverse references). For the filtered DWD tables and their corresponding lineage relationships, each table entry can be individually marked for subsequent processing.

[0066] In step S3, an alternative mapping model is calculated based on the lineage relationship between each data table in the ODS layer model and the filtered DWD layer model of the data warehouse.

[0067] In an exemplary embodiment of this disclosure, step S3 may include: determining the data operation layer model corresponding to each DWD table based on the reference relationship between each DWD table in the data operation layer model and the ODS table in the data operation layer model, so as to generate the alternative mapping model.

[0068] Specifically, for each DWD table, there exists a reference relationship with an ODS table. Based on this reference relationship, the ODS table with which each DWD table has a mapping relationship can be determined; that is, one DWD table can correspond to at least one ODS table. The correspondence between DWD tables and ODS tables can be used as the alternative mapping model.

[0069] In some exemplary embodiments, the above method may further include: obtaining the corresponding field lineage relationship based on the reference relationship between the DWD table and the ODS table; and determining the ODS data operation layer model corresponding to each DWD table based on the field lineage relationship to generate the alternative mapping model.

[0070] Specifically, for each data table, at least one corresponding feature field can be configured based on the table name and the data content recorded in the table. For the DWD table, there is a reference relationship with the ODS table; the feature fields of the DWD table can be generated based on the feature fields of the ODS table. That is, the feature fields of the DWD table can be the same as, or partially the same as, the feature fields of the ODS table. Therefore, the field lineage can be obtained based on the mapping relationship between the feature fields of the DWD table and the feature fields of the ODS table; that is, a certain feature field of the DWD table actually originates from a certain feature field of the ODS table. Based on this field lineage, the mapping relationship between the DWD table and the ODS table can be determined, and based on this mapping relationship, a substitution mapping model can be obtained. The substitution mapping model can be used to describe the mapping relationship between the DWD table and the ODS table; that is, a certain DWD table can be replaced by an ODS table with which it has a mapping relationship; or, a certain ODS table can be replaced by a DWD table with which it has a mapping relationship. Based on the field lineage and table lineage, the ODS table with which one of the DWD tables has a mapping relationship can be obtained. For example, refer to... Figure 5 The substitution relationship model shown.

[0071] In some exemplary implementations, the corresponding ODS table with a mapping relationship to the DWD table can be determined based on lineage and / or field lineage. After obtaining the alternative mapping model, a corresponding alternative relationship form can be established based on the mapping relationship between the DWD table and the ODS table for subsequent use.

[0072] In exemplary embodiments of this disclosure, based on the above content, the method may further include:

[0073] Step S41: Perform lineage analysis on each ADS table in the data application layer and each DWS table in the data service layer of the data warehouse to obtain the corresponding ODS table to be analyzed.

[0074] Step S42: Match the ODS table to be analyzed with the alternative mapping model, and mark the ADS table and / or DWS table corresponding to the successfully matched ODS table to be analyzed.

[0075] Step S43: Based on the successfully matched alternative mapping model, generate corresponding reference governance optimization strategies for the labeled ADS table and / or DWS table.

[0076] Specifically, after determining the hierarchical affiliation of each data table in the cluster to be processed, lineage analysis can be performed on each ADS table in the ADS layer model and each DWS table in the DWS layer model of the data warehouse. The lineage analysis results can include DWS tables, DWD tables, and ODS tables referenced by ADS tables; and DWD tables and ODS tables referenced by DWS tables. For the ODS tables directly referenced upstream by the obtained ADS and DWS tables, these ODS tables are designated as ODS tables to be analyzed; simultaneously, these ADS and DWS tables that directly reference ODS tables can be designated as table entries to be optimized. Next, the ODS table to be analyzed can be compared with the aforementioned alternative mapping model to determine whether a matching ODS table exists in the alternative mapping model. If a matching ODS table entry exists, corresponding optimizable markers can be added to the corresponding ADS and / or DWS tables, indicating that these ADS and / or DWS tables have optimization potential. For successfully matched ODS tables, corresponding replaceable DWD tables can be obtained in the alternative mapping model. Based on the DWD tables corresponding to these ODS tables and their corresponding reference relationships, governance optimization strategies can be used to replace the producers of downstream tables.

[0077] In an exemplary embodiment of this disclosure, based on the above content, the method may further include: when the target DWD table in the successfully matched alternative mapping model is unavailable, expanding the lineage field of the target DWD table based on the lineage field corresponding to the ODS table to be analyzed.

[0078] Specifically, when replacing the producer (ODS table) of downstream tables (i.e., ADS table, DWS table), if the replacement conditions are not met during the process, the corresponding relational model can be marked. If the issue is due to insufficient field coverage in the DWD table, the corresponding DWD model can be expanded. For example, the feature fields corresponding to the ODS table to be analyzed can be added to the feature fields of the corresponding DWD table.

[0079] Alternatively, if the output timeliness is not met, check whether the DWD table can be produced ahead of schedule; if it can be produced ahead of schedule, it means that optimization can be carried out; or, if the DWD table cannot be produced ahead of schedule, add the corresponding explanatory information.

[0080] In exemplary embodiments of this disclosure, the method may further include:

[0081] Step S51: In response to the model analysis request triggered by the pending business, calculate the alternative mapping model corresponding to the pending business;

[0082] Step S52: When it is identified that the ODS table contained in the alternative mapping model is used, determine the alternative DWD table corresponding to the used ODS table.

[0083] Step S53: Generate corresponding model optimization information based on the replaceable DWD table.

[0084] Specifically, the data processing methods described above can be executed on the server side. Users can initiate model analysis requests from their terminal devices to the server. For example, the business to be processed could be a music application or a shopping application. The model analysis request could be a user-initiated request to analyze the application's data table cluster. After receiving the request, the server can use the methods described above to perform hierarchical processing on the data table cluster, create corresponding data warehouses, and perform lineage analysis to calculate the corresponding alternative mapping models. After obtaining the alternative mapping models, the data warehouse can be reorganized. Specifically, ADS tables and DWS tables can be determined based on hierarchical relationships, and the ODS tables referenced upstream by these table entries can be found through lineage relationships. The obtained ODS tables are then matched with the alternative mapping models. If a matching ODS table exists, the corresponding ADS table or DWS table is marked, indicating that there is room for optimization; and a corresponding replaceable DWD table can be provided based on the matching ODS table; and corresponding model optimization information can be generated based on this information. For example, the data warehouse model to be optimized could be as follows: Figure 3 The data warehouse model shown is currently in an idle state, where DWD tables D2 and D3 are actually available for use by DWS and ADS tables. Based on the substitution model relationship, ODS-based references can be replaced with the corresponding DWD tables. After optimizing this data model, the optimized data warehouse model can be as follows: Figure 4 As shown in the figure, the optimization corrected the reference relationships of table S2 in the ADS model, table S1 in the DWS model, and tables A4, A5, A6, and A7 in the ODS model, avoiding excessive dependence on the ODS table. The optimized DWS and ADS tables directly reference their corresponding DWD tables, ensuring that tables D2 and D3 are not idle, thus improving the reusability of the DWD model and consequently increasing the utilization rate of data warehouse assets. This makes the reference relationships in the optimized data warehouse model more reasonable.

[0085] In exemplary embodiments of this disclosure, the method may further include:

[0086] Step S61: In response to the model optimization request for the task to be processed of the target type, calculate the alternative mapping model corresponding to the task to be processed.

[0087] Step S62: Based on the alternative mapping model, perform optimization analysis on the data warehouse model corresponding to the task to be processed, and generate a corresponding model optimization governance strategy; wherein, the model optimization governance strategy includes the substitution method of the reference relationship of data tables at each layer of the data warehouse.

[0088] Specifically, the aforementioned target-type model optimization requests can be automated monitoring and optimization requests regarding the use of ODS tables during the data development phase. For example, during data development, self-service analysis tasks can be created for the data warehouse, embedding model recommendation functionality. Specifically, the ODS tables that replace mapping models can be monitored; if an ODS table is found to be in use, a corresponding prompt message can be generated, indicating that a corresponding DWD table exists and can be used to replace the ODS table. Alternatively, the aforementioned target-type model optimization requests can also occur during the data development process, where only optimization information is prompted during the code saving phase, and then the prompted code content is replaced during the task approval phase.

[0089] Alternatively, it can monitor and analyze the resources and time consumption of running tasks; if there are tasks with abnormal resource or time consumption or that exceed preset standards, it can automatically detect whether there is any optimizable model content for the task; if so, it can generate corresponding model optimization and governance strategies.

[0090] In summary, the data processing method provided in this disclosure analyzes the lineage of data table items at each layer of the data warehouse to obtain the relationship between the ODS table and the DWD table. It then filters out replaceable ODS table relationship models and records the associations between ODS and DWD tables. Based on these associations, at the governance level of the existing model, the source reference relationships of the DWS and ADS tables can be further inspected according to data lineage to identify replaceable non-standard ODS tables and recommend applicable replacement DWD tables for governance. This improves the utilization rate of the data warehouse and avoids redundant calculations, reducing the waste of computing resources. During the model development or code deployment phase, it can be checked whether the ODS tables referenced in the current code can be replaced by existing DWD tables, thereby improving the DWD model reuse rate. This solution optimizes the data warehouse model, enabling downstream models (DWS / ADS) to be generated from the data warehouse's DWD model. This avoids the cost of simultaneously modifying multiple tasks due to changes in ODS layer business logic, as changes are maintained uniformly at the DWD layer, ensuring downstream modules remain unaffected. This solution also provides a basis for addressing non-standard model referencing during data warehouse model construction; correcting model referencing reduces model misuse and increases data warehouse model utilization. This process also reduces wasted computing resources and avoids redundant computations.

[0091] Exemplary device

[0092] After introducing the data processing method according to an exemplary embodiment of the present invention, the data processing apparatus according to an exemplary embodiment of the present invention will now be described with reference to the accompanying drawings.

[0093] refer to Figure 6 As shown in the exemplary embodiment of the present invention, the data processing apparatus 60 may include: a data table hierarchical parsing module 601, a data table filtering module 602, and a substitution mapping model calculation module 603. Wherein,

[0094] The data table hierarchy parsing module 601 can be used to perform hierarchical processing on the cluster of data tables to be processed, so as to determine the hierarchy of each data table in the data warehouse.

[0095] The data table filtering module 602 can be used to perform lineage analysis on the DWD tables corresponding to the data detail layer of the data warehouse, and to filter the DWD tables based on the lineage analysis results.

[0096] The alternative mapping model calculation module 603 can be used to determine the alternative mapping model based on the lineage relationship between each data table in the data warehouse's data operation layer model and the filtered data detail layer model.

[0097] According to an exemplary embodiment of this disclosure, the data table hierarchy resolution module is used to determine the hierarchy in the data warehouse based on the table name and / or storage path corresponding to the data table to be processed; or, in response to the mounting operation of the data table to be processed, to determine the hierarchy in the data warehouse.

[0098] According to an exemplary embodiment of this disclosure, the data table filtering module is used to filter DWD tables where the blood relationship belongs to a non-standard blood relationship; wherein, the non-standard blood relationship includes any one of: a DWD table referencing an ADS table, or a DWD table being referenced by a DWS table.

[0099] According to an exemplary embodiment of this disclosure, the alternative mapping model calculation module is used to determine the data operation layer model corresponding to each DWD table based on the reference relationship between each DWD table in the data detail layer model and the ODS table in the data operation layer model, so as to generate the alternative mapping model.

[0100] According to an exemplary embodiment of this disclosure, the apparatus further includes a field kinship analysis module.

[0101] The field lineage analysis module can be used to obtain the corresponding field lineage based on the reference relationship between the DWD table and the ODS table; based on the field lineage, the data operation layer model corresponding to each DWD table is determined to generate the alternative mapping model.

[0102] According to an exemplary embodiment of this disclosure, the apparatus further includes: a reference governance optimization strategy generation module.

[0103] The reference governance optimization strategy generation module can be used to perform lineage analysis on each ADS table in the data application layer and each DWS table in the data service layer of the data warehouse to obtain the corresponding ODS table to be analyzed; match the ODS table to be analyzed with the alternative mapping model, and mark the ADS table and / or DWS table corresponding to the successfully matched ODS table to be analyzed; and generate corresponding reference governance optimization strategies for the marked ADS table and / or DWS table based on the successfully matched alternative mapping model.

[0104] According to an exemplary embodiment of this disclosure, the apparatus further includes a bloodline field expansion module.

[0105] The lineage field expansion module can be used to expand the lineage field of the target DWD table based on the lineage field corresponding to the ODS table to be analyzed when the target DWD table in the successfully matched alternative mapping model is unavailable.

[0106] According to an exemplary embodiment of this disclosure, the apparatus further includes a model optimization information calculation module.

[0107] The model optimization information calculation module can be used to respond to a model analysis request triggered by a pending business, calculate the alternative mapping model corresponding to the pending business; when it is identified that the ODS table contained in the alternative mapping model is used, determine the replaceable DWD table corresponding to the used ODS table; and generate corresponding model optimization information based on the replaceable DWD table.

[0108] According to an exemplary embodiment of this disclosure, the apparatus further includes a request response module.

[0109] The request-response module can be used to respond to a model optimization request for a task of a target type, calculate the alternative mapping model corresponding to the task, perform optimization analysis on the data warehouse model corresponding to the task based on the alternative mapping model, and generate a corresponding model optimization governance strategy; wherein, the model optimization governance strategy includes the substitution method of the reference relationship of data tables at each layer of the data warehouse.

[0110] Since the functional modules of the data processing apparatus in this embodiment are the same as those in the above-described data processing method embodiment, they will not be described again here.

[0111] Exemplary storage media

[0112] After introducing the multimedia playback control method, apparatus, and system according to exemplary embodiments of the present invention, the following references... Figure 8 The storage medium of an exemplary embodiment of the present invention will be described.

[0113] refer to Figure 8 As shown, a program product 100 for implementing the above-described method according to an embodiment of the present invention is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a device such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0114] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0115] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0116] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0117] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0118] Exemplary electronic devices

[0119] After introducing the storage medium of exemplary embodiments of the present invention, the following references are made. Figure 7 An electronic device according to an exemplary embodiment of the present invention will be described.

[0120] Figure 7 The electronic device 800 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0121] like Figure 7 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different system components (including storage unit 820 and processing unit 810), and a display unit 840.

[0122] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 810 can perform actions such as... Figure 1 The steps are shown in the figure.

[0123] Storage unit 820 may include volatile storage units, such as random access memory (RAM) 8201 and / or cache memory 8202, and may further include read-only memory (ROM) 8203.

[0124] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0125] Bus 830 may include a data bus, an address bus, and a control bus.

[0126] Electronic device 800 can also communicate with one or more external devices 900 (e.g., keyboard, pointing device, Bluetooth device, etc.) via input / output (I / O) interface 850. Electronic device 800 also includes a display unit 840 connected to input / output (I / O) interface 850 for display purposes. Furthermore, electronic device 800 can communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0127] It should be noted that although several modules or sub-modules of the audio playback device and audio sharing device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0128] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0129] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: The cluster of data tables to be processed is processed in layers to determine the hierarchical affiliation of each data table in the data warehouse. Obtain the lineage relationships of each DWD table to other levels, and filter DWD tables whose lineage relationships belong to non-normal lineage relationships; wherein, the non-normal lineage relationships include: any one of the following: the reference relationship between the DWD table and the ADS table, and the reference relationship between the DWD table and the DWS table; Based on the reference relationships between each DWD table in the data detail layer model and the ODS table in the data operation layer model, the corresponding field lineage relationships are obtained; based on the field lineage relationships, the data operation layer model corresponding to each DWD table is determined to generate an alternative mapping model. A lineage analysis is performed on each ADS table in the data application layer and each DWS table in the data service layer of the data warehouse to obtain the corresponding ODS table to be analyzed; the ODS table to be analyzed is matched with the alternative mapping model, and the ADS table and / or DWS table corresponding to the successfully matched ODS table to be analyzed is marked; based on the successfully matched alternative mapping model, a corresponding reference governance optimization strategy is generated for the marked ADS table and / or DWS table. In response to a model analysis request triggered by a pending service, calculate the alternative mapping model corresponding to the pending service; when it is identified that the ODS table contained in the alternative mapping model is used, determine the replaceable DWD table corresponding to the used ODS table; generate corresponding model optimization information based on the replaceable DWD table.

2. The data processing method according to claim 1, characterized in that, The hierarchical processing of the cluster of data tables to be processed includes: The hierarchical affiliation in the data warehouse is determined based on the table name and / or storage path corresponding to the data table to be processed; or In response to the mount operation of the data table to be processed, the hierarchical affiliation in the data warehouse is determined.

3. The data processing method according to claim 1, characterized in that, The method further includes: When the target DWD table in the successfully matched alternative mapping model is unavailable, the lineage field of the target DWD table is expanded based on the lineage field corresponding to the ODS table to be analyzed.

4. The data processing method according to claim 1, characterized in that, The method further includes: In response to a model optimization request for a task of the target type, an alternative mapping model corresponding to the task is calculated. Based on the alternative mapping model, the data warehouse model corresponding to the task to be processed is optimized and analyzed, and a corresponding model optimization and governance strategy is generated; wherein, the model optimization and governance strategy includes the alternative reference relationship of data tables at each layer of the data warehouse.

5. A data processing apparatus, characterized in that, The device includes: The pending data table hierarchy parsing module is used to perform hierarchical processing on the cluster of pending data tables in order to determine the hierarchy of each pending data table in the data warehouse. The data table filtering module is used to obtain the lineage relationship of each DWD table to other levels and filter DWD tables whose lineage relationship belongs to non-standard lineage relationship; wherein, the non-standard lineage relationship includes any one of the following: the reference relationship between DWD table and ADS table, and the reference relationship between DWD table and DWS table; The alternative mapping model calculation module is used to obtain the corresponding field lineage relationship based on the reference relationship between each DWD table in the data detail layer model and the ODS table in the data operation layer model; and to determine the data operation layer model corresponding to each DWD table based on the field lineage relationship in order to generate the alternative mapping model. The reference governance optimization strategy generation module is used to perform lineage analysis on each ADS table in the data application layer and each DWS table in the data service layer of the data warehouse to obtain the corresponding ODS table to be analyzed; match the ODS table to be analyzed with the substitution mapping model, and mark the ADS table and / or DWS table corresponding to the successfully matched ODS table to be analyzed; and generate corresponding reference governance optimization strategies for the marked ADS table and / or DWS table based on the successfully matched substitution mapping model. The model optimization information calculation module is used to respond to a model analysis request triggered by a pending business, calculate the alternative mapping model corresponding to the pending business; when it is identified that the ODS table contained in the alternative mapping model is used, determine the replaceable DWD table corresponding to the used ODS table; and generate corresponding model optimization information based on the replaceable DWD table.

6. The data processing apparatus according to claim 5, characterized in that, The data table hierarchy resolution module is used to determine the hierarchy in the data warehouse based on the table name and / or storage path of the data table to be processed; or, in response to the mounting operation of the data table to be processed, to determine the hierarchy in the data warehouse.

7. The data processing apparatus according to claim 5, characterized in that, The device further includes: The lineage field expansion module is used to expand the lineage field of the target DWD table based on the lineage field corresponding to the ODS table to be analyzed when the target DWD table in the successfully matched alternative mapping model is unavailable.

8. The data processing apparatus according to claim 5, characterized in that, The device further includes: The request-response module is used to respond to a model optimization request for a task of a target type, calculate the alternative mapping model corresponding to the task, perform optimization analysis on the data warehouse model corresponding to the task based on the alternative mapping model, and generate a corresponding model optimization governance strategy; wherein, the model optimization governance strategy includes the substitution method of the reference relationship of data tables at each layer of the data warehouse.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data processing method according to any one of claims 1 to 4.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the data processing method of any one of claims 1 to 4 by executing the executable instructions.

Images