Data relationship processing methods and apparatus, computer-readable storage media

Abstract

This disclosure relates to a data relationship processing method and apparatus, and a computer-readable storage medium. The data relationship processing method includes: establishing a hierarchical data model for data tables within a system; and establishing cascading relationships between data tables within the system and between data tables in different systems, based on the data production workflow. This disclosure can realize a comprehensive display of cross-system and cross-regional relationships in the data production process.

Description

Technical Field

[0001] This disclosure relates to the field of big data technology, and in particular to a data relationship processing method and apparatus, and a computer-readable storage medium. Background Technology

[0002] When massive amounts of heterogeneous data from telecommunications operators are aggregated and transmitted across systems, data lineage is still maintained only within the system itself. Furthermore, when the same system is built in different regions, the data models are not entirely consistent due to local construction factors. This makes it difficult to establish end-to-end data lineage throughout the data production process, hindering the comprehensive display of the entire lineage and making it difficult to promptly notify upstream and downstream systems of dynamic updates to the lineage. Summary of the Invention

[0003] The inventors discovered through research that some data lineage construction methods in related technologies lack hierarchical processing, resulting in complex and intricate lineage relationships during multi-system interactions, which are not readable or valuable.

[0004] In view of at least one of the above technical problems, this disclosure provides a data relationship processing method and apparatus, and a computer-readable storage medium, which can realize a full view of cross-system and cross-regional kinship relationships in the data production process.

[0005] According to one aspect of this disclosure, a data relationship processing method is provided, comprising:

[0006] Establish a hierarchical data model for the data tables within the system;

[0007] Based on the data production workflow, establish cascading relationships between data tables within the system and between data tables in different systems.

[0008] In some embodiments of this disclosure, establishing a hierarchical data model for data tables within the system includes:

[0009] During the data production process, the system's data model is layered according to the production sequence, wherein the system includes upstream systems and downstream systems.

[0010] In some embodiments of this disclosure, the layering of the system's data model according to production sequence includes:

[0011] The system data model is divided into an acquisition layer, an intermediate layer, and a sharing layer, where the acquisition layer is used for data input, the intermediate layer is used for data processing, and the sharing layer is used for data output.

[0012] In some embodiments of this disclosure, establishing cascading relationships between data tables within a system and between data tables across systems, based on the data production workflow, includes:

[0013] Based on the association between data tables and the data production workflow in the data model, a cascading relationship is established between data tables. The cascading relationship is a data table association group established by forward derivation starting from the target table. The data table association group includes the correspondence between the source table, the data production workflow, and the target table.

[0014] In some embodiments of this disclosure, establishing cascading relationships between data tables within a system and between data tables across systems based on the data production workflow further includes:

[0015] Field association groups are established based on the cascading relationships between data tables. These field association groups are derived forward from the fields in the target table. The data table association groups include the correspondence between the source table fields, the data production workflow procedures, the transformation rules, and the target table fields.

[0016] In some embodiments of this disclosure, the data relationship processing method further includes:

[0017] When a field in a data table within the system changes, locate the level of that data table in the data model;

[0018] In the search cascading relationship, the data table at that level is used as the source table to involve the associated group, and the affected cascading associated groups are derived backward.

[0019] The data at that level in the search cascade relationship is taken as the association group involved in the target table, and the affected cascade association groups are derived forward.

[0020] Notifications were sent to the relevant administrators of the affected cascading groups' data tables and systems.

[0021] In some embodiments of this disclosure, establishing cascading relationships between data tables across systems based on the data production workflow includes:

[0022] Use the shared layer of the upstream system as the source table in the cascading relationship;

[0023] The downstream system's acquisition layer is used as the target table in the cascading relationship.

[0024] According to another aspect of this disclosure, a data relationship processing apparatus is provided, comprising:

[0025] The data model building module is used to build hierarchical data models for data tables within the system.

[0026] The cascading relationship establishment module is used to establish cascading relationships between data tables within the system and between data tables in different systems, based on the data production workflow.

[0027] In some embodiments of this disclosure, the data relationship processing apparatus is used to perform operations that implement the data relationship processing method as described in any of the above embodiments.

[0028] According to another aspect of this disclosure, a data relationship processing apparatus is provided, comprising a memory and a processor, wherein:

[0029] Memory, used to store instructions;

[0030] A processor is configured to execute the instructions, causing the data relationship processing apparatus to perform operations implementing the data relationship processing method as described in any of the above embodiments.

[0031] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the data relationship processing method as described in any of the above embodiments.

[0032] This disclosure enables a comprehensive view of the cross-system and cross-regional kinship relationships in the data production process. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of some embodiments of the data relationship processing method disclosed herein.

[0035] Figure 2 This is a schematic diagram of a hierarchical data model within a system according to some embodiments of this disclosure.

[0036] Figure 3 This is a schematic diagram of a hierarchical data model within and between systems according to some embodiments of this disclosure.

[0037] Figure 4 These are schematic diagrams illustrating other embodiments of the data relationship processing method disclosed herein.

[0038] Figure 5 This is a schematic diagram of some embodiments of the data relationship processing apparatus disclosed herein.

[0039] Figure 6 These are schematic diagrams of other embodiments of the data relationship processing apparatus disclosed herein. Detailed Implementation

[0040] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0041] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0042] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0043] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0044] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0045] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0046] Figure 1 This is a schematic diagram of some embodiments of the data relationship processing method of this disclosure. Preferably, this embodiment can be performed by the data relationship processing apparatus of this disclosure. The method includes the following steps:

[0047] Step 11: Establish a hierarchical data model for the data tables within the system. Figure 2 This is a schematic diagram of a hierarchical data model within the system in some embodiments of this disclosure.

[0048] In some embodiments of this disclosure, step 11 may include: during the data production process, layering the data model of the system according to the production sequence, wherein the system includes an upstream system and a downstream system.

[0049] In some embodiments of this disclosure, the step of layering the system's data model according to the production sequence may include: dividing each system data model into an acquisition layer, an intermediate layer, and a sharing layer, wherein: the acquisition layer is used for data input, the intermediate layer is used for data processing, and the sharing layer is used for data output.

[0050] Figure 2 This is a schematic diagram of a hierarchical data model within the system of some embodiments of this disclosure. For example... Figure 2 As shown, each system data model is divided into a data acquisition layer, an intermediate layer, and a sharing layer. The intermediate layer can be divided into one or more layers as needed, such as an integration layer and a summary layer.

[0051] In some embodiments of this disclosure, such as Figure 2 As shown, the shared layer can be used to share with external systems based on the lower levels of this system model.

[0052] Step 12: Based on the data production workflow, establish the cascading relationships between data tables within the system and between data tables in different systems.

[0053] In some embodiments of this disclosure, the cascading relationship between the data tables can be a hierarchical cascading of the lineage between the data tables.

[0054] In some embodiments of this disclosure, step 12 may include:

[0055] Step 121: Establish a cascading relationship between data tables according to the association between data tables and the data production workflow in the data model. The cascading relationship is a data table association group established by forward derivation starting from the target table. The data table association group includes the correspondence between the source table, the data production workflow, and the target table.

[0056] In some embodiments of this disclosure, the data table association group can be: {source table 1, source table 2...}-{program k}-{target table 1}.

[0057] Step 122: Establish field association groups based on the cascading relationships between data tables. The field association groups are established by forward derivation starting from the fields in the target table. The data table association groups include the correspondence between the source table fields, the data production workflow procedures, the conversion rules, and the target table fields.

[0058] In some embodiments of this disclosure, the field association group can be: {source table 1 field i, source table 2 field j…}-{program k, conversion rule}-{target table 1 field a}.

[0059] In some embodiments of this disclosure, such as Figure 2As shown, the system exhibits hierarchical cascading, with the source table at a lower level than the target table, and the generation of lineage relationship data begins from the system's shared layer. Table association descriptions and field association descriptions for lineage relationship data can be generated manually or automatically using extraction scripts.

[0060] In some embodiments of this disclosure, step 12 may include: using the shared layer of the upstream system as the source table in the cascading relationship; and using the acquisition layer of the downstream system as the target table in the cascading relationship.

[0061] Figure 3 This is a schematic diagram illustrating the hierarchical data model within and between systems according to some embodiments of this disclosure. For example... Figure 3 As shown, the systems are cascaded in layers, with the source table being the shared layer of the upstream system and the target table being the acquisition layer of the downstream system.

[0062] In some embodiments of this disclosure, the generated blood relationship data has a one-to-one correspondence between the source table and the target table, and the fields within the tables also correspond one-to-one, without any conversion.

[0063] The above embodiments of this disclosure (e.g.) Figure 1 (Example) By layering the data models of upstream and downstream systems in the data production process, including at least a data acquisition layer that provides data input and a data sharing layer that provides data output, and then cascading the upstream and downstream systems within and between systems to build an end-to-end lineage relationship.

[0064] The above-described embodiments of the present disclosure establish a hierarchical cascading of bloodline relationships by extracting the source table, target table, and program as tuple elements of the table association description method between levels to establish table associations, and by extracting the source field, target field, and program conversion rules as tuple elements of the field association description method to establish field associations.

[0065] Based on the data relationship processing method provided in the above embodiments of this disclosure, the data models of upstream and downstream systems in the data production process can be layered, and the hierarchical relationship of the system hierarchy and system docking can be cascaded. The association description method of the relationship is established by the association of the tables and fields of the data model with the program of the workflow. The linkage logic between the data model and the data production workflow is extracted, thereby realizing the full picture display of the cross-system and cross-regional relationship of the data production process.

[0066] Figure 4 The diagram illustrates some other embodiments of the data relationship processing method of this disclosure. Preferably, this embodiment can be performed by the data relationship processing apparatus of this disclosure. Figure 4 Steps 41-42 of the embodiment are respectively with Figure 1 Steps 11-12 of the embodiment are the same or similar. Figure 4The data relationship processing method of the embodiment may include steps 41-46, wherein:

[0067] Step 41: Establish a hierarchical data model for the data tables within the system.

[0068] In some embodiments of this disclosure, step 41 may include: during the data production process, the upstream and downstream systems divide the system's data model into layers according to the production sequence from data input, data processing to data output, with the bottom layer being the acquisition layer, the top layer being the sharing layer, and the middle layer being divided into one or more layers as needed, such as the integration layer, the aggregation layer, etc.

[0069] Step 42: Based on the data production workflow, establish the cascading relationships between data tables within the system and between data tables in different systems.

[0070] In some embodiments of this disclosure, step 42 may include steps 421-423, wherein:

[0071] Step 421, the hierarchical cascading of lineage relationships: First, based on the association between data tables and the data production workflow in the data model, starting with the target table, a table association description method is established through forward deduction. The association group is: {Source Table 1, Source Table 2...} - {Program k} - {Target Table 1}. Then, starting with fields within the target table, a field association description method is established through forward deduction. The association group is: {Source Table 1 field i, Source Table 2 field j...} - {Program k, Transformation Rule} - {Target Table 1 field a}.

[0072] Step 422: Cascade the systems. The source table is a shared layer of the upstream system, and the target table is a collection layer of the downstream system. The generated lineage data corresponds one-to-one between the source and target tables, and the fields within the tables also correspond one-to-one, without any conversion.

[0073] Step 423 involves hierarchical cascading within the system. The source table's hierarchy is lower than the target table's hierarchy, and the generation of lineage relationship data begins from the system's shared layer. Table association descriptions and field association descriptions for lineage relationship data can be generated manually or automatically using extraction scripts.

[0074] Step 43: After completing the end-to-end cascading of lineage relationships, if a field of a data table in the system changes, locate the level of that data table in the data model.

[0075] Step 44: By locating the level of the data table in the data model, search for the related groups involved in the source table of the data table at that level in the cascading relationship, that is, derive the affected cascading related groups backward.

[0076] Step 45: By locating the level of the data table in the data model, search for the data at that level in the cascading relationship as the associated group involved in the target table, that is, derive the affected cascading association group forward.

[0077] Step 46: Notify the relevant administrators of the data tables and systems of the affected cascading groups.

[0078] The embodiments disclosed above can determine the level to which data tables and fields belong according to the data model, propose hierarchical cascading construction rules for blood relations within and between systems, and extract the linkage logic of tables, fields and workflow sequences of the data model to establish a blood relation association description method, making the hierarchical logic of blood relation description clear, and significantly improving readability and utilization value.

[0079] The embodiments disclosed above can realize a complete display of cross-system and cross-regional kinship relationships in the data production process, and can quickly notify the affected systems of dynamic updates to kinship relationships.

[0080] The embodiments disclosed above provide a more readable and valuable hierarchical cascading description method for data production processes. These embodiments are applicable to constructing lineage relationships between heterogeneous systems such as database systems, data warehouses, Hadoop platforms, and Storm.

[0081] Figure 5 These are schematic diagrams illustrating some embodiments of the data relationship processing apparatus of this disclosure. For example... Figure 5 As shown, the data relationship processing apparatus of this disclosure may include a data model establishment module 51 and a cascading relationship establishment module 52, wherein:

[0082] The data model building module 51 is used to build a hierarchical data model for the data tables in the system.

[0083] In some embodiments of this disclosure, the data model building module 51 can be used to layer the data model of the system according to the production sequence during the data production process, wherein the system includes an upstream system and a downstream system.

[0084] In some embodiments of this disclosure, when the data model building module 51 is layered according to the production sequence of the system's data model, it can be used to divide the system data model into an acquisition layer, an intermediate layer, and a sharing layer, wherein: the acquisition layer is used for data input, the intermediate layer is used for data processing, and the sharing layer is used for data output.

[0085] The cascading relationship establishment module 52 is used to establish cascading relationships between data tables within the system and between data tables in different systems based on the data production workflow.

[0086] In some embodiments of this disclosure, the cascading relationship establishment module 52 can be used to establish a cascading relationship between data tables according to the association relationship between data tables and data production workflow procedures in the data model. The cascading relationship is a data table association group established by forward derivation starting from the target table. The data table association group includes the correspondence between the source table, the data production workflow procedures, and the target table.

[0087] In some embodiments of this disclosure, the cascading relationship establishment module 52 can also be used to establish field association groups based on the cascading relationship between data tables. The field association groups are established by forward derivation starting from the fields in the target table. The data table association groups include the correspondence between the source table fields, the data production workflow program, the conversion rules, and the target table fields.

[0088] In some embodiments of this disclosure, the cascading relationship establishment module 52 can be used to: use a table association description method, which takes the target table as the starting point and uses the association group {source table 1, source table 2...}-{program k}-{target table 1} as the association expression to deduce forward to the source table and form a lineage relationship of table association; or use a field association description method, which takes the field in the target table as the starting point and uses the association group {source table 1 field i, source table 2 field j...}-{program k, conversion rule}-{target table 1 field a} as the association expression to deduce forward to the field in the source table and form a lineage relationship of field association.

[0089] In some embodiments of this disclosure, when the cascading relationship establishment module 52 establishes a cascading relationship between data tables in different systems based on the data production workflow, it can be used to use the shared layer of the upstream system as the source table in the cascading relationship and the acquisition layer of the downstream system as the target table in the cascading relationship.

[0090] In some embodiments of this disclosure, the system is hierarchically cascaded, with the source table being at a lower level than the target table, and the target table for constructing lineage data starting from the system shared layer.

[0091] In some embodiments of this disclosure, the systems are cascaded, with the source table being a shared layer of the upstream system and the target table being a collection layer of the downstream system. The generated lineage data corresponds one-to-one between the source table and the target table, and the fields within the tables also correspond one-to-one, without any conversion.

[0092] In some embodiments of this disclosure, the data relationship processing device can be used to layer the data models of upstream and downstream systems in the data production process, including at least a data acquisition layer that provides data input and a data sharing layer that provides data output, and then to construct an end-to-end lineage relationship by cascading the upstream and downstream systems within and between systems.

[0093] In some embodiments of this disclosure, the data relationship processing device can also be used to locate the hierarchy of a data table in a data model when a field of a data table in the system changes; search the association groups involved in the data table at that hierarchy as the source table in the cascading relationship, and backward deduce the affected cascading association groups; search the association groups involved in the data at that hierarchy as the target table in the cascading relationship, and forward deduce the affected cascading association groups; and issue notifications to the data tables of the affected cascading association groups and the corresponding administrators of the system.

[0094] In some embodiments of this disclosure, the data relationship processing apparatus can be used to perform any of the embodiments described above (e.g., Figures 1-4 The operation of the data relationship processing method described in any embodiment.

[0095] Figure 6 These are schematic diagrams illustrating other embodiments of the data relationship processing apparatus of this disclosure. For example... Figure 5 As shown, the data relationship processing apparatus of this disclosure may include a memory 61 and a processor 62, wherein:

[0096] Memory 61 is used to store instructions.

[0097] Processor 62 is configured to execute the instructions, causing the data relationship processing apparatus to perform any of the above embodiments (e.g., Figures 1-4 The operation of the data relationship processing method described in any embodiment.

[0098] Based on the data relationship processing apparatus provided in the above embodiments of this disclosure, the data models of upstream and downstream systems in the data production process can be layered, and the hierarchical relationships of the system hierarchy and system interfaces can be cascaded. The association description method of the relationship is established by associating the tables and fields of the data model with the program of the workflow. The linkage logic between the data model and the data production workflow is extracted, thereby realizing the full picture display of the cross-system and cross-regional relationship of the data production process, and can quickly notify the affected systems of the dynamic updates of the relationship.

[0099] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement any of the embodiments described above (e.g., Figures 1-4 The data relationship processing method described in any embodiment.

[0100] Based on the computer-readable storage medium provided in the above embodiments of this disclosure, the hierarchy of data tables and fields can be determined according to the data model, hierarchical cascading construction rules for blood relations within and between systems can be proposed, and the linkage logic of tables, fields and workflow sequences of the data model can be extracted to establish a blood relation association description method, making the hierarchical logic of blood relation description clear, and significantly improving readability and utilization value.

[0101] The embodiments disclosed above provide a more readable and valuable hierarchical cascading description method for data production processes. These embodiments are applicable to constructing lineage relationships between heterogeneous systems such as database systems, data warehouses, Hadoop platforms, and Storm.

[0102] This concludes the detailed description of the present disclosure. To avoid obscuring the concept of the disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0103] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0104] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A data relationship processing method, characterized by, The method comprises the following steps: establishing a hierarchical data model for data tables in a system, wherein the step of establishing a hierarchical data model for data tables in a system comprises the following steps of: in a data production process, layering the data model of the system according to a production order, wherein the step of layering the data model of the system according to a production order comprises the following steps of: dividing the data model of the system into a collection layer, an intermediate layer and a shared layer, the collection layer being used for data input, the intermediate layer being used for data processing, and the shared layer being used for data output; establishing a cascading relationship between data tables in the system and a cascading relationship between data tables in different systems according to a data production workflow, wherein the step of establishing a cascading relationship between data tables in the system and a cascading relationship between data tables in different systems according to a data production workflow comprises the following step of: establishing a cascading relationship between data tables according to an association relationship between data tables in the data model and a program of the data production workflow, the cascading relationship being a data table association group established by forward derivation with a target table as a starting point, the data table association group comprising a corresponding relationship between a source table, a program of the data production workflow and a target table; wherein the system comprises an upstream system and a downstream system, and the step of establishing a cascading relationship between data tables in different systems according to a data production workflow comprises the following steps of: taking the shared layer of the upstream system as a source table in the cascading relationship; taking the collection layer of the downstream system as a target table in the cascading relationship.

2. The data relationship processing method of claim 1, wherein, The step of establishing a cascading relationship between data tables in the system and a cascading relationship between data tables in different systems according to a data production workflow further comprises the following step of: establishing a field association group according to the cascading relationship between data tables, wherein the field association group is a field association group established by backward derivation with a field in a target table as a starting point, the data table association group comprising a corresponding relationship between a source table field, a program of the data production workflow, a conversion rule and a target table field.

3. The data relationship processing method of claim 2, wherein, The method further comprises the following steps of: in a case where a field of a data table in the system is changed, locating a level of the data table in the data model; searching for an association group in which the data table at the level is involved as a source table in the cascading relationship, and backward deriving an affected cascading association group; searching for an association group in which the data table at the level is involved as a target table in the cascading relationship, and forward deriving an affected cascading association group; sending a notification to corresponding management personnel of the data tables in the affected cascading association group and the system.

4. A data relationship processing apparatus, characterized by comprising: The method comprises the following steps: a data model establishing module is configured to establish a hierarchical data model for data tables in a system; a cascading relationship establishing module is configured to establish a cascading relationship between data tables in the system and a cascading relationship between data tables in different systems according to a data production workflow; wherein the data relationship processing device is configured to implement the data relationship processing method according to any one of claims 1-3.

5. A data relationship processing apparatus, characterized by comprising: The data relationship processing device comprises a memory and a processor, wherein: the memory is configured to store instructions; the processor is configured to execute the instructions, so that the data relationship processing device implements the data relationship processing method according to any one of claims 1-3.

6. A computer readable storage medium characterized by, The computer readable storage medium stores computer instructions, and the instructions are executed by the processor to implement the data relationship processing method according to any one of claims 1-3.

Images