Data dependency relationship determination method and device, electronic equipment and storage medium

Abstract

The invention provides a data dependency relationship determination method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of big data. According to the specific implementation scheme, the method for determining the data dependency relationship comprises the steps of extracting a target code segment from configuration information of a data synchronization task during the period of executing the data synchronization task for to-be-processed data; wherein the target code segment indicates a conversion operation, needing to be executed, of the to-be-processed data from an initial state to a target state during execution of the data synchronization task; the data synchronization task is used for converting the to-be-processed data from the source end and transmitting the converted to-be-processed data to the target end, so that the to-be-processed data in the initial state of the source end within the target time period is substantially the same as the to-be-processed data in the target state of the target end; and analyzing the target code segment to obtain a data dependency relationship between the initial state and the target state, the data dependency relationship being used for tracing a processing path of the to-be-processed data during execution of the data synchronization task.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the field of big data technology, and in particular to methods, apparatus, electronic devices, and storage media for determining data dependencies. Background Technology

[0002] Data dependencies, also known as data lineage, can clearly and completely record the relationships between the source, movement, transformation process, and destination of data throughout its entire lifecycle. It serves as a reliable basis for data governance and anomaly tracing.

[0003] Data synchronization engines can efficiently complete data synchronization tasks by performing extraction, loading, and transformation operations on massive amounts of data. However, because data synchronization engines lack built-in data dependency tracking capabilities, it is difficult to track the data flow path during the execution of data synchronization tasks. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for determining data dependencies.

[0005] According to one aspect of this disclosure, a method for determining data dependencies is provided, comprising: extracting a target code segment from configuration information of a data synchronization task during the execution of a data synchronization task for data to be processed; wherein the target code segment indicates the transformation operation required to be performed on the data to be processed from an initial state to a target state during the execution of the data synchronization task; the data synchronization task is used to transmit the data to be processed from the source end to the target end after transformation, so that the data to be processed in the initial state of the source end and the data to be processed in the target state of the target end are substantially the same during the target time period; and parsing the target code segment to obtain the data dependency relationship between the initial state and the target state, the data dependency relationship being used to trace the processing path of the data to be processed during the execution of the data synchronization task.

[0006] According to another aspect of this disclosure, a data dependency determination apparatus is provided, comprising: an extraction module and a parsing module.

[0007] An extraction module is used to extract target code segments from the configuration information of a data synchronization task during the execution of the data synchronization task for the data to be processed. The target code segment indicates the transformation operation required to be performed on the data to be processed from its initial state to its target state during the execution of the data synchronization task. The data synchronization task is used to transmit the data to be processed from the source end to the target end after transformation, so that the data to be processed in the initial state of the source end and the data to be processed in the target state of the target end are substantially the same during the target time period.

[0008] The parsing module is used to parse the target code segment to obtain the data dependency relationship between the initial state and the target state. The data dependency relationship is used to trace the processing path of the data to be processed during the execution of the data synchronization task.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the methods described above.

[0011] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 This illustration schematically depicts an exemplary system architecture for applying data dependency determination methods and apparatus according to embodiments of this disclosure;

[0015] Figure 2 A flowchart illustrating a data dependency determination method according to an embodiment of the present disclosure is shown schematically.

[0016] Figure 3 A schematic diagram illustrating a data dependency determination method according to an embodiment of the present disclosure is shown.

[0017] Figure 4 This illustration schematically shows a diagram of extracting field-level mappings from configuration information according to an embodiment of the present disclosure;

[0018] Figure 5 A schematic diagram illustrating a data dependency determination method according to another embodiment of the present disclosure is shown.

[0019] Figure 6 A schematic diagram illustrating a data dependency determination method according to yet another embodiment of the present disclosure is shown.

[0020] Figure 7 This illustration schematically shows a data structure diagram illustrating data dependencies according to embodiments of the present disclosure;

[0021] Figure 8 A data dependency diagram according to an embodiment of the present disclosure is illustrated schematically;

[0022] Figure 9 A block diagram of a data dependency determination apparatus according to embodiments of the present disclosure is illustrated schematically; and

[0023] Figure 10 A block diagram of an electronic device suitable for implementing a data dependency determination method according to embodiments of the present disclosure is shown schematically. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] Data dependency relationships, also known as data lineage, are addressed through the following technical solutions in relevant examples: 1. Utilizing data integration tools to combine with a lineage tracing platform via plugins or external systems. This approach typically relies on external logs and task scheduling systems to collect lineage information, resulting in inconsistent data collection and strong coupling. 2. Relying on lineage tracing functions provided within the metadata system. This solution depends on external tools, such as metadata format conversion tools, and suffers from gaps in the transmission and parsing of data flow information. 3. Inferring data flow through log analysis, which lacks precise field-level mapping and is prone to generating incorrect tracing paths when dealing with complex data processing chains.

[0026] In view of this, the present disclosure provides a method for determining data dependencies for a data synchronization engine. During the execution of a data synchronization task, it extracts target code segments from the configuration information of the data synchronization task, indicating the transformation operation between the initial state and the target state of the data to be processed. The method then parses these target code segments to obtain the data dependencies between the initial and target states. This achieves a closed-loop analysis of the entire data dependency chain, solving the problems of missing information and low accuracy in dependency resolution caused by reliance on external systems or information in related examples. Furthermore, it enables the resolution of data dependencies during the execution of the data synchronization task. Compared to related examples where data dependency resolution can only be performed after the task is completed, this overlaps the time periods of task execution and dependency resolution, further improving the efficiency of data dependency resolution.

[0027] Figure 1 The illustration schematically depicts an exemplary system architecture for a data dependency determination method and apparatus applicable according to embodiments of the present disclosure.

[0028] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the data dependency determination method and apparatus can be applied may include a terminal device, but the terminal device can implement the data dependency determination method and apparatus provided by the embodiments of this disclosure without interacting with the server.

[0029] like Figure 1 As shown, the system architecture 100 according to this embodiment may include: a data synchronization engine 110 and a metadata system 120.

[0030] The data synchronization engine 110 can be a SeaTunnel engine, including a data synchronization device 101 and a data dependency analysis device 102. The data synchronization device 101 executes data synchronization tasks for the data to be processed. The data dependency analysis device 102 analyzes the flow path of the data to be processed during the execution of the data synchronization tasks. The data dependency analysis device 102 is embedded into the execution flow of the data synchronization engine 110 in a plug-in manner, independent of a specific runtime engine, and possesses independent deployment capabilities and lightweight characteristics.

[0031] The data dependency analysis device 102 may include an extraction module 1021, a parsing module 1022, and a reporting module 1023. The extraction module 1021 extracts target code segments from the configuration information of the data synchronization task during its execution, including source information, transformation operation information, and sink information. The parsing module 1022 parses the target code segments extracted by the extraction module 1021 to obtain the data dependencies. The reporting module 1023, upon completion of the data synchronization task, sends the data dependencies to the metadata system 120 via standardized reporting interfaces, such as REST (Representational State Transfer), Kafka, or RPC (Remote Procedure Call).

[0032] The metadata system 120 can be used to manage data dependencies in a unified manner and to visualize them according to actual needs.

[0033] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of any type of information, such as user personal information, comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0034] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0035] Figure 2 A flowchart illustrating a data dependency determination method according to an embodiment of this disclosure is shown schematically.

[0036] like Figure 2 As shown, the method includes operations S210~S220.

[0037] During operation S210, during the execution of a data synchronization task for the data to be processed, the target code segment is extracted from the configuration information of the data synchronization task.

[0038] In operation S220, the target code segment is parsed to obtain the data dependency relationship between the initial state and the target state.

[0039] Data synchronization tasks are used to transmit data to be processed from the source end to the target end after transformation, so that the data to be processed in the initial state of the source end within the target time period is substantially the same as the data to be processed in the target state of the target end. In practical application scenarios, because the source end and the target end have different requirements for data state, the execution of data synchronization tasks involves data state transformation operations, such as renaming, updating, and splitting.

[0040] The target code segment indicates the transformation operations required to be performed on the data to be processed from its initial state to its target state during the execution of the data synchronization task.

[0041] The data synchronization task in this embodiment can be executed by invoking a data synchronization engine, such as the SeaTunnel engine. The configuration information for the data synchronization task includes at least the data source (Source) information, transformation operation (Transform) information, and sink (Sink) information for the data synchronization job. This information can exist in various configuration forms, such as configuration rules or structured query statements.

[0042] The source information is used to configure the initial state of the data to be processed, and the sink information is used to configure the target state of the data to be processed. The transformation information is used to configure the transformation operations that need to be performed to transform the data to be processed from the initial state to the target state.

[0043] The structure of the data to be processed includes, but is not limited to, table data, field data, and file data. Since the data format requirements of the source and target ends may not be the same, when performing data synchronization tasks, it is necessary to read the data to be processed from the source end, perform a transformation operation to convert the data to be processed from the initial state of the source end to the target state required by the target end, and then write the data to be processed to the target end in the target state.

[0044] In some embodiments, the source and target can be any type of database or storage system. For example, a data synchronization task may involve synchronizing data to be processed from storage system A to storage system B. The data type in storage system A can be floating-point numbers; therefore, when executing the data synchronization task, the initial state of the data to be processed read from storage system A is a floating-point number. The data type in storage system B can be integers; therefore, when executing the data synchronization task, it is necessary to convert the floating-point data to integer data to obtain data with an integer target state, and then write the integer data to storage system B to complete the data synchronization task.

[0045] The initial data to be processed can be field F1 in table L1 of storage system A. After the data synchronization task is completed, the target data to be processed can be field F1 in table L2 of storage system B. n Therefore, there is a table-level data dependency between data table L2 in storage system B and data table L1 in storage system A, and field F1 and field F... n There were previously field-level data dependencies.

[0046] Table-level and field-level data dependencies are structured lineage relationships in data lineage. This disclosure can also be applied to tracing unstructured lineage relationships such as file-level relationships. For example, a code segment representing the conversion operation between source and target information of file data can be extracted from configuration information, and then file-level lineage resolution can be performed on this code segment to obtain file-level data dependencies.

[0047] Data dependencies are used to trace the processing path of data to be processed during data synchronization tasks. This processing path can be used for anomaly localization when a data synchronization task fails.

[0048] This disclosure provides a method for determining data dependencies in a data synchronization engine. During the execution of a data synchronization task, it extracts target code segments from the task's configuration information, indicating the transformation operations between the initial and target states of the data to be processed. These target code segments are then parsed to obtain the data dependencies between the initial and target states. This achieves a closed-loop analysis of the entire data dependency chain, resolving issues such as missing information and low accuracy in dependency resolution caused by reliance on external systems or information in related examples. Furthermore, by performing data dependency resolution during the execution of the data synchronization task, compared to related examples that require waiting for the task to complete, the overlap between task execution and dependency resolution time periods further improves the efficiency of data dependency resolution.

[0049] The examples in question typically rely on external logs or scheduling systems to extract information for analyzing data dependencies, making it difficult to establish a unified information extraction scheme across different deployment environments. This approach results in high coupling between the information extraction method and the external systems it depends on, making it difficult to adapt to rapidly changing development and deployment needs. Furthermore, the data dependency resolution in these examples only involves resolving data dependencies at the table or task level, leading to low accuracy in the analysis results.

[0050] Therefore, this disclosure provides a method for resolving data dependencies based on field-level mapping, which reduces the dependence on external systems and further improves the accuracy of analysis results.

[0051] The following is for reference. Figures 3-8 In conjunction with specific embodiments, Figure 2 The method shown will be further explained.

[0052] Figure 3 A schematic diagram of a data dependency determination method according to an embodiment of the present disclosure is shown.

[0053] like Figure 3 As shown in this embodiment 300, the extraction module 1021 can extract table-level / file-level mapping code segments from the configuration information. Simultaneously, it converts field-level mapping code segments with different configuration formats from the configuration information into a predetermined format, which can be a field-level mapping structure in the form of a Mapper. Then, the table-level / file-level mapping code segments and the field-level mapping structures are combined to obtain the target code segment.

[0054] Next, the parsing module 1022 is used to parse the target code segment to obtain the data dependency relationship between the initial state and the target state of the data to be processed.

[0055] Then, the parsing module 1022 transmits the data dependency relationship to the reporting module 1023. The reporting module 1023 determines whether to report the data dependency relationship to the metadata system 103 based on the task status received from the data synchronization device 101. For example, if it is determined that the data synchronization task has been completed, the data dependency relationship is reported to the metadata system 103. If it is determined that the data synchronization task has not been completed or has failed, the data dependency relationship is discarded.

[0056] According to embodiments of this disclosure, extracting a target code segment from the configuration information of a data synchronization task during a data synchronization task performed on data to be processed may include the following operations: determining the heterogeneous field configuration type associated with the transformation operation during a data synchronization task performed on data to be processed; and extracting the target code segment from the configuration information of the data synchronization task according to the configuration type.

[0057] The heterogeneous field configuration type indicates the configuration form of the mapping relationship between the initial state field and the target state field.

[0058] In this embodiment of the disclosure, the heterogeneous field configuration type may include configuration rules for indicating the mapping relationship between the initial state field and the target state field, and a structured query statement for indicating the mapping relationship between the initial state field and the target state field.

[0059] In data synchronization tasks targeting different data sources, the data to be processed comes from different data sources, resulting in differences in data structure, data type, data semantics, and storage format.

[0060] Therefore, the configuration information of the data synchronization task will define the conversion operations between these heterogeneous fields in the form of configuration information. For example, for heterogeneous fields with inconsistent field names, the customer identification field is named ID in storage system A, USER_ID in storage system B, and Customer_code in business system C.

[0061] When defining the mapping relationship between heterogeneous fields using configuration rules, the configuration rule can be ID→USER_ID. When defining the mapping relationship between heterogeneous fields using structured query statements, this field-level mapping may be hidden in the select sub-statement of the structured query statement.

[0062] Therefore, it is necessary to extract the target code segment from the configuration information of the data synchronization task according to the configuration type.

[0063] By identifying the heterogeneous field configuration types associated with the transformation operation and extracting target code segments from the configuration information of the data synchronization task according to the configuration types, it is possible to uniformly extract field-level mappings in the configuration information, thereby further improving the adaptability of data lineage tracing to rapidly changing deployment environments.

[0064] The following is combined with Figure 4 The present disclosure provides a detailed description of the code segment for extracting field-level mappings from configuration information according to the embodiments of this disclosure.

[0065] Figure 4 A schematic diagram illustrating the extraction of field-level mappings from configuration information according to an embodiment of the present disclosure is shown.

[0066] like Figure 4 As shown, in this embodiment, the configuration types of heterogeneous fields in the configuration information include configuration rules and structured query statements.

[0067] For configuration information that defines heterogeneous fields using configuration rules, such as renaming, updating, and splitting operations, the target code segment can be extracted from the configuration information of the data synchronization task according to the configuration type. This can include the following operations: extracting a first code segment associated with the initial state field and a second code segment associated with the target state field from the configuration information; and calling the field mapping method to process the first and second code segments according to the configuration rules to obtain the target code segment.

[0068] The initial state field can refer to the input field defined in the configuration rule, i.e., the source field. The target state field can refer to the output field defined in the configuration rule after the transformation operation, i.e., the target field or alias.

[0069] The first code snippet extracts field names from the configuration rules, and then extracts the corresponding field content from the output fields of the upstream task based on those field names. The upstream task can be the task used to output the initial state field. The second code snippet can be extracted directly from the configuration rules.

[0070] The following section provides a detailed explanation using the configuration information of heterogeneous fields in a renaming operation as an example.

[0071] First, the renaming rules for fields can be read from the configuration information, for example: old field → new field. Then, the source field S1 corresponding to old field is found and used as the input field for the field-level mapping. Next, newfield is used as the output field T1 for the field-level mapping; this output field can inherit the field type or identifier of the source field. Finally, the field-level mapping structure corresponding to the renaming operation is obtained: field S1 — field T1. This transforms the renaming rules into a predetermined field-level mapping structure.

[0072] Similarly, for update and split operations, a similar method can be used to transform the update rules and split rules into a predefined field-level mapping structure. For example, the field-level mapping structure corresponding to an update operation can be field S2—field T2, where field S2 represents the source field and field T2 represents the new field after the update. The field-level mapping structure corresponding to a split operation can be field S3—field T3, where field S3 represents the source field and field T3 represents the derived field after the split.

[0073] It should be noted that the configuration information for any heterogeneous field defined by configuration rules can be directly extracted based on the field-level mapping structure obtained from the configuration rules, which will not be elaborated here.

[0074] By transforming the configuration rules in the configuration information into a field-level mapping structure, when parsing data dependencies in the target code segment, it is no longer necessary to be aware of the original task information. Only the field-level mapping structure in the target code segment needs to be parsed to achieve field-level data lineage tracing, which further improves the accuracy of data lineage tracing.

[0075] According to embodiments of this disclosure, during a data synchronization task performed on data to be processed, extracting a target code segment from the configuration information of the data synchronization task may include the following operations: during the data synchronization task performed on data to be processed, parsing a structured query statement to obtain a selection statement field in the structured query statement; extracting a first code segment associated with an initial state field and a second code segment associated with a target state field from the selection statement field; and processing the first and second code segments by calling a field mapping method according to the conversion operation type to obtain the target code segment.

[0076] The select statement field defines the type of transformation operation to be performed from the initial state field to the target state field.

[0077] Both the first and second code segments can be extracted from the expressions of the selection statement fields based on their semantics.

[0078] For example, for configuration information that defines heterogeneous fields using structured query statements, firstly, the structured query parser can be called to parse the configuration information and obtain a syntax tree. Then, by traversing the Select statement nodes in the syntax tree, the source fields and aliases in the Select statement can be extracted. The source fields in the Select statement are used as input fields for field-level mapping, and the aliases are used as mapped fields, thus obtaining the field-level mapping structure associated with the structured query statement.

[0079] Transforming the structured query statements in the configuration information into field-level mapping structures enables unified lineage resolution with heterogeneous fields defined by configuration rules. Furthermore, during field-level lineage resolution, it only needs to rely on the field-level mapping structures in the target code segment, without needing to perceive other information defined in the original structured statements, thereby further improving the efficiency of field-level resolution.

[0080] According to embodiments of this disclosure, parsing a target code segment to obtain the data dependency relationship between the initial state and the target state may include the following operations: extracting the code segment to be parsed that is associated with the structure of the data to be processed from the target code segment; and parsing the code segment to be parsed to obtain the data dependency relationship associated with the structure.

[0081] The data structure can include at least one of the following: table data, field data, and file data. Based on the data structure, the code segments associated with that structure can be parsed to obtain the data dependencies related to the structure.

[0082] Figure 5 A schematic diagram of a data dependency determination method according to another embodiment of the present disclosure is shown.

[0083] like Figure 5 As shown, the difference between this embodiment 500 and embodiment 300 is that the operations performed by the parsing module 1022 are further refined, for example, they may include table-level parsing, field-level parsing and file-level parsing.

[0084] For table data, the code segments to be parsed that are associated with the table data can be parsed to obtain the data dependencies between the table data.

[0085] In some embodiments, the data dependencies between table data can be obtained by identifying the input-output relationship between the source and sink information in the target code segment.

[0086] For example, the initial data to be processed is read from table L1. After transformation operations to obtain the target data to be processed, the target data is written to table L2. Therefore, a table dependency relationship can be generated between table L1 and table L2.

[0087] For field data, the code segments to be parsed associated with the field data can be parsed to obtain the data dependencies between the field data.

[0088] In some embodiments, field-level data dependencies can be resolved from the field-level mapping structure in the target code segment. For example, field T1 is obtained by renaming field S1.

[0089] For file data, the code segments to be parsed that are associated with the file data can be parsed to obtain the data dependencies between file data.

[0090] In some embodiments, the data dependencies between file data can be obtained from the input-output relationship between the source and sink information in the target code segment.

[0091] For example, the initial data to be processed is read from file File1. After transformation operations to obtain the target data to be processed, the target data is written to file File2. Therefore, a file dependency relationship can be generated between File1 and File2.

[0092] The data dependency determination method provided in this disclosure achieves data lineage analysis at different granularities by parsing the target code segment at the table level, field level, and file level, thereby meeting the accuracy requirements of lineage analysis in different application scenarios.

[0093] In some embodiments, parsing the target code segment to obtain the data dependency relationship between the initial state and the target state may further include the following operation: in response to determining that the data to be processed includes at least two structural forms, combining at least two data dependencies associated with at least two structural forms according to the subordinate relationship between the at least two structural forms to obtain the data dependency relationship between the initial state and the target state.

[0094] The subordinate relationship between at least two structural forms can include, but is not limited to, the subordinate relationship between field data and table data, the subordinate relationship between field data and file data, and the subordinate relationship between table data and file data.

[0095] For example, field-level parsing reveals that field T1 is obtained by renaming field S1. Table-level parsing reveals the table dependency relationship between table L1 and table L2. Field S1 is a field in table L1, and field T1 is a field in table L2. The table dependency relationship and the field dependency relationship can be combined to obtain a more complete data dependency relationship.

[0096] Therefore, for data to be processed that includes at least two structural forms, the data dependencies associated with each of the at least two structural forms can be combined based on the subordinate relationship between them, thereby further improving the accuracy and completeness of data lineage analysis.

[0097] To further improve the completeness of data lineage analysis, data lineage analysis can also be performed by combining data synchronization task information.

[0098] According to embodiments of this disclosure, the above-described data dependency determination method may further include the following operations: obtaining data synchronization task information; combining the data synchronization task information and the target code segment to obtain a combined code segment; and parsing the combined code segment to obtain data dependencies and the association between the data dependencies and the data synchronization task.

[0099] Figure 6 A schematic diagram of a data dependency determination method according to yet another embodiment of the present disclosure is shown.

[0100] like Figure 6 As shown, the difference between this embodiment 600 and embodiment 500 is that in the extraction stage, task information is extracted from configuration information, and in the parsing stage, task parsing and custom parsing are added.

[0101] Task information may include a data synchronization task identifier and its runtime environment information. The runtime environment information includes, but is not limited to, information about the upstream and downstream tasks of the data synchronization task. For example, during task parsing, factors influencing the execution path of the data synchronization task can be identified based on its runtime environment, thereby further improving the completeness of data lineage analysis.

[0102] In some embodiments, custom information can be input into the parsing module 1022 according to actual needs, and the accuracy and completeness of kinship analysis can be controlled by incorporating information such as quality dimension, time dimension, and data risk dimension.

[0103] In this embodiment 600, the reporting module 1023 may include a status interaction unit and a reporting unit.

[0104] The status interaction unit is used to interact with the data synchronization device 101 to determine whether to report data dependencies to the metadata system 103 based on the task status.

[0105] The reporting unit is used to transmit data dependencies to the metadata system 103 via the communication interface between the reporting unit and the metadata system 103 when it is determined that data dependencies should be reported to the metadata system 103. This communication interface is scalable to adapt to different metadata systems.

[0106] In some embodiments, in addition to data dependencies, the reporting unit may also transmit information such as the metadata of the data synchronization task and the topology of upstream and downstream tasks to the metadata system 103.

[0107] Adding task information parsing during the data dependency resolution process further improves the completeness of data lineage resolution.

[0108] Data dependencies can be applied to trace the processing path of data to be processed in data synchronization tasks, so that when abnormal data occurs, the abnormality can be traced along the processing path to locate the cause of the data synchronization task failure.

[0109] Therefore, embodiments of this disclosure may further include the following operations: in response to determining that the data synchronization task has been completed, obtaining a data dependency graph; in response to operations on the target edge between the initial state node and the target state node in the data dependency graph, displaying the transformation operation information and data synchronization task information required for the data to be processed to be transformed from the initial state to the target state.

[0110] Figure 7 A schematic diagram illustrating a data structure representing data dependencies according to an embodiment of this disclosure is provided.

[0111] like Figure 7 As shown, the data dependency graph 700 includes nodes and edges between them. A node can include an initial state and a target state. An edge represents the data dependency between a connected initial state node and a target state node, i.e., the transformation operation required to change the data to be processed from the initial state to the target state.

[0112] In response to operations on the target edge between the initial state node and the target state node in the data dependency graph, the system displays information on the transformation operations and data synchronization tasks required to transform the data to be processed from the initial state to the target state.

[0113] For example, transformation operation information can be represented as field-level, table-level, and / or file-level data dependencies. Data synchronization task information may include the runtime environment, task identifier, and metadata. The runtime environment can be used to indicate that the data synchronization task was created by a specific task.

[0114] By constructing a data dependency graph, the data processing path for the data to be processed in the data synchronization task is visualized, so that when abnormal data occurs, the anomaly can be traced along the processing path to locate the cause of the data synchronization task failure.

[0115] In some embodiments, the initial state node includes table data and a plurality of initial fields associated with the table data. The target state node includes table data and a plurality of target fields associated with the table data. The method may further include the following operation: in response to an operation on any of the plurality of initial fields, displaying a correlation edge between any of the plurality of target fields and any of the initial fields. Any initial field connected by the correlation edge has a data dependency relationship with any of the target fields.

[0116] Figure 8 A data dependency diagram according to an embodiment of the present disclosure is illustrated schematically;

[0117] like Figure 8 As shown, the initial state nodes in the data dependency graph 800 can include data table L1 and initial fields within data table L1, such as: initial field F1, initial field F2, ..., initial field F... m The target state node can include data table L2 and target fields within data table L2, such as: target field F1, target field F2, ..., target field F n .

[0118] As can be seen in the data dependency graph 800, there is a target edge between data table L1 and data table L2 to represent the data dependency relationship.

[0119] When an operation is triggered on the display area of ​​the initial field F2, such as clicking, swiping, or double-tapping, the data lineage analysis result indicates that the initial field F2 and the target field F n There is a data dependency between them; therefore, the initial field F2 and the target field F can be used. n The edges between them are displayed.

[0120] Since fields are usually located within table data, this embodiment of the disclosure only displays the association edges between fields that have data dependencies on each other when the fields are manipulated. This further improves the clarity of the data dependency graph in displaying data lineage and dynamically displays field-level data dependencies according to actual needs, thereby meeting the needs of anomaly tracing in complex data lineage scenarios.

[0121] Figure 9 A block diagram of a data dependency determination apparatus according to an embodiment of the present disclosure is shown schematically.

[0122] like Figure 9 As shown, the data dependency determination device 900 includes an extraction module 910 and a parsing module 920.

[0123] Extraction module 910 is used to extract target code segments from the configuration information of data synchronization task during the execution of data synchronization task for data to be processed; wherein, the target code segment indicates the transformation operation required to be performed on the data to be processed from the initial state to the target state during the execution of data synchronization task; the data synchronization task is used to transmit the data to be processed from the source end to the target end after transformation, so that the data to be processed at the source end and the data to be processed at the target end are the same during the target time period.

[0124] The parsing module 920 is used to parse the target code segment to obtain the data dependency relationship between the initial state and the target state. The data dependency relationship is used to trace the processing path of the data to be processed during the execution of the data synchronization task.

[0125] According to embodiments of this disclosure, the extraction module 910 may include a determination submodule and an extraction submodule.

[0126] The determination submodule is used to determine the heterogeneous field configuration type associated with the transformation operation during a data synchronization task performed on the data to be processed; wherein the heterogeneous field configuration type indicates the configuration form of the mapping relationship between the initial state field and the target state field.

[0127] The extraction submodule is used to extract target code segments from the configuration information of the data synchronization task according to the configuration type.

[0128] According to embodiments of this disclosure, the heterogeneous field configuration type includes configuration rules for indicating the mapping relationship between an initial state field and a target state field. The extraction submodule may include: a first extraction unit and a second extraction unit.

[0129] The first extraction unit is used to extract a first code segment associated with the initial state field and a second code segment associated with the target state field from the configuration information.

[0130] The second extraction unit is used to process the first and second code segments by calling the field mapping method according to the configuration rules, so as to obtain the target code segment.

[0131] According to embodiments of this disclosure, the configuration type includes a structured query statement indicating the mapping relationship between an initial state field and a target state field. The extraction submodule may include a parsing unit, a third extraction unit, and a fourth extraction unit.

[0132] The parsing unit is used to parse the structured query statement during the data synchronization task performed on the data to be processed, and obtain the selection statement field in the structured query statement; the selection statement field defines the type of transformation operation to be performed from the initial state field to the target state field.

[0133] The third extraction unit is used to extract the first code segment associated with the initial state field and the second code segment associated with the target state field from the selection statement field.

[0134] The fourth extraction unit is used to process the first and second code segments by calling the field mapping method according to the conversion operation type, so as to obtain the target code segment.

[0135] According to embodiments of this disclosure, the parsing module 920 may include a code extraction submodule and a code parsing submodule.

[0136] The code extraction submodule is used to extract the code segments to be parsed from the target code segment based on the structure of the data to be processed.

[0137] The code parsing submodule is used to parse the code segment to obtain the data dependencies associated with its structure.

[0138] According to embodiments of this disclosure, the structural form includes at least one of the following: table data, field data, and file data. The code parsing submodule may include: a first parsing unit, a second parsing unit, and a third parsing unit.

[0139] The first parsing unit is used to parse the code segments to be parsed that are associated with the table data, and to obtain the data dependencies between the table data.

[0140] The second parsing unit is used to parse the code segments to be parsed that are associated with the field data, and to obtain the data dependencies between the field data.

[0141] The third parsing unit is used to parse the code segments to be parsed that are associated with the file data, and to obtain the data dependencies between the file data.

[0142] According to embodiments of this disclosure, the code parsing submodule may further include: a combination unit, configured to, in response to determining that the data to be processed includes at least two structural forms, combine at least two data dependencies associated with the at least two structural forms according to the subordinate relationship between the at least two structural forms to obtain the data dependency relationship between the initial state and the target state.

[0143] According to embodiments of this disclosure, the data dependency determination device 900 may further include a first acquisition module, a combination module, and a combination parsing module.

[0144] The first acquisition module is used to acquire data synchronization task information.

[0145] The combination module is used to combine data synchronization task information and target code segments to obtain combined code segments.

[0146] The composite parsing module is used to parse composite code segments to obtain data dependencies and the relationship between data dependencies and data synchronization tasks.

[0147] According to embodiments of this disclosure, the data dependency determination device 900 may further include: a second acquisition module and a first display module.

[0148] The second acquisition module is used to acquire a data dependency graph in response to the completion of the data synchronization task; wherein, the data dependency graph includes nodes and edges between nodes, the nodes include initial state nodes and target state nodes, and the edges represent the data dependency relationships between connected nodes.

[0149] The first display module is used to respond to operations on the target edge between the initial state node and the target state node in the data dependency graph, and to display the transformation operation information and data synchronization task information required for the data to be processed to be transformed from the initial state to the target state.

[0150] According to an embodiment of this disclosure, the data dependency determination device 900 may further include: a second display module, configured to display an association edge between any target field and any initial field in a plurality of target fields in response to an operation on any initial field among a plurality of initial fields; wherein any initial field connected by the association edge has a data dependency relationship with any target field.

[0151] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0152] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described above.

[0153] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described above.

[0154] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0155] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0156] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded into random access memory (RAM) 1003 from storage unit 1008. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0157] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0158] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the data dependency determination method. For example, in some embodiments, the data dependency determination method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the data dependency determination method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform a data dependency determination method by any other suitable means (e.g., by means of firmware).

[0159] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0160] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0161] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0163] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0164] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0165] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0166] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for determining data dependencies, comprising: During the execution of a data synchronization task for the data to be processed, a target code segment is extracted from the configuration information of the data synchronization task; wherein, the target code segment indicates the transformation operation required to be performed on the data to be processed from an initial state to a target state during the execution of the data synchronization task; the data synchronization task is used to transmit the data to be processed from the source end to the target end after transformation, so that the data to be processed in the initial state of the source end during the target time period is substantially the same as the data to be processed in the target state of the target end; and The target code segment is parsed to obtain the data dependency relationship between the initial state and the target state. The data dependency relationship is used to trace the processing path of the data to be processed during the execution of the data synchronization task.

2. The method according to claim 1, wherein, The step of extracting the target code segment from the configuration information of the data synchronization task during the execution of the data synchronization task includes: During a data synchronization task performed on the data to be processed, a heterogeneous field configuration type associated with the transformation operation is determined; wherein the heterogeneous field configuration type indicates the configuration form of the mapping relationship between the initial state field and the target state field; and According to the configuration type, extract the target code segment from the configuration information of the data synchronization task.

3. The method according to claim 2, wherein, The heterogeneous field configuration type includes configuration rules for indicating the mapping relationship between the initial state field and the target state field; The step of extracting the target code segment from the configuration information of the data synchronization task according to the configuration type includes: Extract a first code segment associated with the initial state field and a second code segment associated with the target state field from the configuration information; as well as According to the configuration rules, the field mapping method is called to process the first code segment and the second code segment to obtain the target code segment.

4. The method according to claim 2, wherein, The configuration types include: structured query statements used to indicate the mapping relationship between initial state fields and target state fields; The step of extracting the target code segment from the configuration information of the data synchronization task according to the configuration type includes: During the data synchronization task performed on the data to be processed, the structured query statement is parsed to obtain the selection statement field in the structured query statement; the selection statement field defines the type of transformation operation to be performed from the initial state field to the target state field; Extract the first code segment associated with the initial state field and the second code segment associated with the target state field from the selection statement field; and According to the conversion operation type, the field mapping method is called to process the first code segment and the second code segment to obtain the target code segment.

5. The method according to any one of claims 1-4, wherein, The step of parsing the target code segment to obtain the data dependency relationship between the initial state and the target state includes: Based on the structural form of the data to be processed, extract the code segment to be parsed that is associated with the structural form from the target code segment; and The code segment to be parsed is parsed to obtain the data dependencies associated with the structure.

6. The method according to claim 5, wherein, The structural form includes at least one of the following: table data, field data, and file data; The process of parsing the code segment to be parsed to obtain the data dependencies associated with the structural form includes at least one of the following: The code segments to be parsed that are associated with the table data are parsed to obtain the data dependencies between the table data. The code segment to be parsed, which is associated with the field data, is parsed to obtain the data dependencies between the field data; The code segments to be parsed that are associated with the file data are parsed to obtain the data dependencies between the file data.

7. The method according to claim 6, wherein, The step of parsing the target code segment to obtain the data dependency relationship between the initial state and the target state also includes: In response to determining that the data to be processed includes at least two structural forms, at least two data dependencies associated with the at least two structural forms are combined according to the subordinate relationship between the at least two structural forms to obtain the data dependency relationship between the initial state and the target state.

8. The method according to any one of claims 1-7, further comprising: Obtain the data synchronization task information; The data synchronization task information and the target code segment are combined to obtain a combined code segment; as well as The combined code segment is parsed to obtain the data dependency relationship and the association between the data dependency relationship and the data synchronization task.

9. The method according to any one of claims 1-8, further comprising: In response to determining that the data synchronization task has been completed, a data dependency graph is obtained; wherein, the data dependency graph includes nodes and edges between nodes, the nodes include initial state nodes and target state nodes, and the edges represent the data dependency relationships between connected nodes; In response to an operation on the target edge between the initial state node and the target state node in the data dependency graph, information on the transformation operation to be performed and the data synchronization task information required for the data to be processed to transition from the initial state to the target state are displayed.

10. The method according to claim 9, wherein, The initial state node includes table data and multiple initial fields associated with the table data; The target state node includes the table data and multiple target fields associated with the table data; the method further includes: In response to an operation on any of the plurality of initial fields, an association edge is displayed between any of the plurality of target fields and the any of the initial fields; wherein any initial field connected by the association edge has a data dependency relationship with any of the target fields.

11. A data dependency determination apparatus, comprising: An extraction module is configured to extract a target code segment from the configuration information of a data synchronization task during the execution of the data synchronization task for the data to be processed; wherein the target code segment indicates the transformation operation required to be performed on the data to be processed from an initial state to a target state during the execution of the data synchronization task; the data synchronization task is configured to transmit the data to be processed from the source end to the target end after transformation, so that the data to be processed in the initial state of the source end during the target time period is substantially the same as the data to be processed in the target state of the target end; and The parsing module is used to parse the target code segment to obtain the data dependency relationship between the initial state and the target state. The data dependency relationship is used to trace the processing path of the data to be processed during the execution of the data synchronization task.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images