A method, system, computer equipment, and storage medium for processing indicator data lineage.

Abstract

This invention relates to the field of big data processing technology, and proposes a method, system, computer equipment, and storage medium for indicator data lineage processing. The method includes the following steps: pre-marking the corresponding input and output components of an indicator data processing program; when the indicator data processing program is online, parsing the program to obtain the marked source and destination information of the corresponding indicator data in its input and / or output components, and writing this information into the indicator data lineage storage medium; when the indicator data processing program is taken offline, deleting the marked source and destination information from the indicator data lineage storage medium. This invention can effectively describe the specific source and destination of indicator data, realize full lifecycle management of indicator data lineage, ensure the real-time nature and reliability of indicator data lineage, and can be used to support the traceability of abnormal data during indicator operation.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and more specifically, to a method, system, computer equipment, and storage medium for processing indicator data lineage. Background Technology

[0002] Data metrics are a core component of data-driven management, and the data lineage of metrics, also known as data pedigree or genealogy, describes the source and derivation relationships of metric data—that is, the origin and destination of the metric data. When the business side questions the anomalies of a metric or needs to confirm the metric's definition, it can locate the output table and the most original data source based on the metric data lineage. Simultaneously, when data quality monitoring detects data quality issues, it can be promptly fed back to downstream applications, allowing the applications to alert users and prevent erroneous data from negatively impacting their decision-making. Therefore, constructing a metric data lineage is a prerequisite for conducting data-driven management and metric operations.

[0003] Current methods for constructing indicator lineage diagrams involve merging multiple source data tables based on relevant business themes to generate multiple merged data tables distinguished by business themes. These merged tables record the data correspondence between the source and merged data tables, forming a data warehouse. Further, based on these data correspondences, an indicator lineage diagram is constructed. However, indicator lineage diagrams constructed solely from the data content of the source or merged data tables cannot effectively represent the specific source of the indicator data—that is, which specific data sources or other indicator data were used to calculate the indicator data; nor can they effectively represent the specific destination of the indicator data—that is, which derived or composite indicators used this indicator data to calculate their own indicator data. This presents certain limitations for data-driven management. Summary of the Invention

[0004] To overcome the shortcomings of the prior art in that the lineage of indicator data cannot effectively describe the specific source and destination of indicator data, the present invention provides a method and system for processing the lineage of indicator data.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] A method for processing the lineage of indicator data includes the following steps:

[0007] The corresponding input and output components are pre-labeled based on the indicator data processing program;

[0008] The system monitors the online or offline operation of the monitoring program; when the indicator data processing program is online, it parses the indicator data processing program to obtain the tagged source and destination information of the corresponding indicator data in its input components and / or output components, and writes it into the indicator data lineage storage medium.

[0009] When the indicator data processing program is taken offline, the source information and destination information with corresponding tags are deleted from the indicator data lineage storage medium.

[0010] As a preferred embodiment, the input component includes at least one of a data table, an API, and an indicator, and the output component includes an indicator. Specifically, the indicator includes atomic indicators, derived indicators, and composite indicators. The source information of the atomic indicator includes a specific data table and / or a specific API, and the destination information of the atomic indicator includes the atomic indicator itself. The source information of the derived indicator includes a specific atomic indicator, and the destination information of the derived indicator includes the derived indicator itself. The source information of the composite indicator includes at least two atomic indicators, derived indicators, or composite indicators, and the destination information of the composite indicator is the composite indicator itself.

[0011] As a preferred embodiment, the method further includes the following steps:

[0012] Respond to a query request, which includes a tag for the indicator data;

[0013] Matching is performed on the indicator data lineage storage medium according to the marker in the query request, to obtain the source information and destination information of the indicator data corresponding to the target marker, as well as the source information of the data source and the destination information of the data destination recursively retrieved, and then visualized.

[0014] As a preferred embodiment, the visualization process further includes: generating a directed acyclic graph of the indicator data lineage based on the source and destination information of the target marker, the source information of the data source recursively retrieved, and the destination information of the data destination recursively retrieved, using the input component and / or output component as nodes, and then visualizing the graph.

[0015] As a preferred embodiment, the method further includes the following steps:

[0016] Respond to a program configuration request, which includes one or more input components and / or an output component;

[0017] The configuration design of the indicator data processing program is carried out based on the configuration information in the program configuration request.

[0018] Furthermore, this invention also proposes an indicator data lineage processing system, applying the indicator data lineage processing method proposed in this invention. The system includes:

[0019] The preprocessing module is used to pre-label the corresponding input and output components of the indicator data processing program;

[0020] The program monitoring module is used to monitor the online or offline operation instructions of the indicator data processing program and output monitoring information.

[0021] The parsing module is used to parse the indicator data processing program to obtain the tagged source and destination information of the corresponding indicator data in its input and / or output components;

[0022] An indicator data lineage storage medium is used to store the source information and destination information corresponding to the indicator data processing program.

[0023] The processing module is used to perform lineage construction or deletion operations based on the monitoring information; wherein, when the monitoring information is that the program is online, the processing module sends a working signal to the parsing module and writes the marked source information and destination information output by the parsing module into the indicator data lineage storage medium; when the monitoring information is that the program is taken down, the processing module deletes the marked source information and destination information stored in the indicator data lineage storage medium.

[0024] As a preferred embodiment, the system further includes a human-computer interaction module for inputting a query request, the query request including a label of indicator data; the processing module is further configured to respond to the query request obtained by the human-computer interaction module, match the label in the indicator data lineage storage medium according to the label in the query request, obtain the source information and destination information of the indicator data corresponding to the target label, as well as the source information of the data source and the destination information of the data destination recursively retrieved, and generate a directed acyclic graph of indicator data lineage using the input component and / or the output component as nodes, and then return it to the human-computer interaction module for visualization display.

[0025] As a preferred embodiment, the system further includes a program configuration module, used to configure and design the indicator data processing program according to the program configuration request input by the human-computer interaction module; wherein, the program configuration request includes one or more input components and / or one output component.

[0026] Furthermore, the present invention also proposes a computer device including a memory and a processor, wherein the memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the indicator data lineage processing method as described in the present invention.

[0027] Furthermore, the present invention also proposes a storage medium storing computer-readable instructions thereon, which, when executed by a processor, implement the steps of the indicator data lineage processing method as described in the present invention.

[0028] Compared with existing technologies, the beneficial effects of the technical solution of this invention are as follows: Based on the parsing of indicator data processing programs, this invention can obtain more complete indicator data lineage information, effectively describe the specific source and destination of indicator data, and can be used to support the traceability of abnormal data during indicator operation, as well as help indicator operators quickly identify the impact of indicator changes and make more favorable decisions; This invention performs the writing or deletion of indicator data lineage information by monitoring the online or offline operation of the program, realizing the full lifecycle management of indicator data lineage, ensuring the real-time and reliability of indicator data lineage, and reducing the risk of indicator operators making wrong decisions. Attached Figure Description

[0029] Figure 1 This is a flowchart of the indicator data lineage processing method in Example 1.

[0030] Figure 2 This is a flowchart of the fuzzy search in Example 1.

[0031] Figure 3 This is a directed acyclic graph of the composite index D in Example 1.

[0032] Figure 4 This is a flowchart of the indicator data lineage self-construction process in Example 1.

[0033] Figure 5 This is an architecture diagram of the indicator data lineage processing system in Example 2. Detailed Implementation

[0034] The terms "first," "second," etc., used in the embodiments and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to those processes, methods, products, or apparatuses.

[0035] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent; it is understandable that some well-known descriptions in the drawings may be omitted in order to better illustrate this embodiment.

[0036] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] Example 1

[0038] This embodiment proposes a method for processing the lineage of indicator data, such as... Figure 1 The diagram shown is a flowchart of the indicator data lineage processing method in this embodiment.

[0039] The method for processing the lineage of indicator data proposed in this embodiment includes the following steps:

[0040] S1. Based on the indicator data processing program, pre-label the corresponding input and output components.

[0041] S2. Monitoring program's online or offline operation; When the indicator data processing program is online, parse the indicator data processing program to obtain the marked source information and destination information of the corresponding indicator data in its input components and / or output components, and write them into the indicator data lineage storage medium; When the indicator data processing program is offline, delete the marked source information and destination information from the indicator data lineage storage medium.

[0042] In this embodiment, the indicator data lineage information obtained from the indicator data processing program is more complete and can more effectively describe the specific source and destination of the indicator data. This not only supports the tracing of abnormal data during indicator operation but also helps indicator operators quickly identify the impact of indicator changes and make more informed decisions. Furthermore, this embodiment achieves full lifecycle management of indicator data lineage by executing the write or delete operations of indicator data lineage information through the online or offline operation of the monitoring program. This ensures the real-time reliability of indicator data lineage and reduces the risk of indicator operators making incorrect decisions.

[0043] In this embodiment, when the indicator data processing program is online, the indicator data processing program is parsed to obtain the metadata information of its input components and / or output components, and the source information and destination information of the corresponding indicator data with tags are further identified.

[0044] In this embodiment, the metadata information describes the data, primarily its attributes, and supports functions such as indicating storage location, historical data, resource lookup, and file records. The indicator data in this embodiment measures a specific target or entity during data-driven management and analysis, and generally consists of dimensions and metrics.

[0045] In one alternative embodiment, the input component includes at least one of a data table, an API, and a metric, and the output component includes a metric.

[0046] The indicators include atomic indicators, derived indicators, and composite indicators. The source information of atomic indicators is a specific data table and / or a specific API; the destination information of atomic indicators is the atomic indicator itself; the source information of derived indicators is a specific atomic indicator, and the destination information is the derived indicator itself; the source information of composite indicators is two or more atomic indicators, derived indicators, or composite indicators, and the destination information is the composite indicator itself.

[0047] In an optional embodiment, the method further includes the step of: responding to a query request, the query request including a tag of indicator data. Specifically, it includes the following steps:

[0048] S210. Match the indicator data lineage storage medium according to the marker in the query request;

[0049] S220. Retrieve the lineage of indicator data, obtain the source and destination information of the corresponding indicator data of the target marker, and recursively retrieve the source information of the data source and the destination information of the data destination.

[0050] S230. Visualize the search results.

[0051] like Figure 2 The diagram shown is a flowchart of the fuzzy search in this embodiment.

[0052] As an example, the markers for the indicator data included in the query request specifically include the indicator name or indicator code of the corresponding indicator data. These can be the complete indicator name or indicator code, or keywords or part of the indicator code from the indicator name.

[0053] This embodiment can optionally be used in conjunction with a human-computer interaction module. Users can input a query request for the lineage of indicator data on a graphical interface. Based on the user's input, the system retrieves the specific source and destination of the indicator data from the indicator data lineage storage medium. This also includes recursively retrieving the specific source and destination of the data. This information is used to indicate the specific data source of the indicator data, or the calculation path of the indicator data, as well as the calculation and usage history of the specific derived indicators or composite indicators involved in the indicator data. The system then forms an indicator list and returns it to the graphical interface for the user to select from. After the user completes the selection, they can perform a precise query.

[0054] As an example, the recursive retrieval in this embodiment refers to querying the recursive source of an indicator. For instance, after finding that the source information of indicator A is indicator B, the system continues to query the source information of indicator B. If the source information of indicator B is indicator C, the system continues to query the source information of indicator C, and so on, until no more source information can be found. The indicators B, C, etc., obtained during this process are the specific sources of the recursive retrieval data. Similarly, after finding that the destination information of indicator A is indicator E, the system continues to query the destination information of indicator E. If the destination information of indicator E is indicator F, the system continues to query the destination information of indicator F, and so on, until no more destination information can be found. The indicators E, F, etc., obtained during this process are the specific destinations of the recursive retrieval data.

[0055] Further, in an optional embodiment, in step S220:

[0056] S220. Retrieve the lineage of indicator data, obtain the source and destination information of the corresponding indicator data of the target marker, and recursively retrieve the source information of the data source and the destination information of the data destination.

[0057] S221. Based on the source information and destination information of the target marker, as well as the source information of the data source and the destination information of the data destination recursively retrieved, and using the input component and / or output component as nodes, generate a directed acyclic graph of the indicator data lineage as the retrieval result.

[0058] S230. Visualize the search results.

[0059] like Figure 3 The diagram shows a directed acyclic graph (DAG) of the composite index D. In this embodiment, the corresponding markers of the input components and / or output components are used as nodes, and the data flow direction is used as the direction to construct a directed acyclic graph of the index data lineage, which is used to more intuitively represent the index data lineage relationship.

[0060] In this embodiment, a directed acyclic graph will be generated based on the pedigree information of the retrieved indicator data for graphical display, allowing end users to view it more intuitively and further reducing the risk of indicator operators making incorrect decisions.

[0061] In an optional embodiment, the method further includes the following steps: responding to a program configuration request; and designing the configuration of the indicator data processing program based on the configuration information in the program configuration request.

[0062] The program configuration request includes one or more input components and / or one output component.

[0063] As an example, this embodiment may optionally be used in conjunction with a human-computer interaction module to assist users in designing and implementing indicator data processing programs through a visual interface. Optionally, the visual interface may display component options such as input components, calculation components, and output components, as well as component attribute information, which can be customized by the user.

[0064] As an example, such as Figure 4 As shown, after the program configuration is completed and the operation information of the indicator data processing program is captured, the original source and destination information of the indicator data processing program in the indicator data lineage storage medium can be optionally deleted. Then, the program's online monitoring operation is immediately executed to determine whether the indicator data processing program is online. If not, the process ends; if so, the indicator data processing program is parsed to obtain the tagged source and destination information of the corresponding indicator data in its input components and / or output components, obtain the data lineage information, and write it into the indicator data lineage storage medium to complete the indicator data lineage self-construction processing of the indicator data processing program.

[0065] Example 2

[0066] This embodiment proposes an indicator data lineage processing system, applying the indicator data lineage processing method proposed in Embodiment 1. For example... Figure 5 The diagram shown is an architecture diagram of the indicator data lineage processing system in this embodiment.

[0067] The indicator data lineage processing system proposed in this embodiment includes:

[0068] The preprocessing module is used to pre-label the corresponding input and output components of the indicator data processing program.

[0069] The program monitoring module is used to monitor the online or offline operation instructions of the indicator data processing program and output monitoring information.

[0070] The parsing module is used to parse the indicator data processing program to obtain the tagged source and destination information of the corresponding indicator data in its input components and / or output components.

[0071] The indicator data lineage storage medium is used to store the source information and destination information corresponding to the indicator data processing program.

[0072] The processing module is used to perform lineage creation or deletion operations based on the monitoring information.

[0073] Specifically, when the monitoring information indicates that a program is online, the processing module sends a working signal to the parsing module and writes the marked source information and destination information output by the parsing module into the indicator data lineage storage medium; when the monitoring information indicates that a program is taken down, the processing module deletes the marked source information and destination information stored in the indicator data lineage storage medium.

[0074] The input component includes at least one of a data table, an API, and a metric, and the output component includes a metric. The metric includes atomic metrics, derived metrics, and composite metrics. The source information of an atomic metric is a specific data table and / or a specific API; the destination information of an atomic metric is the atomic metric itself; the source information of a derived metric is a specific atomic metric, and the destination information is the derived metric itself; the source information of a composite metric is two or more atomic metrics, derived metrics, or composite metrics, and the destination information is the composite metric itself.

[0075] In an optional embodiment, the system further includes a human-computer interaction module for inputting a query request, the query request including a label of indicator data.

[0076] The processing module in this embodiment is further configured to respond to the query request obtained by the human-computer interaction module, match the marker in the index data lineage storage medium according to the marker in the query request, obtain the source information and destination information of the corresponding index data of the target marker, as well as the source information of the data source and the destination information of the data destination recursively retrieved, and generate a directed acyclic graph of index data lineage using the input component and / or output component as nodes, and then return it to the human-computer interaction module for visualization display.

[0077] As an example, the markers for the indicator data included in the query request specifically include the indicator name or indicator code of the corresponding indicator data. These can be the complete indicator name or indicator code, or keywords or part of the indicator code from the indicator name.

[0078] Further, optionally, fuzzy matching is performed on the indicator data lineage storage medium according to the marker in the query request to form an indicator list, which is then returned to the graphical interface for the user to make a second selection. After the user completes the selection, a precise query is performed.

[0079] Furthermore, in an optional embodiment, the system further includes a program configuration module, used to configure and design the indicator data processing program according to the program configuration request input by the human-computer interaction module; wherein the program configuration request includes one or more input components and / or one output component.

[0080] As an example, the human-computer interaction module displays options such as input components, calculation components, and output components, as well as options such as displaying component attribute information, for users to select and form a program configuration request.

[0081] As an example, in the specific implementation process, the program monitoring module captures the indicator data processing program operation information of the program configuration module and transmits it to the processing module. The processing module can optionally delete the original source and destination information of the indicator data processing program from the indicator data lineage storage medium. Then, it controls the program monitoring module to immediately perform a monitoring operation to determine whether the indicator data processing program is online. If not, the construction process ends; if so, the processing module initiates a program lineage parsing request to the parsing module. The parsing module then parses the indicator data processing program to obtain the tagged source and destination information of the corresponding indicator data in its input components and / or output components, obtaining the data lineage information and returning it to the processing module. Upon receiving the returned data lineage information, the processing module writes it into the indicator data lineage storage medium, completing the self-construction of the indicator data lineage for the indicator data processing program.

[0082] Example 3

[0083] This embodiment proposes a computer device, including a memory and a processor. The memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the indicator data lineage processing system as described in Embodiment 1.

[0084] Example 4

[0085] This embodiment proposes a storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the indicator data lineage processing system as described in Embodiment 1.

[0086] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. The functional modules or units can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for processing the lineage of indicator data, characterized in that, Includes the following steps: The corresponding input and output components are pre-labeled based on the indicator data processing program; The system monitors the online or offline operation of the monitoring program; when the indicator data processing program is online, it parses the indicator data processing program to obtain the tagged source and destination information of the corresponding indicator data in its input components and / or output components, and writes it into the indicator data lineage storage medium. When the indicator data processing program is taken down, the source information and destination information with corresponding tags are deleted from the indicator data lineage storage medium. The input component includes at least one of a data table, an API, and a metric, and the output component includes a metric. The indicators include atomic indicators, derived indicators, and composite indicators. The source information of the atomic indicators includes specific data tables and / or specific APIs, and the destination information of the atomic indicators includes the atomic indicators themselves. The source information of the derived indicators includes specific atomic indicators, and the destination information of the derived indicators includes the derived indicators themselves. The source information of the composite indicators includes at least two atomic indicators, derived indicators, or composite indicators, and the destination information of the composite indicators is the composite indicators themselves.

2. The method for processing the lineage of indicator data according to claim 1, characterized in that, The method further includes the following steps: Respond to a query request, which includes a tag for the indicator data; Matching is performed on the indicator data lineage storage medium according to the marker in the query request, to obtain the source information and destination information of the indicator data corresponding to the target marker, as well as the source information of the data source and the destination information of the data destination recursively retrieved, and then visualized.

3. The method for processing the lineage of indicator data according to claim 2, characterized in that, The visualization process further includes: based on the source and destination information of the target marker, as well as the source information of the data source and the destination information of the data destination recursively retrieved, and using the input component and / or output component as nodes, generating a directed acyclic graph of the indicator data lineage and visualizing it.

4. The method for processing the lineage of indicator data according to claim 1, characterized in that, The method further includes the following steps: Respond to a program configuration request, which includes one or more input components and / or an output component; The configuration design of the indicator data processing program is carried out based on the configuration information in the program configuration request.

5. A system for processing the lineage of indicator data, using the method for processing the lineage of indicator data according to any one of claims 1 to 4, characterized in that, include: The preprocessing module is used to pre-label the corresponding input and output components of the indicator data processing program; The program monitoring module is used to monitor the online or offline operation instructions of the indicator data processing program and output monitoring information. The parsing module is used to parse the indicator data processing program to obtain the tagged source and destination information of the corresponding indicator data in its input and / or output components; An indicator data lineage storage medium is used to store the source information and destination information corresponding to the indicator data processing program. A processing module is used to perform lineage construction or deletion operations based on the monitoring information; wherein, when the monitoring information indicates that a program is online, the processing module sends a working signal to the parsing module and writes the marked source information and destination information output by the parsing module into the indicator data lineage storage medium; when the monitoring information indicates that a program is taken down, the processing module deletes the marked source information and destination information stored in the indicator data lineage storage medium. The input component includes at least one of a data table, an API, and a metric, and the output component includes a metric. The indicators include atomic indicators, derived indicators, and composite indicators. The source information of the atomic indicators includes specific data tables and / or specific APIs, and the destination information of the atomic indicators includes the atomic indicators themselves. The source information of the derived indicators includes specific atomic indicators, and the destination information of the derived indicators includes the derived indicators themselves. The source information of the composite indicators includes at least two atomic indicators, derived indicators, or composite indicators, and the destination information of the composite indicators is the composite indicators themselves.

6. The indicator data lineage processing system according to claim 5, characterized in that, The system also includes a human-computer interaction module for inputting query requests, the query requests including the labeling of indicator data; The processing module is also used to respond to the query request obtained by the human-computer interaction module, match the marker in the index data lineage storage medium according to the marker in the query request, obtain the source information and destination information of the corresponding index data of the target marker, as well as the source information of the data source and the destination information of the data destination recursively retrieved, and generate a directed acyclic graph of index data lineage using the input component and / or output component as nodes, and then return it to the human-computer interaction module for visualization display.

7. The indicator data lineage processing system according to claim 6, characterized in that, The system further includes a program configuration module, used to configure and design the indicator data processing program according to the program configuration request input by the human-computer interaction module; wherein the program configuration request includes one or more input components and / or one output component.

8. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that, When the computer-readable instructions are executed by the processor, the processor performs the steps of the index data lineage processing method as described in any one of claims 1 to 4.

9. A storage medium having computer-readable instructions stored thereon, characterized in that, When the computer-readable instructions are executed by a processor, they implement the steps of the index data lineage processing method as described in any one of claims 1 to 4.

Images