Data blood relationship establishing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data consanguinity establishing method, which comprises the following steps of: reading metadata of a technical index configured for a data product, and acquiring an identifier of the technical index from the metadata of the technical index; reading metadata of a business index configured for the data product, and obtaining an identifier of the business index and an identifier of a technical index associated with the business index from the metadata of the business index; reading configuration information of an application program of the data product, and obtaining an identifier of an application index related to the application program of the data product and an identifier of a technical index associated with the application index from the configuration information; and determining an association relationship among the technical index, the business index and the application index based on the identifier of the technical index, the identifier of the business index, the identifier of the technical index associated with the business index, the identifier of the application index and the identifier of the technical index associated with the application index. The invention further provides a data blood relationship establishing device, electronic equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic device, storage medium, and program product for establishing data lineage. Background Art

[0002] To meet the demand for rapid iteration of data products, low-code architectures are often used to achieve agile iteration of data products. However, during the daily agile development and iteration of data products using low-code architectures, long data transmission links and complex data mapping relationships often lead to difficulties in analyzing the impact of changes to existing data indicators on the product, high costs for troubleshooting and locating online issues reported by users, and difficulty in automating data consistency testing. Summary of the Invention

[0003] In view of this, the embodiments of the present disclosure provide a data lineage establishment method, apparatus, electronic device, storage medium, and program product, which can solve or partially solve the above-mentioned problems to a certain extent.

[0004] In some embodiments of the present disclosure, the data lineage establishment method described in the embodiments of the present disclosure may include: reading the metadata of the technical indicators configured for the data product, and obtaining the identification of the technical indicators from the metadata of the technical indicators; reading the metadata of the business indicators configured for the data product, and obtaining the identification of the business indicators and the identification of the technical indicators associated with the business indicators from the metadata of the business indicators; reading the configuration information of the application of the data product, and obtaining the identification of the application indicators involved in the application of the data product and the identification of the technical indicators associated with the application indicators from the configuration information; determining the association relationship between the technical indicators, the business indicators, the technical indicators associated with the business indicators, the application indicators and the technical indicators associated with the application indicators based on the identification of the technical indicators, the business indicators, the business indicators, the application indicators and the technical indicators associated with the application indicators.

[0005] In some embodiments of the present disclosure, the application of a data product includes: a data product front-end application and a data product back-end application; reading the configuration information of the data product application, and obtaining the identification of the application indicators involved in the data product application and the identification of the technical indicators associated with the application indicators from the configuration information includes: reading the configuration information of the front-end page in the data product front-end application, and obtaining the identification of the application indicators involved in the front-end page from the configuration information of the front-end page; wherein, the identification of the application indicators includes: the back-end development interface identification and the engineering field index associated with the application indicators; reading the metadata of the back-end development interface involved in the data product back-end application, and determining the identification of the technical indicators associated with the application indicators from the metadata of the back-end development interface; wherein, the identification of the technical indicators includes: the identification of the data model and the data model field index.

[0006] In some embodiments of the present disclosure, the above-mentioned data lineage establishment method further includes: obtaining business indicator dimension information associated with the application indicator from the configuration information of the front-end page; wherein, the business indicator dimension information includes: one or more of: data product identifier, front-end page identifier, front-end module identifier and application indicator name.

[0007] In some embodiments of the present disclosure, the application of a data product includes: a data dashboard application; reading the configuration information of the data product application, and obtaining the identification of application indicators involved in the data product application and the identification of technical indicators associated with the application indicators from the configuration information includes: reading the metadata of the data dashboard application, and obtaining the identification of the application indicators involved in the data dashboard application from the metadata of the data dashboard application; wherein the identification of the application indicators includes: the identification of the data set and the data set field index; reading the configuration information of the data set, and obtaining the identification of technical indicators associated with the application indicators from the configuration information of the data set; wherein the identification of the technical indicators includes: the identification of the data model and the data model field index.

[0008] In some embodiments of the present disclosure, determining the association relationship between technical indicators, business indicators and application indicators based on the identification of technical indicators, the identification of business indicators, the identification of technical indicators associated with business indicators, the identification of application indicators and the identification of technical indicators associated with application indicators includes: for each of the business indicators, establishing the association relationship between the business indicator and the technical indicator according to the identification of the technical indicator associated with the business indicator; for each of the application indicators, establishing the association relationship between the application indicator and the technical indicator according to the identification of the technical indicator associated with the application indicator; and determining the association relationship between the technical indicator, the business indicator and the application indicator based on the association relationship between the business indicator and the technical indicator and the association relationship between the application indicator and the technical indicator.

[0009] In some embodiments of the present disclosure, determining the association relationship between technical indicators, business indicators and application indicators based on the identification of technical indicators, the identification of business indicators, the identification of technical indicators associated with business indicators, the identification of application indicators and the identification of technical indicators associated with application indicators includes: for each of the business indicators, establishing an association relationship between the business indicator and the technical indicator according to the identification of the technical indicator associated with the business indicator; for each of the application indicators, establishing an association relationship between the application indicator and the technical indicator and the business indicator dimension information according to the identification of the technical indicator associated with the application indicator and the business indicator dimension information associated with the application indicator; and determining the association relationship between the technical indicator, the business indicator, the business indicator dimension information and the application indicator based on the association relationship between the business indicator and the technical indicator and the association relationship between the application indicator and the business indicator and the business indicator dimension information.

[0010] In some embodiments of the present disclosure, the above-mentioned data lineage establishment method further includes: receiving the identifier of the business indicator to be queried; and based on the association relationship between the technical indicator, the business indicator and the application indicator, determining and outputting the identifier of the technical indicator and the identifier of the application indicator corresponding to the identifier of the business indicator to be queried.

[0011] In some embodiments of the present disclosure, the above-mentioned data lineage establishment method further includes: receiving the identifier of the business indicator to be queried; and based on the association relationship between the technical indicator, the business indicator, the business indicator dimension information and the application indicator, determining and outputting the business indicator dimension information, the identifier of the technical indicator and the identifier of the application indicator corresponding to the identifier of the business indicator to be queried.

[0012] In some embodiments of the present disclosure, the above-mentioned data lineage establishment method further includes: receiving dimension information of the business indicator to be queried; and based on the association relationship between the technical indicator, the business indicator, the business indicator dimension information and the application indicator, determining and outputting the identifier of the business indicator, the identifier of the technical indicator and the identifier of the application indicator corresponding to the business indicator dimension information to be queried.

[0013] Corresponding to the above-mentioned data lineage establishment method, the embodiment of the present disclosure further discloses a data lineage establishment device, including:

[0014] A technical indicator extraction module is used to read metadata of technical indicators configured for data products and obtain identifiers of the technical indicators from the metadata of the technical indicators;

[0015] A business indicator reading module, configured to read metadata of the business indicator configured for the data product, and obtain an identifier of the business indicator and an identifier of the technical indicator associated with the business indicator from the metadata of the business indicator;

[0016] an application indicator reading module, configured to read configuration information of the application of the data product, and obtain, from the configuration information, an identifier of an application indicator involved in the application of the data product and an identifier of the technical indicator associated with the application indicator; and

[0017] A data lineage establishment module is used to determine the association relationship between the technical indicator, the business indicator and the application indicator based on the identifier of the technical indicator, the identifier of the business indicator, the identifier of the technical indicator associated with the business indicator, the identifier of the application indicator and the identifier of the technical indicator associated with the application indicator.

[0018] In addition, an embodiment of the present disclosure further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data lineage establishment method when executing the computer program.

[0019] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions for causing a computer to execute the above-mentioned data lineage establishment method.

[0020] An embodiment of the present disclosure further provides a computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the above-mentioned data lineage establishment method.

[0021] It can be seen from this that through the above-mentioned data lineage establishment method, device, electronic device, storage medium and program product, for data product applications developed through the low-code framework, the relationship between the technical indicators on the data warehouse side, the business indicators on the product side and the application indicators on the engineering side can be quickly and accurately established to a certain extent, thereby quickly and accurately establishing the data lineage of the data product. In some embodiments, the above-mentioned scheme is intended to solve the problem of automatically cleaning and identifying the mapping relationship between the product-defined indicator caliber, front-end page transparent indicators, interface output fields, and data warehouse output fields for data products developed using a low-code development platform, so as to achieve the query of indicator dependency or usage information at any end of the product, engineering and data warehouse and provide it to technical personnel on the product side, engineering side or data warehouse side, thereby helping technical personnel to effectively solve the above-mentioned difficulties in sorting out and analyzing the impact of the product when modifying the existing data indicators, the high cost of online problem troubleshooting and positioning reported by users, and the difficulty in automating the consistency of test data caliber. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure.

[0024] Figure 2 The implementation process of the data lineage establishment method described in some embodiments of the present disclosure is shown.

[0025] Figure 3 The present invention shows the implementation process of reading the configuration information of the application of the data product described in some embodiments of the present disclosure, obtaining the identification of the application indicators involved in the application of the data product and the identification of the technical indicators associated with the application indicators from the configuration information.

[0026] Figure 4 The present invention shows the implementation process of reading the configuration information of the application of the data product described in other embodiments of the present invention, obtaining the identification of the application indicators involved in the application of the data product and the identification of the technical indicators associated with the application indicators from the configuration information.

[0027] Figure 5 The specific implementation process of determining the association relationship between technical indicators, business indicators and application indicators described in some embodiments of the present disclosure is shown.

[0028] Figure 6 The specific implementation process of determining the association relationship between technical indicators, business indicators and application indicators described in other embodiments of the present disclosure is shown.

[0029] Figure 7 The internal structure of the data lineage establishment device described in some embodiments of the present disclosure is shown.

[0030] Figure 8 A more specific schematic diagram of the hardware structure of an electronic device described in some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0032] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0033] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0034] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0035] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0036] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0037] For the sake of clarity in description, before describing the specific technical solutions of the embodiments of the present disclosure, the technical terms involved in the embodiments of the present disclosure are first explained.

[0038] Data products can refer to standardized service forms based on structured data, which are formed through integration and processing, and aim to release the commercial or social value of data assets through applications.

[0039] Low-code development is a development approach that allows applications to be quickly built with little or no code, typically relying on a visual interface and drag-and-drop components. Currently, low-code development is a common approach to data product development.

[0040] Data lineage refers to the network of relationships formed from the generation to the disappearance of data, including the source of the data, processing logic, flow path and final destination.

[0041] Field-level application lineage is a specific manifestation of data lineage. It refers to the specific relationships between data throughout its lifecycle, from generation, processing, manipulation, integration, circulation, to eventual extinction. Field-level application lineage focuses on the source and destination of data at the specific field level, specifically, which fields or data sources generate the data for a particular field, and how that data is used and transferred during processing.

[0042] The scope of an indicator refers to the unified standards and rules followed during data collection, compilation, analysis, and reporting. The scope of an indicator determines which data is included in the statistical scope and how it is processed and categorized. The clarity of the indicator's scope directly impacts the accuracy and consistency of the data, which in turn influences analytical results and decision-making.

[0043] Data warehouse (DW or DWH for short), also known as data warehouse in Chinese, can be a subject-oriented, integrated, relatively stable data set that reflects historical changes. In the data warehouse system, fields and indicators are different forms in the data processing link. The two have both basic dependencies and different business roles. Among them, fields can be the most basic atomic units in the data warehouse, usually referring to the original data fields in the source data layer. Metrics are quantitative expressions of business goals, which need to be generated through field processing, aggregation or combination, and are often used to measure business results. It can be understood that fields only describe the data itself, while indicators need to be integrated into business logic. On the data warehouse side, an operating platform for managing the data warehouse can usually be set up to process, aggregate or combine the fields stored in the data warehouse to generate indicators on the data warehouse side. In the embodiment of the present disclosure, in order to distinguish them from other indicators, the indicators on the data warehouse side generated by the data warehouse management platform are called technical indicators. Typically, a technical indicator can be mapped to one or more fields in one or more data models (or data tables) within the data warehouse, defining the data processing logic for one or more fields in one or more data models (or data tables). Based on this, technical indicators can be identified using data model identifiers and field indexes within the data models.

[0044] A Product Manager (PM) can be a position in an enterprise that is specifically responsible for product management. The main responsibilities include product planning, R&D, manufacturing, marketing and channels, serving end users, and being responsible for the entire life cycle of the product from conceptualization to market release. The product manager usually represents the product side, and can usually break down one or more data requirements based on the business demand plan of the data product, and then determine the indicators and the caliber of the indicators required for data processing of the data product based on the broken down data requirements. In the embodiment of the present disclosure, in order to distinguish it from other indicators, the indicators determined by the product side are called business indicators. Business indicators usually reflect the product side's demand for data. Normally, through negotiation between the product side and the data warehouse side, the business indicators on the product side and the technical indicators on the data warehouse side are usually one-to-one corresponding. When product managers break down the business requirements of data products, in addition to data requirements, they can also be responsible for defining the secondary business modules involved in the data product (usually one data product corresponds to a primary business module) (for example, a secondary business module can correspond to a front-end page in the data product front-end application) and the tertiary business modules involved in each secondary business module (for example, a tertiary business module can correspond to a front-end module on a front-end page).

[0045] The front-end application of a data product can refer to the portion of the data product development process that involves direct user interaction, primarily including the design of the user interface (UI) and user experience (UX). The front-end application is responsible for creating and designing the components that users directly see and interact with within a webpage or application, including web or mobile application interfaces. For ease of description, webpages and mobile application interfaces are collectively referred to as front-end pages. Generally speaking, as mentioned above, a data product (which can correspond to a first-level business module defined on the product side) can typically involve multiple front-end pages (each of which can correspond to a second-level business module defined on the product side), and a front-end page can also include multiple front-end modules (each of which can correspond to a third-level business module defined on the product side). Front-end modules generally refer to the individual components that make up a front-end page, each responsible for specific functionality or displaying specific content. Examples of front-end modules include UI components (such as buttons and input boxes) or functional modules (such as data display and user authentication). Front-end modules are designed to improve code reusability, reduce coupling, and facilitate maintenance and expansion. It is understandable that some front-end modules of the front-end page may involve some indicators. These indicators can usually be defined as data indicators rendered by the front-end in the corresponding business module based on the caliber of technical indicators of the data product. In the embodiment of the present disclosure, in order to distinguish them from other indicators, the indicators rendered on the front-end page are called application indicators. It can be seen that application indicators should be able to be mapped to business indicators defined on the product side. The same business indicator can be specifically mapped to different application indicators on different pages of a data product or different modules of the same page.

[0046] The back-end application of a data product generally refers to an application located on the server side of the data product software system, which processes requests from the front-end, executes business logic, interacts with the database, and responds data to the front-end application, etc.

[0047] The back-end development interface may refer to a programming interface provided by the back-end application to the front-end application or other system calls, which is mainly used for data interaction and business logic processing. The back-end development interface usually exists in the form of Hypertext Transfer Protocol (HTTP) Application Programming Interface (API) or Remote Procedure Call (RPC). It is understandable that a front-end module may involve a back-end development interface. The back-end development interface identification (ID) may refer to a string or digital code used to uniquely identify a back-end development interface in the front-end application and the back-end application, which is mainly used for service routing, permission verification and call link tracking. Typically, the back-end development interface will involve some information about parameters for data transmission, which are referred to as engineering fields in the embodiments of the present disclosure. Engineering fields can be identified using engineering field indexes. In the embodiments of the present disclosure, it can be considered that application indicators essentially correspond one-to-one with engineering fields. Therefore, each application indicator can be identified using the back-end development interface identification and the engineering field index.

[0048] A data dashboard can be a visualization tool that displays key indicators and important data through a graphical interface, designed to help users quickly understand complex information and support decision-making. In the embodiments of the present disclosure, the indicators that can be displayed by the data dashboard are also referred to as application indicators. In the embodiments of the present disclosure, in the data dashboard application, it can be considered that the application indicators essentially correspond one-to-one with the dataset fields in the dataset involved in the data dashboard application. Therefore, in the data dashboard application, each application indicator can be identified using the dataset identifier involved in the data dashboard application and the index of the dataset field.

[0049] Metadata can be structured information used to describe data attributes. Essentially, metadata is "data about data" and its core function is to provide contextual interpretation and management support for data.

[0050] JavaScript Object Notation (JSON) is a lightweight data exchange format widely used for data transmission between web applications and application programming interfaces (APIs). JSON typically represents data in a concise, easy-to-read and easy-to-write text format.

[0051] As previously mentioned, during the daily agile development and iteration of data products using a low-code architecture, long data transmission links and complex data mapping relationships can make it difficult to analyze the impact of modifications to existing data indicators on products. The high cost of troubleshooting and locating online issues reported by users also makes it difficult to automate the consistency of test data. Therefore, the embodiments of the present disclosure provide a data lineage establishment method, apparatus, electronic device, storage medium, and program product that can partially or partially resolve the aforementioned issues.

[0052] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure.

[0053] like Figure 1 As shown, the system 100 may include a data product business platform 110, a data product business terminal 112, a data warehouse management platform 120, a data warehouse management terminal 122, a data processing platform 130, a data processing terminal 132, and a data product user terminal 134. A medium (e.g., a network) providing a communication link may be included between the data product business platform 110, the data product business terminal 112, the data warehouse management platform 120, the data warehouse management terminal 122, the data processing platform 130, the data processing terminal 132, and the data product user terminal 134. The network may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0054] Exemplarily, the data product business terminal 112 may be installed with a data product management application (APP) or software, which may be used to configure business indicators related to data products, etc. The data warehouse management terminal 122 may be installed with a data warehouse management application or software, which may be used to configure technical indicators related to the data warehouse, etc. Among them, business indicators usually correspond to one or more technical indicators. The data processing terminal 132 may be installed with an application or software for engineering development of the data processing platform 130, which may be used to carry out front-end application development, back-end application development, and data dashboard application development associated with data processing, etc., and to configure application indicators involved in front-end applications, back-end applications, and data dashboard applications. Among them, business indicators also usually correspond to one or more application indicators. The data product user terminal 134 may be installed with a front-end application or a data dashboard application for a data product, so that it can be used to provide a data query and data display interface for data product users.

[0055] The data product business terminal 112, the data warehouse management terminal 122, the data processing terminal 132 and the data product user terminal 134 here can be hardware or software. When the data product business terminal 112, the data warehouse management terminal 122, the data processing terminal 132 and the data product user terminal 134 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, MP3 players, laptop computers (Laptops) and desktop computers (PCs), etc. When the data product business terminal 112, the data warehouse management terminal 122, the data processing terminal 132 and the data product user terminal 134 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0056] The data product business platform 110, data warehouse management platform 120, and data processing platform 130 can be servers that provide various services, such as backend servers that support applications or software displayed on their corresponding terminal devices. Furthermore, the data product business platform 110, data warehouse management platform 120, and data processing platform 130 can also include databases for storing relevant data or metadata.

[0057] The data product business platform 110, data warehouse management platform 120, and data processing platform 130 herein can also be hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software or software modules (for example, for providing distributed services), or as a single software or software module. No specific limitation is made here.

[0058] It should be understood that Figure 1 The number of data product service terminals 112, data warehouse management terminals 122, data processing terminals 132, data product user terminals 134, as well as the data product service platform 110, data warehouse management platform 120, and data processing platform 130, is merely illustrative. Depending on implementation requirements, any number of data product service terminals 112, data warehouse management terminals 122, data processing terminals 132, data product user terminals 134, as well as the data product service platform 110, data warehouse management platform 120, and data processing platform 130 may be provided. Accordingly, any number of users operating the data product service terminals 112, data warehouse management terminals 122, data processing terminals 132, and data product user terminals 134 may also be provided.

[0059] As an exemplary scenario, a product manager 114 of a data product can typically decompose the data requirements from the business requirements plan of the data product and determine the secondary business modules and tertiary business modules involved in the data product. Furthermore, based on the data requirements, the product manager 114 can determine the business indicators and their scope of the data product. The manager then enters the metadata of the business indicators into the data product business platform 110 through the data product business terminal 112, thereby configuring the business indicators. Furthermore, a data warehouse manager 124 can typically enter the metadata of technical indicators into the data warehouse management platform 120 through the data warehouse management terminal 122, thereby completing the configuration of the technical indicators. Furthermore, a data product developer 136 can submit the front-end application and back-end application or data dashboard application of the data product to the data processing platform 130 through the data processing terminal 132. The user 138 of the data product can query one or more application indicators from the data processing platform 130 through the data product user terminal 134.

[0060] Figure 2 The implementation process of the data lineage establishment method described in the embodiment of the present disclosure is shown. Figure 2 As shown, the data lineage establishment method described in the embodiment of the present disclosure may specifically include the following steps.

[0061] In step 210 , metadata of the technical indicators configured for the data product is read, and an identifier of the technical indicator is obtained from the metadata of the technical indicator.

[0062] In step 220 , metadata of the business indicator configured for the data product is read, and an identifier of the business indicator and an identifier of a technical indicator associated with the business indicator are obtained from the metadata of the business indicator.

[0063] In step 230 , configuration information of the application of the data product is read, and identifiers of application indicators involved in the application of the data product and identifiers of technical indicators associated with the application indicators are obtained from the configuration information.

[0064] In step 240, based on the identification of the technical indicator, the identification of the business indicator, the identification of the technical indicator associated with the business indicator, the identification of the application indicator, and the identification of the technical indicator associated with the application indicator, the association relationship between the technical indicator, the business indicator, and the application indicator is determined.

[0065] It can be seen that the above-mentioned data lineage establishment method can quickly and accurately establish the relationship between technical indicators on the data warehouse side, business indicators on the product side, and application indicators on the engineering side for data product applications developed through the low-code framework, thereby quickly and accurately establishing the data lineage of data products, thereby effectively solving the difficulties in sorting out and analyzing the impact of product modifications when existing data indicators are modified, the high cost of online problem troubleshooting and locating user feedback, and the difficulty in automating the consistency of test data caliber.

[0066] The specific implementation of each step of the above-mentioned data lineage establishment method will be described in detail below with reference to the accompanying drawings and specific examples.

[0067] With respect to the above-mentioned step 210, as previously mentioned, the data warehouse manager 124 can enter the metadata of the technical indicators into the data warehouse management platform 120 through the data warehouse management terminal 122, thereby completing the configuration of the technical indicators involved in the data product on the data warehouse side. In some embodiments of the present disclosure, the metadata of a technical indicator may include: the identifier of the technical indicator, the name of the technical indicator, and the caliber of the technical indicator, etc. As previously mentioned, under normal circumstances, a technical indicator can be mapped to one or more data model fields in the data model on the data warehouse side. Based on this, in some embodiments of the present disclosure, the technical indicator can be identified using the identifier of the data model (or the name of the data model can also be used as the identifier of the data model) and the data model field index.

[0068] Specifically, in some embodiments of the present disclosure, the above step 210 may specifically include: first, obtaining metadata of each technical indicator from the data warehouse management platform 120; and then, reading the identifier of each technical indicator from the metadata of each technical indicator.

[0069] Regarding step 220, as previously described, the product manager of the data product can enter the metadata of the business indicator into the data product business platform through the data product business terminal, thereby completing the configuration of the business indicator involved in the data product on the product side. In some embodiments of the present disclosure, the metadata of a business indicator may include: the identification of the business indicator, the name of the business indicator, the scope of the business indicator, the business links involved, and the identification of technical indicators associated with the business indicator, etc.

[0070] Specifically, in some embodiments of the present disclosure, the above step 220 may include: first, obtaining metadata of each business indicator from the data product business platform 110; then, reading the identifier of each business indicator and the identifier of the technical indicator associated with each business indicator from the metadata of each business indicator.

[0071] Regarding step 230 above, in some embodiments of the present disclosure, the data product application may be an application including a data product front-end application and a data product back-end application. In other embodiments of the present disclosure, the data product application may also be a data dashboard application.

[0072] In the application of the data product including the data product front-end application and the data product back-end application, the above step 230 can be specifically implemented as follows: Figure 3 The method shown is implemented.

[0073] In step 310, configuration information of each front-end page in the data product front-end application is read, and identifiers of each application indicator involved in each front-end page are obtained from the configuration information of each front-end page.

[0074] As mentioned above, in the embodiments of the present disclosure, the application indicator essentially corresponds one-to-one to an engineering field in a back-end development interface. Therefore, the identifier of an application indicator can generally specifically include: a back-end development interface identifier associated with the application indicator and an engineering field index.

[0075] In some embodiments of the present disclosure, the data product front-end application may be a front-end application in the form of online code. The online code form may generally refer to code deployed on a server and accessible via the Internet. Online code is typically deployed in a production environment after rigorous testing and optimization, and users can access the code through a browser. Online code needs to ensure high availability, high performance, and security to provide a good user experience. Alternatively, in other embodiments of the present disclosure, the data product front-end application may also be a front-end application in the form of offline code. The offline code form may generally refer to code written in a local development environment, primarily used in the development and testing phases. Developers write, debug, and test code on local computers to ensure that the code can run normally in the development environment. The configuration information of each front-end page in the front-end application developed by the developer may be stored in a storage device used to store the front-end application code. It is understandable that offline code is typically not directly user-facing, but is continuously modified and improved during the development process.

[0076] Based on the above, for front-end applications in the form of online code, in the above step 310, the configuration information of each front-end page in the front-end application can be first read from the code repository; then, the identifiers of each application indicator involved in each front-end page can be obtained from the read configuration information of each front-end page in the front-end application. For front-end applications in the form of offline code, in the above step 310, the configuration information of each front-end page in the front-end application can be first read from the storage device used to store the front-end application code; then, the identifiers of each application indicator involved in each front-end page can be obtained from the metadata of each front-end page in the read front-end application.

[0077] In some embodiments of the present disclosure, the configuration information of each front-end page developed through a low-code framework is usually edited and stored in JSON format, and since JSON is formatted data, the identifiers of each application indicator involved in each front-end page can be extracted from it. In an embodiment of the present disclosure, the configuration information of a front-end page may include: a front-end page identifier, a front-end page name, identifiers of each front-end module involved in the front-end page, and names of each front-end module involved in the front-end page, names of application indicators involved in the front-end module, back-end interface identifiers corresponding to application indicators, project field indexes, and project field names, etc. In some other embodiments of the present disclosure, the configuration information of each front-end page may further include: a data product identifier, which is used to identify the data product applied by the current front-end page. As mentioned above, the data product may correspond to a first-level business module defined by the product side; the front-end page may correspond to a second-level business module defined by the product side, and the front-end module may correspond to a third-level business module defined by the product side.

[0078] As mentioned above, generally speaking, a data product can correspond to a front-end application, and a front-end application generally includes one or more front-end pages. A data product can be uniquely identified using a data product identifier, while each front-end page can be uniquely identified using a front-end page identifier. Furthermore, each front-end page can also include one or more front-end modules. Each front-end module can be uniquely identified using a front-end module identifier. Thus, for the same data product, the application indicators involved in a front-end module can generally correspond one-to-one to a business indicator within the constraints of the front-end page identifier and the front-end module identifier. In other words, a business indicator can correspond to multiple application indicators. However, after adding the constraints of the front-end page identifier and the front-end module identifier to the business indicator, the business indicator can generally correspond one-to-one to a specific application indicator. Furthermore, assuming that multiple data products can reuse business indicators and application indicators, the application indicators involved in a front-end module can generally correspond one-to-one to a business indicator within the constraints of the data product identifier, the front-end page identifier, and the front-end module identifier. In other words, a business indicator can correspond to multiple application indicators. However, after adding restrictions in three dimensions: data product identification, front-end page identification, and front-end module identification, business indicators can usually correspond one-to-one to a specific application indicator.

[0079] Based on the above situation, in the embodiments of the present disclosure, one or a combination of the data product identifier, the front-end page identifier, the front-end module identifier and the application indicator name can be used as business indicator dimension information, so as to be used as a screening condition for business indicators when performing information query. In other words, in some embodiments of the present disclosure, the configuration of business indicators, the configuration of application indicators and the configuration of technical indicators can only be effective in one data product. In this case, the data product identifiers can be considered to be the same, for example, all are default values. Therefore, the business indicator dimension information can only include: one or more of the front-end page identifier, the front-end module identifier and the application indicator name. In other embodiments of the present disclosure, the configuration of business indicators, the configuration of application indicators and the configuration of technical indicators can be reused in multiple data products, and the business indicator dimension information can include: one or more of the data product identifier, the front-end page identifier, the front-end module identifier and the application indicator name. Among them, the name of the application indicator is usually also the name of the indicator displayed on the front-end page. That is, the name of the indicator that the user sees on the front-end page usually corresponds to a technical indicator and also corresponds to a business indicator.

[0080] As mentioned above, in the above steps, the configuration information of the front-end page obtained may include: the front-end page identifier, the front-end page name, the identifiers of each front-end module involved in the front-end page, the names of each front-end module involved in the front-end page, the back-end interface identifier corresponding to the application indicators involved in the front-end module, and the engineering field index. Alternatively, the configuration information of each front-end page may further include: a data product identifier. Thus, in the above step 310, in addition to obtaining the identifiers of each application indicator involved in each front-end page, the business indicator dimension information associated with each application indicator may be further obtained from the configuration information of each front-end page. In some embodiments, the business indicator dimension information may include: one or more of the data product identifier, the front-end page identifier, the front-end module identifier, and the application indicator name.

[0081] In step 320 , metadata of each backend development interface in the data product backend application is read, and identifiers of technical indicators associated with each application indicator are determined from the metadata of each backend development interface.

[0082] In some embodiments of the present disclosure, step 320 may include: first, reading metadata for each backend development interface in the backend application from the code repository; then, obtaining the identifiers of each application indicator from the metadata of each backend development interface read. As previously described, the identifier of an application indicator may include: a backend development interface identifier and a project field index. In embodiments of the present disclosure, the metadata of a backend development interface may generally include: a backend development interface identifier, a backend development interface name, a project field index, and an identifier of a data model associated with the backend development interface identifier and the project field index, as well as a data model field index. As previously described, the identifier of the data model and the data model field index can essentially be considered as an identifier of a technical indicator. Therefore, the identifiers of each application indicator and the identifiers of the technical indicators associated with each application indicator can be determined from the metadata of each backend development interface in the backend application of the data product read. In other embodiments of the present disclosure, the metadata of a backend development interface may further include: an identifier of a business indicator associated with each application indicator.

[0083] From this, it can be seen that by reading the configuration information of each front-end page involved in the front-end application and the metadata of each back-end development interface involved in the back-end application, the identification of each application indicator involved in the data product and the identification of the technical indicators associated with each application indicator can be quickly and accurately obtained. That is, through the above-mentioned data lineage establishment method, the correspondence between the back-end development interface identification + engineering field index on the engineering side and the number model identification + data model field index on the data warehouse side can be quickly and accurately obtained.

[0084] When the application of the data product is a data dashboard application, the above step 230 can be specifically implemented as follows: Figure 4 The method shown is implemented.

[0085] In step 410 , metadata of the data dashboard application is read, and identifiers of various application indicators involved in the data dashboard are obtained from the metadata of the data dashboard application.

[0086] As previously mentioned, in the embodiments of the present disclosure, application metrics essentially correspond one-to-one with dataset fields in the dataset involved in the data dashboard application. Therefore, the identifier of the application metric may include: the identifier of the dataset involved in the data dashboard application and the dataset field index. Step 410 may include: first, reading metadata for the data dashboard application from the code repository; then, obtaining the identifier of the dataset involved in the data dashboard application and the index of each dataset field from the metadata of the data dashboard application.

[0087] In step 420 , the configuration information of the data set is read, and the identifiers of the technical indicators associated with each application indicator are obtained from the configuration information of the data set.

[0088] The above-mentioned step 420 may include: first reading the configuration information of the data set from the code repository; then, obtaining the identifier of the data model and the data model field index associated with each data set field index in each data set from the configuration information of the read data set. As mentioned above, the identifier of the data model and the data model field index can essentially be regarded as the identifier of a technical indicator. Therefore, the identifier of the technical indicator associated with each data set field index can be determined from the configuration information of the read data set. In other embodiments of the present disclosure, the configuration information of a data set may further include: the identifier of the business indicator associated with each application indicator.

[0089] From this, we can see that by reading the metadata and dataset configuration information of the data dashboard application, we can quickly and accurately obtain the identifiers of each application indicator involved in the data dashboard application, as well as the identifiers of the technical indicators associated with each application indicator. In other words, the above-mentioned data lineage establishment method can quickly and accurately obtain the correspondence between the dataset identifier + dataset field index on the engineering side and the numerical model identifier + data model field index on the data warehouse side.

[0090] Regarding the above step 240, Figure 5 The specific implementation process of determining the association relationship between technical indicators, business indicators and application indicators in some embodiments of the present disclosure is shown. Figure 5 As shown, the correlation between technical indicators, business indicators and application indicators can be determined through the following steps.

[0091] In step 510, for each business indicator, an association relationship between the business indicator and the technical indicator is established according to the identifier of the technical indicator associated with the business indicator.

[0092] In step 520 , for each application indicator, an association relationship between the application indicator and the technical indicator is established according to the identifier of the technical indicator associated with the application indicator.

[0093] In step 530 , the association relationship between the technical indicators, the business indicators, and the application indicators is determined based on the association relationship between the business indicators and the technical indicators and the association relationship between the application indicators and the technical indicators.

[0094] In some embodiments of the present disclosure, the associations among the aforementioned business indicators, technical indicators, and application indicators may be referred to as a data lineage table. Table 1 below shows an example of a tabular form (an example of a data lineage table) of the associations among the aforementioned business indicators, technical indicators, and application indicators established using the aforementioned data lineage establishment method. In Table 1, the identifiers for the application indicators include: a backend development interface identifier and an engineering field index.

[0095]

[0096]

[0097] Table 1

[0098] Table 2 below shows another example of a tabular form of the relationship between business indicators, technical indicators, and application indicators established using the above data lineage establishment method (another example of a data lineage table). In Table 2, the identifier of the application indicator includes: a dataset identifier and a dataset field index.

[0099]

[0100] Table 2

[0101] Figure 6 The specific implementation process of determining the association relationship between technical indicators, business indicators and application indicators described in other embodiments of the present disclosure is shown. Figure 6 The method shown is applicable to the case where in addition to obtaining the identifiers of the application indicators involved in each front-end page in the above step 310, the business indicator dimension information associated with each application indicator is also obtained. Figure 6 As shown, the correlation between technical indicators, business indicators and application indicators can be determined through the following steps.

[0102] In step 610, for each business indicator, an association relationship between the business indicator and the technical indicator is established according to the identifier of the technical indicator associated with the business indicator.

[0103] In step 620 , for each application indicator, an association relationship between the application indicator and the technical indicator and the business indicator dimension information is established based on the identifier of the business indicator associated with the application indicator and the business indicator dimension information associated with the application indicator.

[0104] In step 630 , the association relationship among the technical indicators, business indicators, business indicator dimensional information and application indicators is determined based on the established association relationship between the business indicators and the technical indicators and the association relationship between the application indicators and the technical indicators and the business indicator dimensional information.

[0105] In some embodiments of the present disclosure, the associations among the aforementioned technical indicators, business indicators, business indicator dimension information, and application indicators may also be referred to as a data lineage table. Table 3 below shows an example of a table format (another example of a data lineage table) for determining the associations among technical indicators, business indicators, business indicator dimension information, and application indicators, established using the aforementioned data lineage establishment method.

[0106]

[0107] Table 3

[0108] From the examples shown in Table 1 and Table 3 above, it can be seen that a business indicator can usually be associated with a technical indicator and multiple application indicators. Furthermore, under the limitation of business indicator dimension information, a business indicator can usually be associated with a technical indicator and an application indicator.

[0109] From this, it can be seen that through the above-mentioned data lineage establishment method, for data products including front-end applications and back-end applications developed through the low-code framework, the correlation relationship between technical indicators on the data warehouse side, business indicators on the product side, and application indicators on the engineering side can be established quickly and accurately.

[0110] In addition, the above-mentioned data lineage establishment method can realize the mapping of business indicators and technical indicators to engineering fields or data-level fields. Therefore, it can realize the establishment of field-level application lineage on the product side, data warehouse side and engineering side, so that the data lineage of data products can be established quickly and accurately, effectively solving the difficulties in sorting out and analyzing the impact of modifications to existing data indicators on products, the high cost of online problem troubleshooting and locating user feedback, and the difficulty in automating the consistency of test data caliber.

[0111] The specific application method of data lineage established by the above-mentioned data lineage establishment method will be further described in detail below with reference to the accompanying drawings and specific examples.

[0112] In some embodiments of the present disclosure, it is assumed that during the development of a data product or after the deployment of the data product is completed, there is still a need on the product side to modify the scope of a certain business indicator. In this case, the data warehouse side usually needs to determine which technical indicators are related to it and whether they need to be updated accordingly; the engineering side needs to determine which application indicators are related to it and need to be updated. For example, if the data product is implemented through front-end applications and back-end applications, the engineering side needs to determine which front-end pages and front-end modules in the front-end application need to be updated, and also needs to determine which back-end development interfaces and engineering fields in the back-end application need to be updated. If the data product is implemented through a data dashboard application, the engineering side needs to determine which data sets and which data set fields in the data dashboard application need to be updated. In response to the above needs, in some embodiments of the present disclosure, the data lineage establishment method may further include: receiving the identifier of the business indicator to be queried; based on the determined association relationship between the technical indicator, business indicator and application indicator, determining and outputting the identifier of the technical indicator and the identifier of the application indicator corresponding to the identifier of the business indicator to be queried. Alternatively, in other embodiments of the present disclosure, the above-mentioned data lineage establishment method may further include: receiving the identifier of the business indicator to be queried; based on the determined association relationship between the technical indicators, business indicators, business indicator dimension information and application indicators, determining and outputting the business indicator dimension information, the identification of the technical indicator and the identification of the application indicator corresponding to the identifier of the business indicator to be queried. It can be seen from this that through the above-mentioned data lineage establishment method, when adjusting the scope of the business indicator, based on the established association relationship between the business indicators, technical indicators and application indicators, the technical indicators and application indicators associated with the business indicator to be adjusted can be quickly located, and the associated business indicator dimension information can be further located, thereby assisting the data warehouse side managers and the engineering side developers to quickly locate the application indicators and codes that need attention, effectively solving the problem of difficulty in sorting out and analyzing the impact of the modification of the stock data indicators on the product, without the need for manual inventory, greatly improving the development and iteration process of the data product.

[0113] In other embodiments of the present disclosure, assuming that a data product user feedbacks that a business indicator data on a certain front-end page does not meet expectations, in this case, the developer usually needs to determine the back-end development interface and engineering fields that need to be excluded. In response to the above requirements, in some embodiments of the present disclosure, the above-mentioned data lineage establishment method may further include: receiving the dimension information of the business indicator to be queried; based on the association relationship between technical indicators, business indicators, business indicator dimension information and application indicators, determining and outputting the identifier of the business indicator, the identifier of the technical indicator and the identifier of the application indicator corresponding to the dimension information of the business indicator to be queried. Normally, the dimension information of the above-mentioned business indicator to be queried may include: the name of the application indicator. In addition, the dimension information of the above-mentioned business indicator to be queried may also include the identifier of the front-end page, the identifier of the front-end module and other information that the user can see and extract when browsing the front-end page, thereby serving as a screening condition for the business indicator. From this, it can be seen that through the above-mentioned data lineage establishment method, when a data product user feedbacks that a business indicator data on a certain front-end page does not meet expectations, based on the established technical indicators, business indicators, business indicator dimension information and the association between application indicators, the technical indicator identifier and the application indicator identifier associated with the name of the application indicator on the front-end page reported by the data product user can be quickly located, thereby assisting developers on the engineering side to quickly locate the application indicators (back-end development interface identifiers and engineering field indexes) and codes that need attention, effectively solving the problem of high cost of online problem troubleshooting and positioning reported by users, without the need for manual inventory and without relying on the experience of developers, greatly improving the efficiency of online problem troubleshooting for data products.

[0114] In some other embodiments of the present disclosure, assuming that the testers of the application of the data product need to perform consistent interface automation testing for a certain business indicator, it is usually necessary to confirm in advance which back-end development interfaces or which data sets reveal this business indicator. In response to the above requirements, in some embodiments of the present disclosure, the above-mentioned data lineage establishment method may further include: receiving the identifier of the business indicator to be queried; based on the determined association relationship between the technical indicators, business indicators and application indicators, determining and outputting the identifier of the application indicator corresponding to the identifier of the business indicator to be queried. Alternatively, in some other embodiments of the present disclosure, the above-mentioned data lineage establishment method may further include: receiving the identifier of the business indicator to be queried; based on the determined association relationship between the technical indicators, business indicators, business indicator dimension information and application indicators, determining and outputting the business indicator dimension information, the identifier of the technical indicator and the identifier of the application indicator corresponding to the identifier of the business indicator to be queried. From this, it can be seen that through the above-mentioned data lineage establishment method, when the testers of the data product application need to perform consistent interface automation testing for a certain business indicator, based on the established relationship between technical indicators, business indicators and application indicators, they can quickly locate the application indicators associated with the tested business indicators, thereby assisting the testers on the engineering side to quickly locate the back-end development interface or data set that needs attention, effectively solving the problem that the consistency of test data caliber is difficult to automate, without the need for manual inventory or reliance on the experience of testers, greatly improving the efficiency of online problem troubleshooting of data products.

[0115] As can be seen, the above-mentioned data lineage establishment method, by cleaning and processing structured data from the product, engineering, and data warehouse sides, can determine the corresponding business indicators on the product side and technical indicators on the data warehouse side for each application indicator on the engineering side, thereby linking the data flow relationships of data products along the data processing chain. Ultimately, a comprehensive data lineage table is generated based on the indicator granularity. Each row in the data lineage table records the complete data processing chain from the product side to the engineering side and then to the data warehouse side for each indicator (referred to as business indicators on the product side, application indicators on the engineering side, and technical indicators on the data warehouse side). Furthermore, during the development, testing, and deployment of data products, the established data lineage table allows for quick and accurate locating of related data indicators from a specific data indicator along the data processing chain. This effectively addresses various issues encountered during the daily agile development and iteration of data products using a low-code architecture, such as the difficulty in analyzing the impact of changes to existing data indicators on the product due to long data transmission links and complex data mapping relationships, the high cost of troubleshooting and locating online issues reported by users, and the difficulty in automating the consistency of test data caliber.

[0116] Corresponding to the above-mentioned data lineage establishment method, some embodiments of the present disclosure further disclose a data lineage establishment device. Figure 7 The internal structure of the data lineage establishment device described in the embodiment of the present disclosure is shown. Figure 7 As shown, the data lineage establishment device may include the following modules:

[0117] The technical indicator extraction module 710 is used to read the metadata of the technical indicators configured for the data product and obtain the identifier of the technical indicator from the metadata of the technical indicator;

[0118] The business indicator reading module 720 is used to read the metadata of the business indicator configured for the data product, and obtain the identifier of the business indicator and the identifier of the technical indicator associated with the business indicator from the metadata of the business indicator;

[0119] The application indicator reading module 730 is used to read the configuration information of the application of the data product, and obtain the identifier of the application indicator involved in the application of the data product and the identifier of the technical indicator associated with the application indicator from the configuration information;

[0120] The data lineage establishment module 740 is used to determine the association relationship between technical indicators, business indicators and application indicators based on the identification of technical indicators, the identification of business indicators, the identification of technical indicators associated with business indicators, the identification of application indicators and the identification of technical indicators associated with application indicators.

[0121] It should be noted that the implementation methods of each module in the above-mentioned data lineage establishment device and the specific technical effects that can be achieved can all refer to the implementation methods of each step in the above-mentioned embodiment, and will not be repeated here.

[0122] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the computer program, the data lineage establishment method described in any of the above embodiments is implemented.

[0123] Figure 8 1 is a schematic diagram showing the hardware structure of a more specific electronic device provided in this embodiment. The device may include: a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. The processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are communicatively connected to each other within the device via the bus 2050.

[0124] The processor 2010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0125] The memory 2020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 2020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called and executed by the processor 2010.

[0126] The input / output interface 2030 is used to connect input / output devices to enable information input and output. Input / output devices can be configured as components within the device or externally connected to the device to provide corresponding functions. Input devices may include microphones and various sensors, while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0127] The communication interface 2040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0128] The bus 2050 comprises a path for transmitting information between the various components of the device (eg, the processor 2010 , the memory 2020 , the input / output interface 2030 , and the communication interface 2040 ).

[0129] It should be noted that although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040, and the bus 2050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0130] The electronic device of the above embodiment is used to implement the corresponding data lineage establishment method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0131] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the data lineage establishment method described in any of the above embodiments.

[0132] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0133] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the data lineage establishment method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0134] Based on the same inventive concept, corresponding to any of the aforementioned data lineage establishment methods, the present disclosure further provides a computer program product comprising computer program instructions. In some embodiments, when executed on a computer, the computer executes each step in each of the embodiments of the data lineage establishment method. Corresponding to the execution entity corresponding to each step in each of the embodiments of the data lineage establishment method, the processor executing the corresponding step may belong to the corresponding execution entity.

[0135] The computer program product of the above embodiment is used to enable the processor to execute the data lineage establishment method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0136] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0137] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0138] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0139] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for establishing data lineage, comprising: Reading metadata of technical indicators configured for the data product, and obtaining an identifier of the technical indicator from the metadata of the technical indicator; Reading metadata of the business indicator configured for the data product, and obtaining an identifier of the business indicator and an identifier of the technical indicator associated with the business indicator from the metadata of the business indicator; Reading configuration information of the application of the data product, and obtaining, from the configuration information, an identifier of an application indicator involved in the application of the data product and an identifier of the technical indicator associated with the application indicator; Based on the identification of the technical indicator, the identification of the business indicator, the identification of the technical indicator associated with the business indicator, the identification of the application indicator and the identification of the technical indicator associated with the application indicator, the association relationship between the technical indicator, the business indicator and the application indicator is determined.

2. The method according to claim 1, wherein The application of the data product includes: a data product front-end application and a data product back-end application; the reading of the configuration information of the application of the data product and obtaining the identifier of the application indicator involved in the application of the data product and the identifier of the technical indicator associated with the application indicator from the configuration information include: Reading configuration information of a front-end page in the data product front-end application, and obtaining an identifier of the application indicator involved in the front-end page from the configuration information of the front-end page; wherein the identifier of the application indicator includes: a back-end development interface identifier and a project field index associated with the application indicator; Read the metadata of the back-end development interface involved in the back-end application of the data product, and determine the identifier of the technical indicator associated with the application indicator from the metadata of the back-end development interface; wherein the identifier of the technical indicator includes: the identifier of the data model and the data model field index.

3. The method according to claim 2, further comprising: Business indicator dimension information associated with the application indicator is obtained from the configuration information of the front-end page; wherein the business indicator dimension information includes: one or more of: a data product identifier, a front-end page identifier, a front-end module identifier, and an application indicator name.

4. The method according to claim 1, wherein The application of the data product includes: a data dashboard application; the reading of configuration information of the application of the data product, and obtaining from the configuration information an identifier of an application indicator involved in the application of the data product and an identifier of the technical indicator associated with the application indicator includes: Reading metadata of the data dashboard application, and obtaining an identifier of the application indicator involved in the data dashboard application from the metadata of the data dashboard application; wherein the identifier of the application indicator includes: an identifier of a data set and a data set field index; The configuration information of the data set is read, and an identifier of a technical indicator associated with the application indicator is obtained from the configuration information of the data set; wherein the identifier of the technical indicator includes: an identifier of a data model and a data model field index.

5. The method according to claim 1, wherein The determining, based on the identifier of the technical indicator, the identifier of the business indicator, the identifier of the technical indicator associated with the business indicator, the identifier of the application indicator, and the identifier of the technical indicator associated with the application indicator, of the association relationship among the technical indicator, the business indicator, and the application indicator includes: For each of the business indicators, establishing an association relationship between the business indicator and the technical indicator according to the identifier of the technical indicator associated with the business indicator; For each of the application indicators, establishing an association relationship between the application indicator and the technical indicator according to the identifier of the technical indicator associated with the application indicator; and The association relationship between the technical indicator, the business indicator and the application indicator is determined based on the association relationship between the business indicator and the technical indicator and the association relationship between the application indicator and the technical indicator.

6. The method according to claim 3, wherein: The determining, based on the identifier of the technical indicator, the identifier of the business indicator, the identifier of the technical indicator associated with the business indicator, the identifier of the application indicator, and the identifier of the technical indicator associated with the application indicator, of the association relationship among the technical indicator, the business indicator, and the application indicator includes: For each of the business indicators, establishing an association relationship between the business indicator and the technical indicator according to the identifier of the technical indicator associated with the business indicator; For each of the application indicators, establishing an association relationship between the application indicator and the technical indicator and the business indicator dimension information according to the identifier of the technical indicator associated with the application indicator and the business indicator dimension information associated with the application indicator; and The association relationship between the technical indicator, the business indicator, the business indicator dimensional information and the application indicator is determined based on the association relationship between the business indicator and the technical indicator and the association relationship between the application indicator and the business indicator and the business indicator dimensional information.

7. The method according to claim 1, further comprising: Receive the identifier of the business indicator to be queried; as well as Based on the association relationship among the technical indicator, the business indicator and the application indicator, the identifier of the technical indicator and the identifier of the application indicator corresponding to the identifier of the business indicator to be queried are determined and output.

8. The method according to claim 6, further comprising: Receive the identifier of the business indicator to be queried; as well as Based on the association between the technical indicator, the business indicator, the business indicator dimension information and the application indicator, the business indicator dimension information, the identifier of the technical indicator and the identifier of the application indicator corresponding to the identifier of the business indicator to be queried are determined and output.

9. The method according to claim 6, further comprising: Receive dimension information of business indicators to be queried; as well as Based on the association between the technical indicator, the business indicator, the business indicator dimension information and the application indicator, the identifier of the business indicator, the identifier of the technical indicator and the identifier of the application indicator corresponding to the business indicator dimension information to be queried are determined and output.

10. A data lineage establishment device, comprising: A technical indicator extraction module is used to read metadata of technical indicators configured for data products and obtain identifiers of the technical indicators from the metadata of the technical indicators; A business indicator reading module, configured to read metadata of the business indicator configured for the data product, and obtain an identifier of the business indicator and an identifier of the technical indicator associated with the business indicator from the metadata of the business indicator; an application indicator reading module, configured to read configuration information of the application of the data product, and obtain, from the configuration information, an identifier of an application indicator involved in the application of the data product and an identifier of the technical indicator associated with the application indicator; as well as A data lineage establishment module is used to determine the association relationship between the technical indicator, the business indicator and the application indicator based on the identifier of the technical indicator, the identifier of the business indicator, the identifier of the technical indicator associated with the business indicator, the identifier of the application indicator and the identifier of the technical indicator associated with the application indicator.

11. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data lineage establishment method according to any one of claims 1 to 9 when executing the computer program.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the data lineage establishment method described in any one of claims 1-9.

13. A computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the data lineage establishment method according to any one of claims 1 to 9.

Images