Data lineage analysis method, product, device and medium of e-commerce data warehouse

Abstract

The application discloses an e-commerce data warehouse data blood relationship analysis method, product, equipment and medium, relates to the computer technical field, and includes the following steps: performing entity analysis on a data blood relationship analysis question sentence through a pre-trained language model, performing vectorization processing on obtained entity information, then matching the vectorized entity with the vectorized entity in the vector database in the pre-created blood relationship knowledge graph, querying the entity associated with the matched target vectorized entity from the graph database in the blood relationship knowledge graph, and finally inputting the matched entities from the two databases into the pre-trained language model to perform data blood relationship analysis. Based on the pre-trained language model and in combination with the blood relationship knowledge graph created based on the metadata management platform, automatic data blood relationship analysis is realized, which not only improves the efficiency and accuracy of e-commerce data warehouse data blood relationship analysis, but also saves labor cost and time cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to an e-commerce data warehouse data lineage analysis method, product, device and medium. BACKGROUND

[0002] With the rapid development of big data, the total amount of data resources generated is also increasing, and the data warehouse has the functions of integration, cleaning and storing data, and supports analysis and decision-making, so it can well store and manage data (such as enterprise data). At present, in the process of data warehouse processing, a large number of tables and data will be designed, in order to analyze the relationship between the tables and data in the whole process, the data lineage function of the data governance tool is usually used for sorting out. Data lineage can track the source, flow and dependency relationship of data, and can track the cause of the abnormality when the data is abnormal.

[0003] However, the current data governance tool (such as OpenMetadata, an open source metadata management platform) has the following problems in the process of data lineage analysis: usually based on SQL (Structured Query Language) syntax rules to analyze the data lineage relationship, if the parsing rule does not support the SQL syntax of the database, it will not be parsed; unable to automatically parse the configuration file of the ETL (Extract Transform Load) task across the data source; the current lineage relationship only supports visual display, when the business problem occurs, the business development engineer usually finds the related table based on the visual lineage map to locate the problem cause, which consumes a lot of manpower. SUMMARY

[0004] Therefore, the purpose of the present application is to provide an e-commerce data warehouse data lineage analysis method, product, device and medium, which can realize automatic data lineage analysis and improve the efficiency of data lineage analysis, save manpower cost and time cost. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses an e-commerce data warehouse data lineage analysis method, comprising:

[0006] obtaining a data lineage analysis question sentence for a target e-commerce data warehouse, performing entity analysis on the data lineage analysis question sentence by using a pre-trained language model to obtain current entity information;

[0007] performing vectorization processing on the current entity information to obtain a current vectorized entity;

[0008] The current vectorized entity is compared with the vectorized entities in the vector database to obtain an entity associated with the current vectorized entity, to obtain a target vectorized entity;

[0009] An entity associated with the target vectorized entity is queried from a graph database to obtain a target associated entity; the graph database and the vector database are located in a bloodline knowledge graph created in advance, the bloodline knowledge graph is a knowledge graph constructed based on a task configuration file and a structured query language file of the target e-commerce data warehouse and by using a metadata management platform;

[0010] The target vectorized entity and the target associated entity are input into a pre-trained language model to perform data bloodline analysis on a data bloodline analysis question sentence based on the target vectorized entity and the target associated entity, to obtain a data bloodline analysis result of the target e-commerce data warehouse.

[0011] In a second aspect, the present application further discloses a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the data bloodline analysis methods of the e-commerce data warehouse.

[0012] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the processor implements the data bloodline analysis method of the e-commerce data warehouse when executing the computer program stored in the memory.

[0013] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the data bloodline analysis method of the e-commerce data warehouse.

[0014] It can be seen that the present application performs entity analysis on the data bloodline analysis question sentence input by the user by using the pre-created pre-trained language model, and performs vectorization processing on the obtained entity information, then matches the vectorized entity with the vectorized entities in the pre-created vector database, queries the entities associated with the matched target vectorized entity from the graph database, and finally inputs the entities matched from the two databases into the pre-trained language model to perform data bloodline analysis; it should be noted that the two databases are located in the bloodline knowledge graph created in advance based on the task configuration file and the structured query language file of the e-commerce data warehouse and by using the metadata management platform, and the above-mentioned method based on the pre-trained language model and combined with the bloodline knowledge graph created based on the metadata management platform realizes automatic data bloodline analysis, which not only improves the efficiency of data bloodline analysis of the e-commerce data warehouse, but also saves the labor cost and time cost, and improves the accuracy of data bloodline analysis. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only constitute a part of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0016] Figure 1 A data bloodline analysis method flowchart of an e-commerce data warehouse disclosed by the present application;

[0017] Figure 2 A specific bloodline knowledge graph creation flowchart disclosed by the present application;

[0018] Figure 3 Another specific bloodline knowledge graph creation flowchart disclosed by the present application;

[0019] Figure 4 A specific data bloodline analysis method flowchart of an e-commerce data warehouse disclosed by the present application;

[0020] Figure 5 A data bloodline analysis method flowchart of an e-commerce data warehouse disclosed by the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only constitute a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0022] The embodiments of the present application disclose a data bloodline analysis method of an e-commerce data warehouse, as shown in Figure 1 The method comprises the following steps.

[0023] Step S11: Obtain a data bloodline analysis question sentence for a target e-commerce data warehouse, and perform entity analysis on the data bloodline analysis question sentence by using a pre-trained language model to obtain current entity information.

[0024] In the embodiments, first, a data bloodline analysis question sentence input by a user for a certain e-commerce data warehouse is obtained, and then a pre-trained language model (LLM, Large Language Model), such as GPT (a natural language processing model based on the Transformer architecture) or Claude, is used to perform entity analysis on the data bloodline analysis question sentence to obtain current entity information.

[0025] The e-commerce data warehouse is a system that integrates, cleans, and stores e-commerce data to support analysis and decision-making. The specific construction process is as follows: data access: extract raw e-commerce data (i.e., ETL data) from business systems, logs, etc.; layered processing: clean and integrate (such as de-duplication, association) according to the ODS layer (Operational Data Store) -> DWD layer (Data Warehouse Detail) -> DWS layer (Data Warehouse Summary) -> ADS layer (Application Data Service) layering; modeling and storage: design dimension models (such as star / snowflake, etc.) and store them in the data warehouse (such as Hive); service application: for BI (a solution that uses big data technology and tools to collect, store, analyze, and visualize massive data to support enterprise decision-making), reporting, and analysis.

[0026] Specifically, the ODS layer is the raw data layer of the data warehouse, used to store raw data, which is a collection of data tables such as start log tables, order tables, and user tables. The data source is other business systems and logs, and the data remains unchanged without processing. The DWD layer is the detailed data layer of the data warehouse, with the same structure and granularity as the original table. It is used to clean the ODS layer table data (such as removing null values, dirty data, and data beyond the limit range), and is a collection of data tables such as user operation event tables and product tables. The DWS layer is the service data layer of the data warehouse, based on the DWD layer, and performs light aggregation. It is a collection of data tables such as daily active user tables and weekly active user tables. The ADS layer is the data application layer of the data warehouse, used to provide data for various statistical reports, and is a collection of data tables such as daily new user tables and recent three-day active user tables.

[0027] It should be noted that in the data access process, the ETL task (such as Chunjun, DataX, etc.) imports external data source data (such as MySQL, a relational database management system) to the data warehouse (such as Hive), and the ETL task reads the Json format configuration file (i.e., ETL configuration file) to implement data import from the table in the MySQL service to the table in the Hive service. The ETL configuration file contains source information and destination information. Specifically, the source information includes service information, table information, and field information, and the destination information includes service information, table information, and field information. A specific example is as follows:

[0028] {

[0029] Source information:

[0030] Service IP: 192.168.1.1

[0031] Service Port: 3306

[0032] Table Name: user_info

[0033] Table Field Information: id, name

[0034] Destination Information: {

[0035] Service IP: 192.168.1.2

[0036] Service Port: 10000

[0037] Table Name: ods_user_info

[0038] Table Field Information: id, name

[0039] }.

[0040] Specifically, in the process of data layer processing, a large number of tables and SQL statements are involved. Most of the SQL statements are to process the data in the previous layer table and insert it into the next layer table. Each SQL involves data flow between tables. In order to analyze the relationship between data flow in tables in the whole process, the data lineage function of the data governance tool (such as OpenMetadata) needs to be used to sort out the relationship. Specific cases are as follows:

[0041] insert table dwd_user_info from select id, name from ods_user_info where id!=null;

[0042] In the above SQL case, the data flow from ods_user_info to dwd_user_info is realized, and the data with an empty id (Identity document, identity card identification number) is filtered out.

[0043] Further, through data lineage, the source, flow, and dependency relationship of data can be tracked, and field-level data lineage (Field-Level Lineage) is further refined to a single field (such as a column of a database table), recording its generation logic, conversion process, and final destination.

[0044] Step S12: vectorizing the current entity information to obtain a current vectorized entity.

[0045] In this embodiment, after the data blood relationship analysis question sentence input by the user is parsed to obtain the current entity information, the current entity information obtained after the entity parsing is further vectorized to obtain the current vectorized entity.

[0046] Step S13: Compare the current vectorized entity with the vectorized entities in the vector database to obtain the entities associated with the current vectorized entity, and obtain the target vectorized entity.

[0047] In this embodiment, after the current entity information is vectorized, the current vectorized entity obtained after the processing is compared with the vectorized entities in the pre-created vector database, so as to obtain the entities associated with the current vectorized entity, and obtain the target vectorized entity.

[0048] It should be noted that the vector database is a database system for storing, indexing and querying high-dimensional vector data, and through calculating the similarity (such as cosine similarity, Euclidean distance, etc.) between vectors, efficient semantic search and matching of unstructured data (such as images, texts, audios, etc.) can be realized, which can be widely applied in recommendation systems, image retrieval and natural language processing fields. Specifically, all information such as tables, columns, service types, etc. in the entity information can be vectorized and stored in the vector database.

[0049] Step S14: Query the entities associated with the target vectorized entity from the graph database to obtain the target associated entity; the graph database and the vector database are located in the pre-created blood relationship knowledge graph, and the blood relationship knowledge graph is a knowledge graph constructed based on the task configuration file and the structured query language file of the target e-commerce data warehouse and using the metadata management platform.

[0050] In this embodiment, the target vectorized entity can also be matched with the entities in the pre-created graph database, so as to query the target associated entity associated with the target vectorized entity; wherein the graph database and the vector database are located in the pre-created blood relationship knowledge graph, and the blood relationship knowledge graph is a knowledge graph constructed based on the task configuration file (such as ETL task configuration file, used for realizing data integration and cleaning, for example, extracting, converting and loading the data in the MySQL table to the Hive data warehouse) and the structured query language file (i.e. SQL file, a file of a standard programming language for managing and operating a relational database) of the target e-commerce data warehouse, and using the metadata management platform (such as OpenMetadata) to construct.

[0051] It should be noted that the graph database is a point and edge-based storage unit, which is a non-relational database specially used for efficient storage and query of highly interconnected data, including but not limited to Neo4j, Galaxybase, etc. For example, when the entity information includes table name, table field information, service information, table metadata ID, column metadata ID, table description, and column description information, the graph database includes:

[0052] Table name: user_info_table;

[0053] Columns: user_id, user_name;

[0054] Service type: MySQL;

[0055] Service IP: 192.168.1.1;

[0056] Table metadata ID: 1;

[0057] Column metadata ID: 1.1, 1.2;

[0058] Table description: original business user information table;

[0059] Column description: user ID, user name;

[0060] Among them, the information of the edge is as follows: {source_table_id: 1, target_table_id: 2}.

[0061] Among them, OpenMetadata is an open source metadata management platform that supports data discovery, lineage tracking, data quality monitoring, and collaboration, which can help developers and operation and maintenance personnel to uniformly manage data assets, and one of its core functions is data lineage tracking. It can automatically parse SQL and build complete links from raw data to final reports through manual editing, and visually display field or table-level dependency relationships in a visual graph. Through dynamic lineage analysis, users can quickly locate data error sources, assess the impact of table metadata changes (such as changing the field name of a user table's phone field from phone to phone_number), and meet compliance audit requirements. In addition, OpenMetaData supports data asset acquisition, which can be achieved by running a timing script to obtain real-time information about data assets. In this platform, data assets are table information and metadata information corresponding to the table in databases such as MySQL and Hive. A specific data asset is as follows:

[0062] Data asset:

[0063] Table name: user_info;

[0064] Service name: MysqlService;

[0065] Description: User business information table;

[0066] Field list: user_id, user_name;

[0067] Field description: user ID, user name;

[0068] Field type: numeric type, string type;

[0069] Table id: 1;

[0070] Column id: 1.1, 1.2.

[0071] Further, after obtaining the data asset, OpenMetaData supports data quality monitoring of the asset. The process is to perform data quality analysis on the data in the table through a scheduled timing script and store it in the platform. The data quality analysis content includes the number of null fields in a column in the table, the total number of data in the table, whether the value of a column in the table contains negative numbers, and other information. The specific analysis of what data quality is configured by the development and operation and maintenance personnel.

[0072] In addition, OpenMetadata supports cross-system bloodline integration (such as databases, data lakes, BI tools), combined with version management functions, which can not only monitor data flow in real time, but also trace historical bloodline status, providing end-to-end observability for data governance. Ultimately, through automatic parsing of SQL and manual maintenance, a data flow map can be drawn to know where the data comes from, how it is processed, and where it is used.

[0073] The creation process of the blood relationship knowledge graph can specifically include: obtaining a task configuration file and a corresponding structured query language file for a target e-commerce data warehouse; performing entity analysis on the task configuration file by using a pre-trained language model to obtain first historical entity information; obtaining data asset information pre-constructed in a metadata management platform, and perfecting the first historical entity information by using the data asset information to obtain first perfected entity information; analyzing the mapping relationship between different entities in the task configuration file by using the pre-trained language model to obtain first entity blood relationship information, and saving the first entity blood relationship information and the first perfected entity information to a graph database; vectorizing the first perfected entity information to obtain first vectorized entities, and saving the first vectorized entities to a vector database; performing entity analysis on the structured query language file by using the pre-trained language model to obtain second historical entity information; perfecting the second historical entity information by using the data asset information to obtain second perfected entity information; analyzing the mapping relationship between different entities in the second perfected entity information by using the pre-trained language model to obtain second entity blood relationship information; saving the second entity blood relationship information and the second perfected entity information to the graph database; vectorizing the second perfected entity information to obtain second vectorized entities, and saving the second vectorized entities to the vector database, to obtain a blood relationship knowledge graph including the graph database and the vector database. Figure 2 As shown in FIG. 15, the task configuration file (such as an ETL task configuration file) and the corresponding structured query language file (i.e., SQL file) for any e-commerce data warehouse can be obtained first, and then the pre-trained language model (such as LLM) is used to perform entity analysis on the task configuration file (such as ETL task configuration file) to obtain first historical entity information (including source entity and destination entity). Then, the data asset information (metadata information obtained from the existing functions of OpenMetadata) pre-constructed in the metadata management platform (i.e., OpenMetadata) is obtained, and then it is determined whether there is an entity corresponding to the first historical entity information in the data asset information based on table and field comparison. If there is, a preset interface (such as API, Application Programming Interface) is called and the corresponding data asset information is used to perfect the content in the first historical entity information to obtain corresponding first perfected entity information, and if there is not, the data blood relationship analysis process is ended.

[0074] Further, the preset prompt word is used to guide the pre-training language model (i.e., LLM) to analyze the mapping relationship between different entities in the task configuration file (i.e., ETL task configuration file), to obtain first entity blood relationship information (including table entity, field entity, table entity relationship and field entity relationship), and then save the table entity and field entity in the first entity blood relationship information as points and the table entity relationship and field entity relationship as edges to the graph database. In addition, referring to Figure 3 As shown, the preset prompt word can also be used to guide the pre-training language model (i.e., LLM) to perform entity analysis on the structured query language file (i.e., SQL file) to obtain second historical entity information, and then the data asset information in the metadata management platform is used to perfect the content in the second historical entity information to obtain second perfect entity information, and then the second entity blood relationship information and the second perfect entity information are saved to the graph database, and finally, the second perfect entity information (such as table entity, field entity, etc.) is vectorized, and the obtained second vectorized entity is saved to the vector database, thereby constructing a blood knowledge graph including a graph database and a vector database.

[0075] In addition, through the API of OpenMetaData, not only the existence of the entity can be verified, but also the asset ID, field type and other key attributes can be supplemented. By combining the pre-training language model (i.e., LLM) with the metadata management platform (i.e., OpenMetadata), a blood knowledge graph including a graph database and a vector database is constructed, solving the two bottleneck problems existing in traditional data blood tools. One is to solve the blood analysis problem of cross-source ETL tasks: the traditional tools (such as OpenMetaData) cannot automatically analyze the configuration file (JSON format) of ETL tools such as DataX / Chunjun, and manual configuration of cross-source table field mapping is required; the second is to solve the SQL syntax compatibility problem: traditional blood analysis relies on pre-set SQL syntax rules, which is difficult to cover the SQL dialect of heterogeneous databases. In addition, automatic data blood analysis is realized, thereby improving the efficiency of real-time data blood analysis.

[0076] Specifically, the pre-training language model is used to perform entity analysis on the task configuration file to obtain first historical entity information, which can include: using the pre-training language model to perform entity analysis on the task configuration file to obtain first source entity and first destination entity; wherein the first source entity and the first destination entity each include table name, table field information and service information. In this embodiment, the preset prompt word 1 is used to guide the pre-training language model (i.e., LLM) to perform entity analysis on the source entity and the destination entity in the ETL task configuration file (which can be provided by the development and operation personnel) to obtain the corresponding first source entity and the first destination entity.

[0077] In a specific implementation, the prompt word 1 can be specifically:

[0078] You are a professional data integration expert, and you need to accurately parse the ETL job configuration file. Please extract the following key information from the input ETL JSON configuration and output it in standard JSON format:

[0079] # Input requirements

[0080] - Complete ETL job JSON configuration

[0081] # Processing rules

[0082] 1. Extract the configuration information of reader and writer respectively;

[0083] 2. Automatically desensitize sensitive fields such as passwords (display as ****);

[0084] 3. Maintain the original case of fields;

[0085] 4. If there are multiple tables in the configuration, group the output by table;

[0086] # Output JSON format requirements

[0087] {

[0088] "job": {

[0089] "setting": {

[0090] "speed": {"channel": "actual value"},

[0091] "errorLimit": {"record": "actual value"}

[0092] },

[0093] "content": [

[0094] {

[0095] "reader": {

[0096] "type": "data source type",

[0097] "connection": {

[0098] "jdbcUrl": ["connection address"],

[0099] "table": ["table name"],

[0100] "schema": "database / schema name"

[0101] },

[0102] "username": "username",

[0103] "password": "******",

[0104] "column": ["column1", "column2"],

[0105] "where": "query condition (if any)"

[0106] },

[0107] "writer": {

[0108] "type": "data source type",

[0109] "connection": {

[0110] "jdbcUrl": ["connection address"],

[0111] "table": ["table name"],

[0112] "schema": "database / schema name"

[0113] },

[0114] "username": "username",

[0115] "password": "******",

[0116] "column": ["column1", "column2"],

[0117] "preSql": ["pre-SQL (if any)"]

[0118] }

[0119] } ]

[0121] },

[0122] "remark": "automatically generated configuration summary"

[0123] }

[0124] # Special handling instructions

[0125] 1. Special handling for different data source types:

[0126] - MySQL / Oracle: extract jdbcUrl, schema, table

[0127] - HDFS: extract path, defaultFS

[0128] - MongoDB: extract uri, collectionName

[0129] 2. The field mapping relationship maintains the original order;

[0130] 3. If there is a sharding configuration, note it in remark;

[0131] Please analyze the following ETL configuration:

[0132] [User input paste area].

[0133] Through the above prompt 1, the pre-trained language model (i.e., LLM) can be guided to perform entity analysis on the source entities and destination entities in the ETL task configuration file (which can be provided by development and operation personnel), obtaining the corresponding first source entities and first destination entities. It should be noted that the design of the prompt for the ETL task configuration file (JSON format) and the SQL file needs to ensure that the structured lineage relationship can be output.

[0134] Specifically, the first source entity (including table name, table field information, service information) is:

[0135] Table name: user_info;

[0136] Columns: user_id, user_name;

[0137] Service type: MySQL;

[0138] Service IP: 192.168.1.1.

[0139] Correspondingly, the first destination entity (including table name, table field information, service information) is specifically:

[0140] Case: Table name: ods_user_info;

[0141] Columns: id, name;

[0142] Service type: Hive;

[0143] Service IP: 192.168.1.2.

[0144] In a specific embodiment, the data asset information pre-constructed in the metadata management platform is acquired, and the first historical entity information is perfected through the data asset information to obtain first perfected entity information, which can specifically include: calling a metadata query interface of the metadata management platform to acquire data asset information corresponding to the first source end entity and the first destination end entity from the metadata management platform; and perfecting the first source end entity and the first destination end entity based on the data asset information to obtain a second source end entity and a second destination end entity; wherein the second source end entity and the second destination end entity each include a table name, table field information, service information, a table metadata identifier, a column metadata identifier, table description, and column description. In this embodiment, the metadata query interface of OpenMetaData can be called, the data asset information already constructed in OpenMetaData is queried based on the table name, table field, and service information in the first source end entity and the first destination end entity, and then the relevant information of the first source end entity and the first destination end entity is perfected based on the corresponding data asset information queried to obtain the second source end entity and the second destination end entity after perfection.

[0145] For example, the metadata query interface of OpenMetaData is called to query the following data asset information:

[0146] Data asset 1:

[0147] Table name: user_info;

[0148] Service name: MysqlService;

[0149] Description: user business information table;

[0150] Field list: user_id, user_name;

[0151] Field description: user ID, user name;

[0152] Field type: numeric type, string type;

[0153] Table id: 1;

[0154] Column id: 1.1, 1.2;

[0155] Data asset 2:

[0156] Table name: ods_user_info;

[0157] Service name: HiveService;

[0158] Description: user information original table;

[0159] Field list: id, name;

[0160] Field description: user ID, user name;

[0161] Field type: number type, string type;

[0162] Table id: 2;

[0163] Column id: 2.1, 2.2.

[0164] Then, based on the queried data asset 1 and data asset 2, the related information of the first source end entity and the first destination end entity is perfected to obtain the perfected second source end entity and the second destination end entity.

[0165] Among them, the second source end entity (including table name, table field information, service information, table metadata ID, column metadata ID, table description and column description) is:

[0166] Table name: user_info_table;

[0167] Column: user_id, user_name;

[0168] Service type: MySQL;

[0169] Service IP: 192.168.1.1;

[0170] Table metadata ID: 1;

[0171] Column metadata ID: 1.1, 1.2;

[0172] Table description: original business user information table;

[0173] Column description: user ID, user name.

[0174] Correspondingly, the second destination end entity (including table name, table field information, service information, table metadata ID, column metadata ID, table description and column description) is:

[0175] Table name: ods_user_info_table;

[0176] Column: id, name;

[0177] Service type: Hive;

[0178] Service IP: 192.168.1.1;

[0179] Table metadata ID: 2;

[0180] Column metadata ID: 2.1, 2.2;

[0181] Table description: user information table of the data layer of the source data;

[0182] List description: user ID, user name.

[0183] Among them, the table metadata ID, column metadata ID, table description, and column description are information already existing in OpenMetaData, which is returned by querying the interface for supplementation.

[0184] Perfecting the entity obtained by parsing the ETL task configuration file through the data asset information helps to build a more accurate blood relationship knowledge graph, thereby improving the accuracy of real-time data blood relationship analysis.

[0185] Specifically, the mapping relationship between different entities in the task configuration file is parsed using a pre-trained language model to obtain first entity blood relationship information, which can include: the field mapping relationship between the second source entity and the second destination entity is parsed using a pre-trained language model to obtain first entity blood relationship information. In this embodiment, the pre-trained language model (i.e. LLM) can be guided by a pre-set prompt word 2 to parse the mapping relationship (in Json format) between the second source entity and the second destination entity, and obtain the corresponding first entity blood relationship information.

[0186] Among them, the first entity blood relationship information includes table blood relationship information and field blood relationship information. For example, the table blood information: source table metadata ID, destination table metadata ID; the field blood information: the list composed of the source table field metadata ID and the destination table field metadata ID.

[0187] In a specific implementation, the prompt word 2 can be specifically:

[0188] You are a professional data blood relationship analysis engine. Please strictly follow the following steps to process the input ETL configuration file (JSON format):

[0189] ---

[0190] 1. **Entity Extraction**

[0191] - Source entity: extract `database` (library name), `table` (table name), `columns` (field list) in the `source` field

[0192] - Destination entity: extract `database` (library name), `table` (table name), `columns` (field list) in the `target` field

[0193] - Example: {"source": {"database": "sales_db", "table": "orders", "columns": ["order_id","amount"]},...}

[0194] 2. **Field Mapping Analysis**

[0195] - Establish mapping pairs (1:1) between source fields and target fields

[0196] - Example: {"mappings": [{"source": "order_id", "target": "id"},...]}

[0197] 3. **Structured Output**

[0198] Must generate the following JSON structure:

[0199] {

[0200] "source_entity": {

[0201] "database": "string",

[0202] "table": "string",

[0203] "columns": ["string"]

[0204] },

[0205] "target_entity": {

[0206] "database": "string",

[0207] "table": "string",

[0208] "columns": ["string"]

[0209] },

[0210] "field_mappings": [

[0211] {"source_field": "string", "target_field": "string"} ]

[0213] }

[0214] **Processing Rules**:

[0215] 1. Ignore comments and empty fields in JSON;

[0216] 2. If the field has conversion logic (such as `CONCAT(first_name, last_name)`), keep the original expression.

[0217] The above prompt 2 can guide the pre-trained language model (i.e., LLM) to parse the mapping relationship (in Json format) between the second source entity and the second destination entity, and obtain the corresponding first entity blood relationship information, for example:

[0218] table_info:{“source_table_id”: 1,“target_table_id”: 2}

[0219] column_info:[{“source_id”: 1.1,“target_table_id”: 2.1},{“source_id”: 1.2,“target_table_id”: 2.2}]。

[0220] Specifically, the first entity blood relationship information and the first improved entity information are saved to the graph database, which can include: based on the first entity blood relationship information, the initial data blood graph in the metadata management platform is improved to obtain the first improved data blood graph; the second source entity and the second destination entity are taken as the points of the graph database, and the table blood relationship information and the field blood relationship information in the first entity blood relationship information are taken as the edges of the graph database, and saved to the graph database. In the embodiment, as shown in Figure 2 , based on the first entity blood relationship information (including table entity, field entity, table entity relationship and field entity relationship), the existing data blood information (i.e., blood graph) in the metadata management platform is improved to obtain the first improved data blood graph, and then the improved second source entity and the second destination entity (including the related information of the entity) are taken as the points of the graph database, and the table blood relationship information and the field blood relationship information in the first entity blood relationship information are taken as the edges of the graph database, and saved to the graph database. At the same time, the improved second source entity and the second destination entity (including the related information of the entity) are vectorized by the vector model (a technology for converting discrete data (such as text, image, etc.) into continuous numerical vectors, and realizing semantic matching and retrieval by calculating the similarity between vectors), and stored in the vector database.

[0221] In a specific embodiment, the initial data bloodline graph in the metadata management platform is perfected based on the first entity blood relationship information, and a first perfected data bloodline graph is obtained, which can specifically include: sending the first entity blood relationship information to a bloodline construction interface of the metadata management platform, constructing a data bloodline connection between corresponding data assets in the data asset information based on the identifier information in the second source entity and the second destination entity and the first entity blood relationship information, perfecting the initial data bloodline graph in the metadata management platform, and obtaining the first perfected data bloodline graph. In this embodiment, the related ID information in the first entity blood relationship information can be sent to the bloodline construction interface of OpenMetaData first, and a data bloodline connection between data asset 1 and data asset 2 is constructed based on the identifier information in the second source entity and the second destination entity and the first entity blood relationship information, so as to perfect the initial data bloodline graph in OpenMetaData.

[0222] It should be noted that after the initial data bloodline graph is perfected, the development and operation personnel can check the generated data bloodline connection through the OpenMetaData platform, and if it is not accurate, it can be manually perfected, so as to ensure the accuracy of the obtained data bloodline graph.

[0223] Further, after saving the first vectorized entity into the vector database, the method can further include: calling a data lineage acquisition interface of the metadata management platform to obtain a first perfected data lineage graph; saving table information in the first perfected data lineage graph as a point of a graph database and saving data connections between tables in the first perfected data lineage graph as edges of the graph database into the graph database; vectorizing the second source end entity and the second destination end entity through the vector model to obtain a third vectorized entity, and saving the third vectorized entity into the vector database. In this embodiment, the data lineage acquisition interface in OpenMetadata can be called to obtain the first perfected data lineage graph in the OpenMetadata platform, and then the table information in the first perfected data lineage graph can be saved as a point of a graph database and the data connections between tables can be saved as edges of the graph database into the graph database. Further, the table information in the second source end entity and the second destination end entity can be converted into vectors by using the vector model and stored in the vector database, so as to perfect the overall data lineage knowledge graph. In addition, when the information obtained from the OpenMetaData platform conflicts with the first perfected data lineage graph, the OpenMetaData platform is used as the reference. By analyzing the ETL task configuration file and perfecting the data lineage relationship based on the ETL task configuration file, the lineage relationship is pushed to the metadata management platform (i.e., OpenMetadata), which can supplement the function of manually configuring the ETL data lineage relationship for the metadata management platform (i.e., OpenMetadata).

[0224] In a specific implementation, the second historical entity information is obtained by performing entity analysis on the structured query language file by using the pre-trained language model, and specifically can include: performing entity analysis on the structured query language file by using the pre-trained language model to obtain third source end entities and third destination end entities; the third source end entities and the third destination end entities each include table names, table field information, and service information. In this embodiment, the pre-trained language model (i.e., LLM) can be guided by using the preset prompt word 3 to perform entity analysis on the source end entities and the destination end entities (including table name lists in the SQL file) in the SQL file of the data warehouse (which can be provided by a development and operation personnel and includes service names in the OpenMetaData and corresponding SQL statements) to obtain corresponding third source end entities and third destination end entities.

[0225] In a specific implementation, the prompt word 3 can be:

[0226] ## Role definition

[0227] You are a professional SQL parsing engineer who needs to accurately extract all physical table names from various complex SQL statements and output the results in a structured Markdown format.

[0228] ## Core Tasks

[0229] Identify and output all physical table names (excluding subqueries / views / temporary tables) from the input SQL, with the following requirements:

[0230] 1. Identify table names after the following keywords:

[0231] - `FROM`

[0232] - `JOIN` (including `INNER / LEFT / RIGHT JOIN`)

[0233] - `UPDATE`

[0234] - `INTO`

[0235] - `INSERT INTO`

[0236] 2. Handle cases with schema names (`schema.table`) and database names (`db.schema.table`);

[0237] 3. Automatically remove aliases after the `AS` keyword (e.g., `a AS b` → `a`);

[0238] 4. Exclude table names in nested subqueries (only keep the outermost tables);

[0239] 5. Handle special symbols (e.g., tables with quotes `` `table` `` or `"table"`).

[0240] ### Processing Rules

[0241] 1. **Schema Extraction**: `schema.table` → `table`;

[0242] 2. **Database Filtering**: Ignore non-target database prefixes like `other_db`;

[0243] 3. **Alias Handling**: Remove aliases after the `AS` keyword (e.g., `a AS b` → `a`);

[0244] 4. **Special Symbols**: Uniformly remove symbols such as `、`, `、`, etc.

[0245] 5. **Duplicate Filtering**: Automatically remove duplicates and maintain the order of appearance.

[0246] The pre-trained language model (i.e., LLM) can be guided by the above prompt word 3 to parse the table name list in the SQL file, for example:

[0247] The SQL file is:

[0248] Service name: HiveService;

[0249] SQL: insert table dwd_user_info from select id, name from ods_user_info where id!= null.

[0250] The result of parsing the table name list in the above SQL file by the LLM is: dwd_user_info, ods_user_info.

[0251] Correspondingly, the second historical entity information is perfected by the data asset information to obtain the second perfected entity information, which can specifically include: calling a metadata query interface to obtain data asset information corresponding to the third source entity and the third destination entity from a metadata management platform; based on the data asset information, the third source entity and the third destination entity are perfected to obtain the fourth source entity and the fourth destination entity; wherein the fourth source entity and the fourth destination entity each include table name, table field information, service information, table metadata identifier, column metadata identifier, table description, and column description. For example, by calling the metadata query interface of the metadata management platform (i.e., OpenMetadata), data asset 2 and data asset 3 corresponding to the third source entity and the third destination entity are obtained; wherein data asset 2 is specifically:

[0252] Table name: ods_user_info;

[0253] Service name: HiveService;

[0254] Description: User Information Original Table;

[0255] Field list: id, name;

[0256] Field description: User ID, User Name;

[0257] Field type: numeric type, string type;

[0258] Table metadata ID: 2;

[0259] Column metadata ID: 2.1, 2.2.

[0260] Data asset 3 is specifically:

[0261] Table name: dwd_user_info;

[0262] Service name: HiveService;

[0263] Description: User information cleaning table, cleaning ID part empty user information;

[0264] Field list: id, name;

[0265] Field description: User ID, user name;

[0266] Field type: numeric type, string type;

[0267] Table metadata ID: 3;

[0268] Column metadata ID: 3.1, 3.2.

[0269] Then, based on the data asset 2 and the data asset 3, the information in the third source end entity and the third destination end entity is perfected, and the fourth source end entity and the fourth destination end entity are obtained;

[0270] Among them, the fourth source end entity (including table name, table field information, service information, table metadata ID, column metadata ID, table description and column description) is specifically:

[0271] Table name: ods_user_info;

[0272] Column: id, name;

[0273] Service type: HiveService;

[0274] Service IP: 192.168.1.1;

[0275] Table metadata ID: 2;

[0276] Column metadata ID: 2.1, 2.2;

[0277] Table description: User information table of the source data layer;

[0278] List description: User ID, user name.

[0279] Correspondingly, the fourth destination end entity (including table name, table field information, service information, table metadata ID, column metadata ID, table description and column description) is specifically:

[0280] Table name: dwd_user_info;

[0281] Column: id, name;

[0282] Service type: Hive;

[0283] Service IP: 192.168.1.1;

[0284] Table Metadata ID: 3;

[0285] Column Metadata ID: 3.1, 3.2;

[0286] Table Description: Processed User Information Table;

[0287] Column Description: User ID, User Name.

[0288] Among them, the table metadata ID, column metadata ID, table description, and column description are information already existing in OpenMetaData, which can be returned by querying the interface for supplementation.

[0289] In a specific embodiment, the mapping relationship between different entities in the second perfected entity information is analyzed using a pre-trained language model to obtain second entity blood relationship information, which can specifically include: the field mapping relationship between the fourth source entity and the fourth destination entity is analyzed using a pre-trained language model to obtain second entity blood relationship information; the second entity blood relationship information includes table blood relationship information and field blood relationship information.

[0290] In this embodiment, the pre-trained language model (i.e., LLM) can be guided by the preset prompt word 4 to analyze the mapping relationship between different entities in the second perfected entity information to obtain the corresponding first entity blood relationship information (including table blood relationship information and field blood relationship information).

[0291] In a specific embodiment, the prompt word 4 can be specifically:

[0292] You are a SQL semantic analysis engine. Please parse the SQL statement according to the following framework, output in Json format:

[0293] ---

[0294] 1. **Entity Recognition**

[0295] - Extract table entities (including alias resolution) in FROM / JOIN clauses

[0296] - Identify SELECT / INSERT fields and associated tables

[0297] - Example: {"tables": ["sales.orders", "customers"]}

[0298] 2. **Field-level Bloodline Analysis**

[0299] - Parsing field transformation logic (especially expression fields):

[0300] ```sql

[0301] SELECT

[0302] o.id AS order_id, -- Source: orders.id

[0303] CONCAT(c.first_name, c.last_name) AS customer_name -- Source concatenation field

[0304] ```

[0305] - Establishing field-level dependency links

[0306] 3. **Structured Output**

[0307] The following JSON structure must be generated:

[0308] {

[0309] "tables_involved": ["schema.table"],

[0310] "field_lineage": [

[0311] {

[0312] "target_field": "string", -- Target field name

[0313] "target_table": "schema.table", -- Target table

[0314] "source_expression": "string", -- Original SQL expression

[0315] "source_fields": [ -- Dependent source fields

[0316] {"field": "string", "table": "schema.table"} ]

[0318] }

[0319] ],

[0320] "table_lineage": [

[0321] {

[0322] "source_table": "schema.table",

[0323] "target_table": "schema.table",

[0324] "join_condition": "string" -- Join condition (if any)

[0325] } ]

[0327] }

[0328] **Parsing Rules**:

[0329] 1. Recursive parsing when processing CTEs: Treat the WITH clause as a temporary table.

[0330] 2. Subqueries are marked as {"subquery": "SELECT ..."}

[0331] 3. Special syntax compatibility:

[0332] - Window functions: Preserve the OVER() expression

[0333] - Cross-database queries: Preserve the complete database.schema.table path

[0334] - Macros / functions: Preserve function names and parameters.

[0335] The aforementioned prompt word 4 guides the pre-trained language model (LLM) to parse the field mapping relationship between the fourth source entity and the fourth destination entity, obtaining the corresponding second entity lineage information (in JSON format), specifically including table lineage information and field lineage information. The table lineage information includes the source table metadata ID and the destination table metadata ID, while the field lineage information includes a list composed of the source table field metadata ID and the destination table field metadata ID. For example:

[0336] table_info:{"source_table_id": 2,"target_table_id": 3}

[0337] column_info: [{“source_id”: 2.1, “target_table_id”: 3.1}, {“source_id”: 2.2, “target_table_id”: 3.2}].

[0338] Correspondingly, the second entity blood relationship information and the second improved entity information are saved into the graph database, which can specifically include: improving the first improved data blood graph in the metadata management platform based on the second entity blood relationship information to obtain a second improved data blood graph; taking the fourth source entity and the fourth destination entity as points of the graph database, and taking the table blood relationship information and the field blood relationship information in the second entity blood relationship information as edges of the graph database, and saving them into the graph database. In this embodiment, the related ID information in the second entity blood relationship information can be sent to the blood construction interface of the OpenMetaData platform to construct the data blood connection between the data asset 2 and the data asset 3, so as to improve the first improved data blood graph in the OpenMetaData platform to obtain the second improved data blood graph. At this time, the development and operation personnel can check the generated blood connection through the OpenMetaData platform, and if it is not accurate, it can be manually improved to ensure the accuracy of the obtained data blood graph.

[0339] Further, after saving the second vectorized entity into the vector database, it can further include: taking the table information in the second improved data blood graph as the point of the graph database, and taking the data connection between the tables in the second improved data blood graph as the edge of the graph database, and saving them into the graph database; vectorizing the fourth source entity and the fourth destination entity through the vector model to obtain a fourth vectorized entity, and saving the fourth vectorized entity into the vector database. In this embodiment, the table information in the second improved data blood graph can be taken as the point of the graph database, and the data connection between the tables in the second improved data blood graph can be taken as the edge of the graph database, and saved into the graph database to supplement the ETL information for the blood knowledge graph. At the same time, the fourth source entity and the fourth destination entity (including the related information of the entity) can also be vectorized through the vector model to obtain a fourth vectorized entity, and then the fourth vectorized entity is saved into the vector database.

[0340] Further, the vector database and the graph database can be merged to obtain the final blood knowledge graph. Specifically, the data blood acquisition interface in the OpenMetadata platform can be called first to acquire the data blood information in the OpenMetadata platform, and then the table information in the blood information is taken as the point of the graph database, and the data connection between the tables is taken as the edge of the graph database, and stored into the graph database. At the same time, the table information is vectorized and stored into the vector database, so as to improve the overall data blood knowledge graph. In addition, when the information obtained from the OpenMetaData platform conflicts with the information in the second improved data blood graph, the OpenMetaData platform is used as the standard.

[0341] By parsing SQL files and combining the table information in the SQL files with the data asset relationships in the metadata management platform (i.e., OpenMetadata), a data lineage relationship (i.e., lineage graph) based on SQL is generated, and the lineage relationship is pushed to the metadata management platform (i.e., OpenMetadata). This can solve the problem of needing to adapt a large number of SQL syntaxes when parsing data lineage according to rules.

[0342] Step S15: Input the target vectorized entity and the target associated entity into the pre-trained language model to perform data lineage analysis on the data lineage analysis query statement based on the target vectorized entity and the target associated entity, and obtain the data lineage analysis results of the target e-commerce data warehouse.

[0343] In this embodiment, the two entities obtained after querying two databases (i.e., the target vectorized entity and the target associated entity) can be input into a pre-trained language model (i.e., LLM). The model (i.e., LLM) is guided by a preset prompt word 5 to perform real-time data lineage analysis on the data lineage analysis question currently input by the user based on the target vectorized entity and the target associated entity, so as to obtain the data lineage analysis results for the target e-commerce data warehouse.

[0344] In one specific implementation, prompt word 5 can be:

[0345] # Character Setting

[0346] You are a data governance expert with access to full-chain lineage mapping and data quality reports. Please respond to user issues using the following process:

[0347] ## Enter information

[0348] 1. **User Question**: <Question text entered by the user>

[0349] 2. **Philanthropology** (from Vector Search):

[0350] json

[0351] {

[0352] "entities": [

[0353] {"type": "Table", "name": "sales.orders", "desc": "Order Facts Table","fields": ["order_id","amount"]},

[0354] {"type": "Column", "name": "total_sales", "belongs_to": "dw.daily_sales"}

[0355] ],

[0356] "relations": [

[0357] {"source": "sales.orders.amount", "target": "dw.daily_sales.total_sales", "transform": "SUM(amount)"} ]

[0359] }```

[0360] 3. Data Quality Report

[0361] json

[0362] {

[0363] "sales.orders": [

[0364] {"metric": "Null value rate", "field": "amount", "value": "38%", "status": "Critical"},

[0365] {"metric": "data latency", "value": ">2h", "status": "Warning"} ]

[0367] }```

[0368] ## Parsing Rules:

[0369] Step 1: Problem Analysis

[0370] Extract the key entities (tables / fields) and time range from the problem.

[0371] Example:

[0372] "Today's total sales have dropped abnormally" → {"entities": ["dw.daily_sales.total_sales"], "time": "today"}

[0373] Step 2: Geographical Origin Tracing

[0374] Tracing back along the bloodline:

[0375] Upstream tables and fields of the target field

[0376] Label key transformation nodes (ETL / SQL)

[0377] Output path:

[0378] dw.daily_sales.total_sales ← sales.orders.amount (SUM transformation)

[0379] Step 3: Quality report correlation

[0380] Check quality metrics of entities in the path:

[0381] | Entity | Quality metric | Value | Status |

[0382] |--------------------|---------|--------|----------|

[0383] | sales.orders.amount | Null rate | 38% | Critical |

[0384] | sales.orders | Data latency | >2h | Warning |

[0385] Step 4: Root cause reasoning (focus)

[0386] Analyze according to this framework:

[0387] 1. **Direct correlation**: Does the quality issue affect the target field calculation?

[0388] - Null rate 38% → SUM(amount) underestimated

[0389] 2. **Timeliness**: Does the latency cause data loss?

[0390] - Latency >2h → Today's data not updated in time

[0391] 3. **Compound factors**: Multiple problems superimposed effects

[0392] - Null + latency → Double data loss

[0393] Step 5: Action recommendations

[0394] Provide actionable repair suggestions:

[0395] - Check ETL task immediately: `order_import_job` (recently failed)

[0396] - Verify data source: MySQL `sales_db.orders` connection status

[0397] - Null handling: Add default value rule `COALESCE(amount,0)`

[0398] Final output format

[0399] {

[0400] "question_analysis": {"key_entities": [], "time_range": ""},

[0401] "lineage_path": "Entity A → Entity B →... → Target Entity",

[0402] "quality_issues": [

[0403] {"entity": "name", "metric": "type", "value": "", "impact": "Impact description"}

[0404] ],

[0405] "root_cause": "No more than 3-point root cause analysis",

[0406] "solutions": ["Executable action recommendations"]

[0407] }

[0408] ## Special handling

[0409] ! Immediately terminate if the following conditions are met:

[0410] Retrieve entity unrelated to the question → Return: "No relevant data assets found"

[0411] Quality report has no abnormalities → Prompt: "Suggest checking business logic"

[0412] Detect data security fields → Fuzzing processing: "Involved sensitive fields [Shielded]".

[0413] It should be noted that based on the constructed blood relationship knowledge graph, a blood relationship graph analysis question and answer assistant based on a pre-trained language model (i.e., LLM) can be created, so that the development and operation personnel can ask questions to the assistant to inquire about data flow direction, analyze data quality problems, analyze the impact of table field changes, and analyze data task link changes, thereby solving the problems of data quality analysis, metadata change analysis, and data link change analysis in the e-commerce data warehouse scenario.

[0414] Among them, the data quality analysis refers to the data quality problem that the total sales of the sales statistics table is negative. Specifically, the operation personnel inquires the reason through the data analysis question and answer assistant in the blood relationship graph analysis question and answer assistant, and the question and answer assistant finds the related tables and related tasks of the blood relationship link through the blood relationship knowledge graph, combines the data quality analysis report of the OpenMetaData platform, provides the above information to the LLM model, and outputs the problem analysis report, so as to assist in manually positioning the table and related information lacking the field.

[0415] The metadata change analysis refers to the need of the developer to change the sales field name of the sales statistics table. Specifically, the operation personnel can ask questions to the data analysis question and answer assistant in the blood relationship graph analysis question and answer assistant to analyze the influence range, and the question and answer assistant finds the related tables and related tasks of the blood relationship link through the blood relationship knowledge graph, and then provides the above information to the LLM model to output the problem analysis report, so as to confirm the influence range of the change of the field on the table and the task.

[0416] The data link change analysis refers to the need of the development task to change the processing logic of an ETL task. Specifically, the operation personnel can ask questions to the data analysis question and answer assistant in the blood relationship graph analysis question and answer assistant to analyze the influence range, and the question and answer assistant finds the related tables and related tasks of the blood relationship link through the blood relationship knowledge graph, and then provides the above information to the LLM model to output the problem analysis report, so as to confirm the influence range of the change of the field on the table and the task.

[0417] It can be seen that the embodiment of the application performs entity analysis on the question sentence of the data blood relationship analysis input by the user through the pre-created pre-training language model, performs vectorization processing on the obtained entity information, then matches the vectorized entity after vectorization with the vectorized entity in the pre-created vector database, queries the entity associated with the matched target vectorized entity from the graph database, and finally inputs the matched entities from the two databases into the pre-training language model to perform data blood relationship analysis. It should be pointed out that the two databases are located in the blood relationship knowledge graph created based on the task configuration file and the structured query language file of the e-commerce data warehouse and the metadata management platform. Through the above method based on the pre-training language model and combined with the blood relationship knowledge graph created based on the metadata management platform, automatic data blood relationship analysis is realized, which not only improves the efficiency and accuracy of data blood relationship analysis of the e-commerce data warehouse, but also saves the labor cost and time cost.

[0418] The embodiment of the application discloses a specific data blood relationship analysis method of an e-commerce data warehouse, as shown in Figure 4 The method comprises the following steps:

[0419] Step S21: Obtain a data blood relationship analysis question sentence for a target e-commerce data warehouse, and perform entity analysis on the data blood relationship analysis question sentence by using a pre-training language model to obtain current entity information.

[0420] In the embodiment, the data blood relationship analysis question sentence for the e-commerce data warehouse scenario which needs to perform data quality analysis, metadata change analysis, and data link change analysis can be obtained first, and the question sentence is analyzed. It is considered that the question raised by the user may not be able to completely specify the table name, and it is very likely that the question is why today's total sales data is so small. Therefore, the keywords in the question can be extracted by the LLM large model.

[0421] Step S22: Perform vectorization processing on the current entity information to obtain a current vectorized entity.

[0422] Step S23: Compare the current vectorized entity with the vectorized entity in the vector database to obtain an entity associated with the current vectorized entity, and obtain a target vectorized entity.

[0423] In the embodiment, as shown in Figure 5 The extracted keywords can be vectorized to obtain a current vectorized entity, and the correlation comparison is performed in the vector database to obtain the table entity and the field entity with greater correlation (such as the matched TopN entities) as the answer corpus.

[0424] Step S24: querying the entities associated with the target vectorized entity from the graph database to obtain target associated entities; the graph database and the vector database are located in a pre-created blood relationship knowledge graph, the blood relationship knowledge graph is a knowledge graph constructed based on a task configuration file and a structured query language file of the target e-commerce data warehouse and a metadata management platform.

[0425] In this embodiment, based on the table entities and field entities obtained in step S23, the entities associated with the table entities and field entities in the graph database can be queried as the answer corpus.

[0426] Step S25: determining the data asset identifier of the target vectorized entity and the target associated entity in the metadata management platform.

[0427] In this embodiment, after the target vectorized entity and the target associated entity are queried, the data asset identifier (i.e., data asset ID) of the target vectorized entity and the target associated entity in the metadata management platform (i.e., OpenMetaData) can be further determined.

[0428] Step S26: calling a data quality query interface of the metadata management platform based on the data asset identifier to query the running result of the data quality task of the corresponding asset based on the data asset identifier.

[0429] In this embodiment, the data quality query interface of the metadata management platform (i.e., OpenMetaData) can be called based on the data asset ID, so as to query the running result of the data quality task of the corresponding asset based on the data asset ID as the answer corpus.

[0430] Step S27: inputting the target vectorized entity, the target associated entity and the running result into the pre-trained language model to perform data blood relationship analysis on the data blood relationship analysis question based on the target vectorized entity, the target associated entity and the running result, and obtaining the data blood relationship analysis result of the target e-commerce data warehouse.

[0431] In this embodiment, the corpus in steps S23, S24 and S26 and the preset prompt word can be used to guide the LLM model to answer the question raised by the user, so as to help the user to perform data blood relationship analysis on the corresponding question.

[0432] The more specific processing process of step S22 can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0433] As can be seen, in the embodiments of the present application, when the table data in the data warehouse scenario is missing or the table has a problem, the LLM model analyzes where the problem is by using the pre-created blood relationship knowledge graph containing a graph database and a vector database (vector storage, embedding graph attributes into a vector database, supporting semantic search, and providing a basis for blood relationship graph-based analysis and diagnosis), and combining the running results of the data quality task in OpenMetaData. The problem may be that the SQL execution fails or the ETL task has a problem, that is, through the mode of user questioning → LLM extracting keywords → vector matching TopN entities → graph tracing → combining data quality reports to generate root cause analysis, the automatic positioning of data abnormal root causes (such as field missing, ETL task failure) is realized. In addition, cross-system blood relationship integration is supported, such as cross-source blood relationship analysis of ETL tools such as DataX / Chunjun and databases and data warehouses (such as Hive).

[0434] Correspondingly, the embodiments of the present application also disclose a data blood relationship analysis device for an e-commerce data warehouse, which comprises:

[0435] An acquisition module is configured to acquire a data blood relationship analysis question sentence for a target e-commerce data warehouse;

[0436] An entity analysis module is configured to perform entity analysis on the data blood relationship analysis question sentence by using a pre-trained language model to obtain current entity information;

[0437] A vectorization processing module is configured to perform vectorization processing on the current entity information to obtain current vectorized entities;

[0438] An entity comparison module is configured to compare the current vectorized entities with vectorized entities in a vector database to obtain entities associated with the current vectorized entities, and obtain target vectorized entities;

[0439] A query module is configured to query entities associated with the target vectorized entities from a graph database to obtain target associated entities; the graph database and the vector database are located in a pre-created blood relationship knowledge graph, the blood relationship knowledge graph is a knowledge graph constructed by using a metadata management platform based on a task configuration file and a structured query language file of the target e-commerce data warehouse;

[0440] A blood relationship analysis module is configured to input the target vectorized entities and the target associated entities into a pre-trained language model to perform data blood relationship analysis on the data blood relationship analysis question sentence based on the target vectorized entities and the target associated entities, and obtain a data blood relationship analysis result of the target e-commerce data warehouse.

[0441] The specific working procedures of the above-mentioned modules can be referred to the corresponding contents disclosed in the foregoing embodiments, and will not be described here.

[0442] Embodiments of the present application also provide an electronic device, comprising a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the steps in any of the above-mentioned embodiments of the method for analyzing data lineage of an e-commerce data warehouse.

[0443] Embodiments of the present application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above-mentioned embodiments of the method for analyzing data lineage of an e-commerce data warehouse when executed.

[0444] In an example embodiment, the above-mentioned computer-readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing a computer program.

[0445] Embodiments of the present application also provide a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the method for analyzing data lineage of an e-commerce data warehouse.

[0446] Embodiments of the present application also provide another computer program product comprising a non-volatile computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the method for analyzing data lineage of an e-commerce data warehouse.

[0447] Those skilled in the art will further appreciate that the functions implemented by the examples described herein can be implemented in digital electronic circuitry, in analog electronic circuitry, in computer software, in firmware, or in combinations of them. The disclosure includes a computer program product applicable to a computer system or a computer network. The computer program product can be a storage medium or a computer program product directly implementable in a computer system or a computer network. The storage medium either stores computer program directly or stores the computer program after being transformed by a computer. The storage medium can include, but is not limited to, a floppy disk, a USB flash disk, a RAM, a ROM, a PROM, an EPROM, an EEPROM, a WORM, a semiconductor, a DRAM, also including the registered trademark of the company, an SRAM, a SDRAM, a RDRAM, a DDR DRAM, an eDRAM, a Flash memory, an IRAM, a memory chip, and the like. The computer program product, directly or indirectly, makes the computer system or the computer network perform a predetermined function when the computer program is executed by one or more computer systems or one or more computer networks. The computer program product, directly or indirectly, causes one or more computer systems or one or more computer networks to perform a predetermined function when the computer program is executed by one or more computer systems or one or more computer networks.

[0448] The data blood relationship analysis method, product, device and storage medium of the e-commerce data warehouse provided by the application are described in detail. The principles and implementation modes of the application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the application and its core idea. It should be pointed out that, for ordinary skilled persons in the technical field, some improvements and modifications can be made to the application without departing from the principles of the application, and these improvements and modifications also fall within the protection scope of the application.

Claims

1. A data lineage analysis method of an e-commerce data warehouse, characterized in that, The method comprises the following steps: obtaining a data bloodline analysis question sentence for a target e-commerce data warehouse, performing entity analysis on the data bloodline analysis question sentence by using a pre-trained language model to obtain current entity information; vectorizing the current entity information to obtain current vectorized entities; comparing the current vectorized entities with vectorized entities in a vector database to obtain entities associated with the current vectorized entities, thereby obtaining target vectorized entities; the vector database is a database system for storing, indexing, and querying high-dimensional vector data, and achieves efficient unstructured data semantic search and matching by calculating the similarity between vectors; querying entities associated with the target vectorized entities from a graph database to obtain target associated entities; the graph database and the vector database are located in a pre-created bloodline knowledge graph, which is a knowledge graph constructed based on a task configuration file and a structured query language file of the target e-commerce data warehouse and by using a metadata management platform; the graph database is a non-relational database that uses points and edges as basic storage units to store and query highly interconnected data, wherein the points are table entities and field entities in entity bloodline relationship information, and the edges are table entity relationships and field entity relationships; inputting the target vectorized entities and the target associated entities into the pre-trained language model to perform data bloodline analysis on the data bloodline analysis question sentence based on the target vectorized entities and the target associated entities, thereby obtaining a data bloodline analysis result of the target e-commerce data warehouse; The method further comprises the following steps: obtaining a task configuration file and a corresponding structured query language file for a target e-commerce data warehouse; performing entity analysis on the task configuration file by using a pre-trained language model to obtain first historical entity information; obtaining data asset information pre-built in a metadata management platform, and perfecting the first historical entity information by using the data asset information to obtain first perfected entity information; analyzing the mapping relationship between different entities in the task configuration file by using the pre-trained language model to obtain first entity bloodline relationship information, and saving the first entity bloodline relationship information and the first perfected entity information to a graph database; vectorizing the first perfected entity information to obtain first vectorized entities, and saving the first vectorized entities to a vector database; performing entity analysis on the structured query language file by using the pre-trained language model to obtain second historical entity information; perfecting the second historical entity information by using the data asset information to obtain second perfected entity information; analyzing the mapping relationship between different entities in the second perfected entity information by using the pre-trained language model to obtain second entity bloodline relationship information; saving the second entity bloodline relationship information and the second perfected entity information to the graph database; Vectorize the second perfected entity information to obtain a second vectorized entity, and save the second vectorized entity to the vector database to obtain a blood relationship knowledge graph comprising the graph database and the vector database.

2. The method for data lineage analysis of an e-commerce data warehouse according to claim 1, wherein, The entity parsing on the task configuration file by using the pre-trained language model comprises: The entity parsing on the task configuration file by using the pre-trained language model comprises: The first source end entity and the first destination end entity each comprise a table name, table field information, and service information. 3.The method of claim 2, wherein, The data asset information pre-constructed in the metadata management platform is acquired, and the first historical entity information is perfected by using the data asset information to obtain first perfected entity information, which comprises: A metadata query interface of the metadata management platform is called to acquire data asset information corresponding to the first source end entity and the first destination end entity from the metadata management platform; The first source end entity and the first destination end entity are perfected based on the data asset information to obtain a second source end entity and a second destination end entity; The second source end entity and the second destination end entity each comprise a table name, table field information, service information, a table metadata identifier, a column metadata identifier, table description, and column description.

4. The method for data lineage analysis of an e-commerce data warehouse according to claim 3, wherein, The mapping relationship between different entities in the task configuration file is parsed by using the pre-trained language model to obtain first entity blood relationship information, which comprises: The field mapping relationship between the second source end entity and the second destination end entity is parsed by using the pre-trained language model to obtain first entity blood relationship information. The first entity blood relationship information comprises table blood relationship information and field blood relationship information.

5. The method for data lineage analysis of an e-commerce data warehouse according to claim 4, wherein, The first entity blood relationship information and the first perfected entity information are saved to a graph database, which comprises: The initial data blood relationship graph in the metadata management platform is perfected based on the first entity blood relationship information to obtain a first perfected data blood relationship graph; The second source end entity and the second destination end entity are taken as points of a graph database, and the table blood relationship information and the field blood relationship information in the first entity blood relationship information are taken as edges of the graph database, and are saved to the graph database.

6. The method for data lineage analysis of an e-commerce data warehouse according to claim 5, wherein, The initial data blood relationship graph in the metadata management platform is perfected based on the first entity blood relationship information to obtain a first perfected data blood relationship graph, which comprises: The first entity blood relationship information is sent to a blood relationship construction interface of the metadata management platform to construct data blood relationship connections between corresponding data assets in the data asset information based on the identifier information in the second source end entity and the second destination end entity and the first entity blood relationship information, so as to perfect the initial data blood relationship graph in the metadata management platform to obtain a first perfected data blood relationship graph.

7. The method for data lineage analysis of an e-commerce data warehouse according to claim 5, wherein, After the first vectorized entity is saved to the vector database, the method further comprises: Call the data bloodline acquisition interface of the metadata management platform to obtain the first perfected data bloodline graph; Save the table information in the first perfected data bloodline graph as a point of the graph database, and save the data connection between the tables in the first perfected data bloodline graph as an edge of the graph database into the graph database. The second source end entity and the second destination end entity are vectorized by a vector model to obtain a third vectorized entity, and the third vectorized entity is saved into the vector database.

8. The method for data lineage analysis of an electrical business data warehouse according to claim 7, wherein, The entity parsing of the structured query language file by the pre-trained language model includes: The entity parsing of the structured query language file by the pre-trained language model includes: Correspondingly, the second perfected entity information is obtained by perfecting the second historical entity information by using the data asset information, and includes: Call the metadata query interface to obtain data asset information corresponding to the third source end entity and the third destination end entity from the metadata management platform; The fourth source end entity and the fourth destination end entity are obtained by perfecting the third source end entity and the third destination end entity based on the data asset information. The fourth source end entity and the fourth destination end entity include table name, table field information, service information, table metadata identifier, column metadata identifier, table description and column description.

9. The method of claim 8, wherein, The second entity bloodline relationship information is obtained by analyzing the mapping relationship between different entities in the second perfected entity information by using the pre-trained language model, and includes: The second entity bloodline relationship information is obtained by analyzing the field mapping relationship between the fourth source end entity and the fourth destination end entity by using the pre-trained language model; the second entity bloodline relationship information includes table bloodline relationship information and field bloodline relationship information; Correspondingly, the second entity bloodline relationship information and the second perfected entity information are saved into the graph database, including: The second perfected data bloodline graph is obtained by perfecting the first perfected data bloodline graph in the metadata management platform based on the second entity bloodline relationship information; The fourth source end entity and the fourth destination end entity are saved into the graph database as points of the graph database, and the table bloodline relationship information and the field bloodline relationship information in the second entity bloodline relationship information are saved into the graph database as edges of the graph database.

10. The method of claim 9, wherein, After saving the second vectorized entity into the vector database, it further includes: The table information in the second perfected data bloodline graph is saved into the graph database as a point of the graph database, and the data connection between the tables in the second perfected data bloodline graph is saved into the graph database as an edge of the graph database. vectorize the fourth source end entity and the fourth destination end entity through a vector model to obtain a fourth vectorized entity, and save the fourth vectorized entity into the vector database.

11. The method of data lineage analysis of an e-commerce data warehouse according to any one of claims 1 to 10, wherein, The target vectorized entity and the target associated entity are input into the pre-trained language model to perform data lineage analysis on the data lineage analysis question sentence based on the target vectorized entity and the target associated entity, and obtain a data lineage analysis result of the target e-commerce data warehouse, including: determining a data asset identifier of the target vectorized entity and the target associated entity in the metadata management platform; calling a data quality query interface of the metadata management platform based on the data asset identifier to query a running result of a data quality task of a corresponding asset based on the data asset identifier; inputting the target vectorized entity, the target associated entity and the running result into the pre-trained language model to perform data lineage analysis on the data lineage analysis question sentence based on the target vectorized entity, the target associated entity and the running result, and obtain a data lineage analysis result of the target e-commerce data warehouse.

12. An electronic device, comprising: including: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the data lineage analysis method of the e-commerce data warehouse according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and when executed by the processor, the steps of the data lineage analysis method of the e-commerce data warehouse according to any one of claims 1 to 11 are implemented.

14. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the data lineage analysis method of the e-commerce data warehouse according to any one of claims 1 to 11.

Images