Data question answering method and device and medium

Abstract

The invention provides a data question answering method and device and a medium, and relates to the technical field of computers. The method comprises the steps of obtaining a data problem and determining that the data problem points to a first data table generated by continuous operation; positioning a first node corresponding to the first data table in the data blood relationship knowledge graph of the continuous operation; obtaining a preorder data consanguinity knowledge graph comprising the first node and the preorder data consanguinity of the first node; analyzing the preorder data blood relationship knowledge graph by using a large language model (LLM) to obtain an answer to the data question; wherein the data blood relationship knowledge graph records the dependency relationship of each data table in the continuous operation process and the operation on each data table in the continuous operation process. According to the method, the large model is used for analyzing the data blood relationship knowledge graph related to the data question, the data question is answered, automatic question answering of the data question is achieved, the data operation efficiency is improved, and the data operation cost is reduced.

Description

Technical Field

[0001] This application relates at least to the field of computer technology, and particularly to a data question answering method, device, and medium. Background Art

[0002] During the process of data operation and use, business personnel may have questions about data, including troubleshooting data errors and understanding the meaning of data. Currently, the general method for handling such data questions is to rely on manual processing by data back-end operation and maintenance personnel, which makes the back-end operation and maintenance personnel deal with and solve problems, consuming a large amount of workload and affecting the efficiency and cost of data operation. Summary of the Invention

[0003] In view of the above deficiencies, this application provides a data question answering method, device, and medium to solve the following technical problem: how to automatically implement data question answering.

[0004] In a first aspect, this application provides a data question answering method, which includes:

[0005] Obtain a data question and determine a first data table generated by a continuous job pointed to by the data question;

[0006] Locate a first node corresponding to the first data table in the data lineage knowledge graph of the continuous job;

[0007] Obtain a pre-order data lineage knowledge graph including the first node and its pre-order data lineage;

[0008] Use a large language model (LLM) to analyze the pre-order data lineage knowledge graph to obtain an answer to the data question;

[0009] Among them, the data lineage knowledge graph records the dependency relationships between each data table during the continuous job process and the operations performed on each data table during the continuous job process.

[0010] Furthermore, the data lineage knowledge graph specifically includes at least one of the following:

[0011] A first data lineage knowledge graph, with the external source data tables of the continuous job and the target data tables generated in each job link as nodes, the dependency relationships between the data tables during the continuous job process as edges, and each edge marked with the operator operator that operates on the job source data table connected by the edge as the job target data table during the job;

[0012] The second data lineage knowledge graph takes the external source data tables of continuous operations and the target data tables generated in each operation link as nodes, the dependency relationships between the data tables in the continuous operation process as edges, and marks the operation descriptions of the operations completed by the source data tables of the operations connected to the node through all its direct preceding edges on the nodes that are the target data tables of the operations;

[0013] The third data lineage knowledge graph divides communities on the basis of the second data lineage knowledge graph and obtains the community summaries of each community according to the operation descriptions of all the nodes in the same community.

[0014] Further, before obtaining the data problem and determining that the data problem points to the first data table generated by the continuous operation, the method further includes:

[0015] Use the open-source stream processing framework Flink to perform continuous operations on the data, obtain the source data tables and target data tables of each operation link of the continuous operation, and the operator operators or operation descriptions used by each operation link to operate the source data tables as target data tables;

[0016] Taking the source data tables and target data tables of the operations as nodes and the dependency relationships of the operations between the source data tables and target data tables of the operations as edges, mark the operator operators used by each edge or mark the operation descriptions of each node of the target data table of the operation to establish the first or second data lineage knowledge graph.

[0017] Further, using the open-source stream processing framework Flink to perform continuous operations on the data, obtaining the source data tables and target data tables of each operation link of the continuous operation, and the operator operators or operation descriptions used by each operation link to operate the source data tables as target data tables, specifically including:

[0018] Use the structured query language SQL of Flink to generate the external source data tables of the continuous operation and generate the target data tables of each operation link according to the external source data tables and / or the target data tables of the previous operation link, and record the SQL tables of the continuous operation and / or use the data flow graph StreamGraph of Flink to record the process topology graph of the continuous operation;

[0019] Obtain the operator operators used by each operation link to operate the source data tables as target data tables from the SQL tables and / or the process topology graph, or use the LLM to analyze the source data tables, target data tables, and operator operators in the SQL tables and / or the process topology graph to obtain the operation descriptions of each operation link.

[0020] Further, in response to the establishment of the second data lineage knowledge graph, the method further includes:

[0021] Determine the maximum number of nodes that can be included in a community according to the upper limit of tokens, which is the minimum unit for processing text by the LLM;

[0022] Divide several nodes in the second data lineage knowledge graph into the same community according to the operation description and the maximum number of nodes;

[0023] Use the LLM to summarize the operation descriptions of several nodes in the same community to generate a community summary and add it to the second data lineage knowledge graph to form the third data lineage knowledge graph.

[0024] Furthermore, obtain data problems and determine the first data table generated by continuous operations pointed to by the data problems, specifically including:

[0025] Obtain data problems where business personnel inquire about the meaning of data or verify data errors;

[0026] Use the LLM to identify the data table names that business personnel inquire about or verify in the data problems;

[0027] Match the data table names inquired about or verified with the names of the data tables generated by continuous operations to determine the first data table pointed to by the data problems.

[0028] Furthermore, use the large language model LLM to analyze the previous data lineage knowledge graph to obtain answers to data problems, specifically including:

[0029] In response to using the LLM to identify that the data problem is that business personnel inquire about the meaning of a data table and the previous data lineage knowledge graph includes several communities, use the LLM to summarize the community summaries of several communities to obtain the meaning of the inquired data table as the answer to the data problem;

[0030] In response to using the LLM to identify that the data problem is that business personnel inquire about the meaning of data items in a data table, sequentially match the inquired data items with the headers of the data tables recorded in each node in the previous data lineage knowledge graph to determine the second node where the inquired data item first appears, and use the LLM to analyze the source data tables and operator descriptors or operation descriptions of all directly connected previous edges of the second node to obtain the meaning of the inquired data item as the answer to the data problem;

[0031] In response to using the LLM to identify that the data problem is that business personnel verify incorrect data items in a data table, sequentially match the verified data items with the headers of the data tables recorded in each node in the previous data lineage knowledge graph to determine the third node where the verified data item is first incorrect, and use the LLM to analyze the third node to obtain the reason for the error of the verified data item as the answer to the data problem.

[0032] Further, use the LLM to analyze the third node to obtain the reasons for the errors in the verified data items, specifically including:

[0033] In response to the third node being an external source data table node, use the LLM to generate that the reason for the error in the verified data item is an external source data error;

[0034] In response to the third node not being an external source data table node, use the LLM to analyze the operator descriptors of all directly connected previous edges of the third node or the operation description of the third node to obtain that the reason for the error in the verified data item is a certain data operation error of the third node.

[0035] In a second aspect, the present application provides a data question answering device, and the device includes:

[0036] A question module, configured to obtain a data question and determine a first data table generated by a continuous operation pointed to by the data question;

[0037] A positioning module, connected to the question module, configured to locate a first node corresponding to the first data table in the data lineage knowledge graph of the continuous operation;

[0038] A previous module, connected to the positioning module, configured to obtain a previous data lineage knowledge graph including the first node and its previous data lineage;

[0039] An answering module, connected to the previous module, configured to use the large language model LLM to analyze the previous data lineage knowledge graph to obtain an answer to the data question;

[0040] Wherein, the data lineage knowledge graph records the dependency relationships between data tables during the continuous operation and the operations performed on each data table during the continuous operation process.

[0041] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is run by a processor, the data question answering method as described above is implemented.

[0042] The present application provides a data question answering method, device and medium, which combines the knowledge graph and large model technology, and realizes automatic answering of data questions by using the large model to analyze the data lineage knowledge graph related to the data question and answer the data question, which can be used to help business personnel understand the data meaning or troubleshoot data errors, etc., thereby improving data operation efficiency and reducing data operation costs. Description of the Drawings

[0043] Figure 1 is a flowchart of a data question answering method according to an embodiment of the present application;

[0044] Figure 2 It is a schematic structural diagram of a data question answering device according to an embodiment of the present application;

[0045] Figure 3 It is an overall architecture diagram of a data question answering method according to an embodiment of the present application;

[0046] Figure 4 It is an example diagram of continuous data operation according to an embodiment of the present application;

[0047] Figure 5 It is an example diagram of data lineage according to an embodiment of the present application;

[0048] Figure 6 It is an example diagram of the generation of a data lineage knowledge graph according to an embodiment of the present application;

[0049] Figure 7 It is an example diagram of the generation of a third data lineage knowledge graph according to an embodiment of the present application;

[0050] Figure 8 It is an example diagram of data error verification according to an embodiment of the present application. Detailed implementation manners

[0051] To enable those skilled in the art to better understand the technical solutions of the present application, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0052] It can be understood that the specific embodiments and the accompanying drawings described herein are only used to explain the present application, rather than limiting the present application.

[0053] It can be understood that, without conflict, the various embodiments and the various features in the embodiments of the present application can be combined with each other.

[0054] It can be understood that for the convenience of description, only the parts related to the present application are shown in the accompanying drawings of the present application, and the parts unrelated to the present application are not shown in the accompanying drawings.

[0055] It can be understood that each module and unit involved in the embodiments of the present application may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple modules and units may also be integrated into one entity structure.

[0056] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present application may occur in an order different from that marked in the accompanying drawings.

[0057] It can be understood that in the flowcharts and block diagrams of the present application, the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present application are shown. Among them, each block in the flowchart or block diagram may represent a module, unit, program segment, or code, which contains executable instructions for implementing the specified function. Moreover, each block or combination of blocks in the block diagram and flowchart can be implemented by a hardware-based device for implementing the specified function, or by a combination of hardware and computer instructions.

[0058] It can be understood that the modules and units involved in the embodiments of the present application can be implemented in software or in hardware. For example, the modules and units can be located in the processor.

[0059] Embodiment 1:

[0060] As Figure 1 shown, the present application provides a method for data Q&A. The method includes:

[0061] S1. Obtain a data problem and determine the first data table generated by continuous operations pointed to by the data problem;

[0062] S2. Locate the first node corresponding to the first data table in the data lineage knowledge graph of continuous operations;

[0063] S3. Obtain the pre-order data lineage knowledge graph including the first node and its pre-order data lineage;

[0064] S4. Use the large language model LLM to analyze the pre-order data lineage knowledge graph to obtain the answer to the data problem;

[0065] Among them, the data lineage knowledge graph records the dependency relationships between data tables during continuous operations and the operations performed on each data table during continuous operations.

[0066] In this embodiment, the method combines the knowledge graph and large model technology. By using the large model to analyze the data lineage knowledge graph related to the data problem and answer the data problem, it realizes automatic Q&A for data problems, which can be used to help business personnel understand the meaning of data or troubleshoot data errors, etc., thereby improving data operation efficiency and reducing data operation costs. As Figure 1 shown, the method corresponds to being applied to the device as Figure 2 shown.

[0067] Specifically, this embodiment provides a real-time data operation method based on a large model, which is applied to the field of intelligent data operation. Intelligent operation refers to an operation model that uses artificial intelligence and big data technologies to optimize operation strategies, improve operation efficiency, and thus achieve business growth. The core of intelligent operation lies in enhancing operation effects and user experiences through technical means. Currently, front-line marketing and data analysis personnel may not be clear about the business implications behind the data when using data sets, resulting in an inability to quickly locate the data sets they need, and thus turning to submitting data extraction requests or making manual inquiries. This requires back-end operation and maintenance personnel to handle and solve problems, consuming a large amount of workload. The method provided in this example intends to use large model technology, utilize a knowledge graph, and through the combination of real-time calculation and large models, conduct investigations and provide answers to some data problems, especially to achieve investigations and answers to the processed data of real-time tables, which can help front-line marketing and data analysis personnel understand the business implications behind the data without the intervention of back-end operation and maintenance personnel, thereby making data operation more efficient, improving operation efficiency, and reducing operation costs.

[0068] More specifically, the overall architecture of this method is as Figure 3 shown. Business personnel (such as front-line marketing and data analysis personnel, etc.) pose data problems to the LLM (Large Language Model) large model. The LLM large model is connected to the FLINK (an open-source stream processing framework, also written as Flink) real-time data warehouse and the SQL (Structured Query Language) template vector database. The SQL template vector database records the SQL statements used by the Flink continuous job to operate data. The continuous job process is as Figure 4 shown. Each job link (Job 1, Job 2) has its corresponding upstream table and downstream table. Complex data dependency relationships, that is, data lineage relationships, are formed among these data tables. Figure 5 The LLM large model establishes a data lineage relationship knowledge graph based on the real-time data warehouse and SQL statements. The data lineage relationship is a directed acyclic graph as Figure 4 shown. Combining Figure 4 , the upstream table (job source data table) and downstream table (job target data table) of each job link are data nodes. The number of upstream tables whose data affects a downstream table determines the number of node connection edges formed. Each edge is added with the corresponding Flink operator to represent the data operation therein, and thus a data lineage relationship knowledge graph can be formed. The data lineage relationship knowledge graph can be established and updated in real time dynamically following the Flink continuous job process, and can also be as Figure 6As shown, the operation description (DES) of each operation link is formed using the large language model (LLM) to speed up the answering of questions. In the face of data questions raised by business personnel, in the complete data lineage knowledge graph, the lineage of the problem data targeted by the data question is obtained, including the problem data itself, its previous data, and the lineage, such as Figure 4 In Figure 4 , the downstream table of Job 2 is the data table that generates doubts for users. The Jobs 2, 1, intermediate nodes, and data sources before this table are obtained for analysis. That is, according to the direction indicated by its edges, the upstream tables and upstream operations are searched in reverse, so as to obtain the answer to the data question as the output result. The output result may include explanations of processing specifications and verification of data missing, etc.

[0069] In one implementation, the data lineage knowledge graph specifically includes at least one of the following:

[0070] The first data lineage knowledge graph uses the external source data tables of continuous operations and the target data tables generated in each operation link as nodes, the dependency relationships between data tables in the continuous operation process as edges, and marks the operator operators for the operation of the job source data table connected by the edge as the job target data table on each edge;

[0071] The second data lineage knowledge graph uses the external source data tables of continuous operations and the target data tables generated in each operation link as nodes, the dependency relationships between data tables in the continuous operation process as edges, and marks the operation descriptions for the operation of the job source data tables connected by all its direct previous edges on the nodes that are job target data tables;

[0072] The third data lineage knowledge graph divides communities on the basis of the second data lineage knowledge graph, and obtains the community summary of each community according to the operation descriptions of all nodes in the same community.

[0073] In this embodiment, by introducing the lineage between data tables, the large language model can continuously trace the content in the table until the processing error or data error at the source is found. For this reason, on the basis of the data lineage as shown in Figure 5 The operator operators can be directly marked on its edges. When the large language model (LLM) needs to answer data questions, the LLM directly analyzes the data and operator operators to obtain the answer to the question. Or as shown in Figure 6 First, use the large language model (LLM) to generate the description (DES) of each operation link. It can also be as shown in Figure 7 Use the large language model (LLM) to divide communities and summarize to obtain the community summary. The formed data lineage knowledge graph can further improve the speed of subsequent answering of questions.

[0074] In one embodiment, before S1, obtaining a data problem and determining that the data problem points to the first data table generated by continuous operations, the method further includes:

[0075] Using the open-source stream processing framework Flink to perform continuous operations on the data, obtaining the source data table and the target data table of each operation link of the continuous operation, as well as the operator operator or operation description used by each operation link to operate the source data table into the target data table;

[0076] Taking the source data table and the target data table as nodes, and the dependency relationship between the source data table and the target data table as edges, marking each edge with the operator operator used or marking the operation description of each target data table node to establish the first or second data lineage knowledge graph.

[0077] In this embodiment, not only can the processing function of offline data be considered, but also the processing of real-time data can be considered. Processing refers to performing continuous operations on the data as Figure 4 shown, so as to provide the data required by front-line marketing and data analysis personnel. By introducing the relevant functions of FLINK real-time computing, the large model can analyze the relevant data of the real-time table, and the real-time table is the data obtained by FLINK real-time computing. Use the function of analyzing the SQL syntax tree to analyze according to the processing dependency relationship between the tables to obtain the data lineage knowledge graph.

[0078] In one embodiment, using the open-source stream processing framework Flink to perform continuous operations on the data, obtaining the source data table and the target data table of each operation link of the continuous operation, as well as the operator operator or operation description used by each operation link to operate the source data table into the target data table, specifically includes:

[0079] Using the structured query language SQL of Flink to generate the external source data table of the continuous operation and generate the target data table of each operation link according to the external source data table and / or the target data table of the previous operation link, and record the SQL table of the continuous operation and / or use the data flow graph StreamGraph of Flink to record the process topology graph of the continuous operation;

[0080] Obtaining the operator operator used by each operation link to operate the source data table into the target data table from the SQL table and / or the process topology graph, or using the LLM to analyze the source data table, the target data table, and the operator operator in the SQL table and / or the process topology graph to obtain the operation description of each operation link.

[0081] In this embodiment, FLINK uses a flow table model to describe the real-time processed data, and represents its processing process through a data stream. Logically, this data stream is embodied as an SQL table, but at the bottom layer, it is still a data stream. It can only call APIs (Application Programming Interfaces) similar to SQL tables and use operator operators to perform various operations on the data stream, providing rich operators such as Map (mapping), Filter (filtering), Reduce (aggregation), Join (joining), etc. These operators can be combined to form complex data processing flows, and these operators will form a lineage relationship within a data processing application. Similar to the requirements of offline processing tasks, in the data processing tasks of a real-time data warehouse, for many reports, data dependencies between multiple processes are often required for processing. Subsequently, the data in the downstream flow table Kafka (an open-source stream processing platform) may be updated as the upstream flow table is updated. If there are problems with the data in the downstream flow table, it may be due to deficiencies in the processing logic itself, or it may be due to problems with the data updated by the upstream data source. For example, in Figure 4 In the lineage relationship shown, it can be seen that the upstream table of Job 1 writes data into the upstream table of Job 2 after processing, and the upstream table of Job 2 writes data into the downstream table of Job 2 after processing. If there is a problem with the downstream table of Job 2, it may be that the data sent by the upstream table of Job 1 is inherently problematic, or it may be that the processing logic of Job 2 has problems; in Figure 5 In the lineage relationship shown, it is a similar relationship. F_PROCESS represents the job link, DF_COUNT represents the number of data table items, and the downstream data table items may come from the upstream data table or may be obtained by processing the items in the upstream data table. For each processing job internally, FLINK also provides a model of StreamGraph (data stream graph, encapsulating the process topology graph of Flink task execution) to describe the specific situation within the processing task. Then, if StreamGraph is used to express the knowledge graph and the large model synthesizes the content of the knowledge graph for expression, the large model can analyze the structure in the FLINK statement. The structure in the FLINK statement refers to the data processing flow formed by the combination of operators.

[0082] In one embodiment, in response to establishing the second data lineage relationship knowledge graph, the method further includes:

[0083] Determine the maximum number of nodes that can be included in a community according to the token upper limit of the minimum processing text unit of the LLM;

[0084] Divide several nodes in the second data lineage relationship knowledge graph into the same community according to the operation description and the maximum number of nodes;

[0085] Use an LLM to summarize the operation descriptions of several nodes in the same community to generate a community summary and add it to the second data lineage knowledge graph to form a third data lineage knowledge graph.

[0086] In this embodiment, as Figure 7 shown, by combining FLINK and the knowledge graph, the knowledge graph can be preprocessed by an LLM, and more natural language-like descriptive information can be added to the knowledge graph to improve the speed and accuracy of subsequent question answering. This processing process may include:

[0087] Element instance to element summary: Extract entities and relationships, and use an LLM to summarize the descriptive text blocks of each element. Entities / elements are the points in the data lineage within the processing flow, that is, the real-time data tables generated by FLINK. Relationships are the edges of the data lineage within the processing flow, and the descriptive text blocks are the summaries of the processing flow generated by the large model according to the knowledge graph algorithm;

[0088] Element summary to graph community: These entity summaries form a graph, and then algorithms such as Leiden (an optimized community discovery algorithm) are used to divide it into communities to achieve a hierarchical structure. First, summarize each individual node into a community (a small annotated graph), and then splice these small graphs into a large graph. This is because the large model has an input token (the basic building block for text processing, that is, the smallest unit for processing text) limit, and if the entire graph is directly input, it will exceed the limit;

[0089] Graph community to community summary: The summary of each community is generated by an LLM to facilitate the large model's understanding of the global topic structure and semantics of the dataset;

[0090] By integrating the above two parts (element summary and community summary), and combining with the large model's chain of thought technology, a more detailed analysis of data maintenance problems can be carried out.

[0091] In one implementation, S1. Obtain a data problem and determine the first data table generated by a continuous job pointed to by the data problem, specifically including:

[0092] Obtain a data problem where business personnel inquire about the meaning of data or verify data errors;

[0093] Use an LLM to identify the name of the data table that business personnel inquire about or verify in the data problem;

[0094] Match the name of the data table that is inquired about or verified with the name of the data table generated by the continuous job to determine the first data table pointed to by the data problem.

[0095] In this embodiment, in the daily operation of the data middle platform, frontline business personnel often have some questions about data. For example: "Why don't the users with user codes 31002 and 31003 appear in this sales channel change table?", or: "Why can't the names of the developers of the users with user codes 31004 and 31005 be found in this user information derivative table?". In the past, such questions would take up the time of operation and maintenance personnel to check and process in the underlying logic of data processing code, which was time-consuming and laborious. In addition, since the real-time processed data often depends on each other among multiple applications, if there are problems with the data in the subsequent flow table, it may be due to deficiencies in the processing logic itself, or it may be because of problems with the data updated by the pre-posed data source. The back-end personnel even need to troubleshoot among multiple different applications. In this embodiment, the large model analyzes and identifies user problems. The large model connects to the data lineage knowledge graph and answers user questions by analyzing the data lineage knowledge graph. First, it locates the data table queried by the user, such as the "sales channel change table" or the "user information derivative table" in the aforementioned questions, and then locates the data items queried by the user, such as "user code" and "developer name", etc. Business personnel usually generate these questions during the process of using data, and it is easy for the large model to guide users to provide this information through design.

[0096] In one embodiment, S4. Use the large language model LLM to analyze the previous data lineage knowledge graph to obtain the answer to the data problem, specifically including:

[0097] In response to using the LLM to identify that the data problem is that the business personnel inquire about the meaning of the data table, and the previous data lineage knowledge graph includes several communities, use the LLM to summarize the community summaries of the several communities to obtain the meaning of the inquired data table as the answer to the data problem;

[0098] In response to using the LLM to identify that the data problem is that the business personnel inquire about the meaning of the data items in the data table, match the inquired data items with the headers of the data tables recorded by each node in the previous data lineage knowledge graph in turn to determine the second node where the inquired data item first appears, and use the LLM to analyze the job source data tables and operator descriptors or the operation descriptions of the second node of all directly connected previous edges to obtain the meaning of the inquired data item as the answer to the data problem;

[0099] In response to using the LLM to identify that the data problem is that the business personnel check the incorrect data items in the data table, match the checked data items with the headers of the data tables recorded by each node in the previous data lineage knowledge graph in turn to determine the third node where the checked data item is first incorrect, and use the LLM to analyze the third node to obtain the reason for the error of the checked data item as the answer to the data problem.

[0100] In this embodiment, since the large model can not only view the processing code (data processing flow formed by the combination of operator operators) within a FLINK application, but also see the data dependency relationships (i.e., data lineage relationships) between different FLINK applications, at this time, by connecting FLINK and analyzing layer by layer, the large model can independently judge and analyze whether the problem lies in the data source or the processing logic, so as to timely remind the operation and maintenance personnel.

[0101] The pseudo-code of the algorithm is as follows:

[0102]

[0103] As Figure 8 shown, for the applications of a single node, a chain-of-thought troubleshooting method is adopted. The processing caliber information is the graph generated according to the lineage relationship before. The error caliber is for the LLM to guess and verify based on the existing processing caliber and the problems raised by front-line personnel. For example, now there is a user-derived table DERIVE_RT, and there is a missing data of user_id = '21000' in it. Then the LLM can speculate that one is that this information does not exist in the data source USER_INFO originally, and the other is that in the derived table, IS_INNET = '1' is restricted, and all users must be all online to be displayed, so this data is filtered out. The large model connects to the database, generates relevant troubleshooting code through the sql template in the vector database for troubleshooting, and can more accurately judge the location where the data fails, so as to remind the operation and maintenance personnel to perform corresponding processing.

[0104] In an implementation manner, the LLM is used to analyze the third node to obtain the reason for the error of the verified data item, specifically including:

[0105] In response to the third node being an external source data table node, the LLM is used to generate that the reason for the error of the verified data item is an external source data error;

[0106] In response to the third node not being an external source data table node, the LLM is used to analyze the operator descriptors of all directly connected predecessor edges of the third node or the operation description of the third node to obtain that the reason for the error of the verified data item is an error in a certain data operation of the third node.

[0107] In this embodiment, for the vast majority of flow tables, a directed acyclic graph can be used to represent them mathematically. For nodes, they can be divided into two categories: The first type is nodes without predecessors, indicating that this data source is directly connected to an external data source (such as Figure 4 the upstream table of job 1 in Figure 4The intermediate node and the first node in it), indicating that this flow table is only an intermediate result of the flow calculation. If a problem is found at the intermediate node, it may be a problem with the data source or a problem with the processing. If a problem is found at the data source, then there must be a problem with the data distribution. The node where the problem first appears can be found, and by analyzing the nature and operations of this node, the cause of the error can be obtained. For example, if the third node is Figure 4 the first node in it, since the first node does not introduce data from the outside, then the problem data will not be introduced from the outside, and the error can only occur during the operation of Job 2. Figure 4 The intermediate node and the first node in it can also introduce data from the outside and use the data generated in the previous link at the same time. If the error occurs at the external source data table node, such as the missing data has never appeared in all nodes, then the data has not been introduced.

[0108] In this embodiment, by combining the graph model of FLINK and the application of the knowledge graph, a more efficient and perfect description of the data processing caliber is realized; through the dependency relationship between the tables in the FLINK program, the analysis of data loss is realized in the way of a thinking chain, which can at least simplify part of the data verification work.

[0109] Embodiment 2:

[0110] As Figure 2 shown, the present application provides a data answering device, and the device includes:

[0111] A problem module 1, configured to obtain a data problem and determine a first data table generated by a continuous job pointed to by the data problem;

[0112] A positioning module 2, connected to the problem module 1, configured to locate a first node corresponding to the first data table in the data lineage knowledge graph of the continuous job;

[0113] A preorder module 3, connected to the positioning module 2, configured to obtain a preorder data lineage knowledge graph including the first node and its preorder data lineage;

[0114] An answering module 4, connected to the preorder module 3, configured to use the large language model LLM to analyze the preorder data lineage knowledge graph to obtain an answer to the data problem;

[0115] Wherein, the data lineage knowledge graph records the dependency relationship between each data table during the continuous job and the operations performed on each data table during the continuous job process.

[0116] In an implementation manner, the data lineage knowledge graph specifically includes at least one of the following:

[0117] The first data lineage knowledge graph takes the external source data tables of continuous jobs and the target data tables generated in each job link as nodes, takes the dependency relationships between the data tables in the continuous job process as edges, and marks the operator operators that operate the job source data tables connected by the edges as job target data tables on each edge;

[0118] The second data lineage knowledge graph takes the external source data tables of continuous jobs and the target data tables generated in each job link as nodes, takes the dependency relationships between the data tables in the continuous job process as edges, and marks the operation descriptions of the job source data tables completed by the nodes through all their directly connected previous edges on the nodes that are job target data tables;

[0119] The third data lineage knowledge graph divides communities on the basis of the second data lineage knowledge graph, and obtains the community summaries of each community according to the operation descriptions of all the nodes in the same community.

[0120] In one embodiment, the device further includes:

[0121] A job module, which is used to perform continuous jobs on data using the open-source stream processing framework Flink, obtain the job source data tables and job target data tables of each job link in the continuous job, and the operator operators or operation descriptions used to operate the job source data tables as job target data tables in each job link;

[0122] A knowledge graph module, connected to the job module and the problem module 1, is used to take the job source data tables and job target data tables as nodes, take the dependency relationships of the operations between the job source data tables and job target data tables as edges, mark the operator operators used on each edge or mark the operation descriptions of each node of the job target data table, so as to establish the first or second data lineage knowledge graph.

[0123] In one embodiment, the job module specifically includes:

[0124] A job record unit, which is used to generate the external source data tables of the continuous job using the structured query language SQL of Flink and generate the target data tables of each job link according to the external source data tables and / or the target data tables of the previous job link, and record the SQL tables of the continuous job and / or record the process topology graph of the continuous job using the StreamGraph of the data flow graph of Flink;

[0125] A record analysis unit, connected to the job record unit, is used to obtain the operator operators used by each job link to operate the job source data table as the job target data table from the SQL table and / or the process topology diagram, or use the LLM to analyze the job source data table, the job target data table, and the operator operators in the SQL table and / or the process topology diagram to obtain the operation description of each job link.

[0126] In one embodiment, in response to the establishment of the second data lineage knowledge graph, the knowledge graph module is further used for:

[0127] Determine the maximum number of nodes that can be included in a community according to the upper limit of the processing text minimum unit token of the LLM;

[0128] Divide several nodes in the second data lineage knowledge graph into the same community according to the operation description and the maximum number of nodes;

[0129] Use the LLM to summarize the operation descriptions of several nodes in the same community to generate a community summary and add it to the second data lineage knowledge graph to form the third data lineage knowledge graph.

[0130] In one embodiment, the problem module 1 specifically includes:

[0131] A problem receiving unit, used to obtain data problems where business personnel inquire about the meaning of data or verify data errors;

[0132] A problem analysis unit, connected to the problem receiving unit, is used to use the LLM to identify the data table names in the data problems inquired or verified by business personnel;

[0133] A matching unit, connected to the problem analysis unit, is used to match the inquired or verified data table names with the names of the data tables generated by continuous jobs to determine the first data table pointed to by the data problem.

[0134] In one embodiment, the answer module 4 specifically includes:

[0135] A data table meaning analysis unit, used to, in response to using the LLM to identify that the data problem is that business personnel inquire about the meaning of the data table and the previous data lineage knowledge graph includes several communities, use the LLM to summarize the community summaries of the several communities to obtain the meaning of the inquired data table as the answer to the data problem;

[0136] The data item meaning analysis unit is used to, in response to identifying, using the LLM, that the data problem is that a businessperson asks about the meaning of a data item in a data table, sequentially match the asked data item with the headers of the data tables recorded in each node of the previous data lineage knowledge graph to determine the second node where the asked data item first appears, and use the LLM to analyze the job source data tables and operator descriptors of all the directly connected previous edges of the second node or the operation description of the second node to obtain the meaning of the asked data item as the answer to the data problem;

[0137] The data error analysis unit is used to, in response to identifying, using the LLM, that the data problem is that a businessperson verifies an incorrect data item in a data table, sequentially match the verified data item with the headers of the data tables recorded in each node of the previous data lineage knowledge graph to determine the third node where the verified data item is first incorrect, and use the LLM to analyze the third node to obtain the reason for the error of the verified data item as the answer to the data problem.

[0138] In one embodiment, the data error analysis unit specifically includes:

[0139] The data source error unit is used to, in response to the third node being an external source data table node, use the LLM to generate that the reason for the error of the verified data item is an external source data error;

[0140] The data processing error unit is used to, in response to the third node not being an external source data table node, use the LLM to analyze the operator descriptors of all the directly connected previous edges of the third node or the operation description of the third node to obtain that the reason for the error of the verified data item is a certain data operation error of the third node.

[0141] Example 3:

[0142] Embodiment 3 of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is run by a processor, it implements the data question-answering method as described in Embodiment 1 or implements the data question-answering device as described in Embodiment 2.

[0143] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, computer program units, or other data. The computer-readable storage medium includes, but is not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), digital versatile discs (DVDs) or other optical disc storage, magnetic cassettes, tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0144] In addition, the present application can also provide a computer device, including a memory and a processor. A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the data answering method described in Embodiment 1. The computer device can be the data answering device described in Embodiment 2.

[0145] Among them, the memory is connected to the processor. The memory can be a flash memory, a read-only memory, or other memories. The processor can be a central processing unit or a single-chip microcomputer.

[0146] Embodiments 1-3 of the present application provide a data answering method, device, and medium. By combining knowledge graph and large model technologies, the data lineage knowledge graph related to the data problem is analyzed using the large model and the data problem is answered, realizing automatic answering of data problems, which can be used to help business personnel understand the meaning of data or troubleshoot data errors, etc., thereby improving data operation efficiency and reducing data operation costs.

[0147] It can be understood that the above embodiments are merely exemplary embodiments adopted to illustrate the principle of the present application. However, the present application is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present application, and these modifications and improvements are also regarded as the protection scope of the present application.

Claims

1. A data question answering method, characterized in that: The method comprises: Acquire a data problem and determine that the data problem points to a first data table generated by a continuous job; Locate the first node corresponding to the first data table in the data lineage relationship knowledge graph of the continuous operation; Obtain a preceding data blood relationship knowledge graph including the first node and its preceding data blood relationship; Use the large language model LLM to analyze the previous data kinship knowledge graph to obtain answers to data questions; Among them, the data lineage knowledge graph records the dependencies between each data table in the continuous operation process and the operations performed on each data table in the continuous operation process.

2. The method according to claim 1, characterized in that The data lineage relationship knowledge graph includes at least one of the following: The first data lineage relationship knowledge graph uses the external source data table of the continuous operation and the target data table generated by each operation link as nodes, the dependency relationship between the data tables in the continuous operation process as edges, and marks each edge with the operator that operates the operation source data table connected by the edge as the operation target data table in the operation; The second data lineage relationship knowledge graph uses the external source data table of the continuous operation and the target data table generated by each operation link as nodes, and the dependency relationship between the data tables in the continuous operation process as edges. The node that is the target data table of the operation is marked with the operation description of the operation completed by the source data table of the operation connected by the node through all its directly connected preceding edges; The third data kinship knowledge graph divides communities based on the second data kinship knowledge graph, and obtains a community summary of each community based on the operation descriptions of all nodes in the same community.

3. The method according to claim 2, characterized in that Before obtaining the data problem and determining that the data problem points to the first data table generated by the continuous job, the method further includes: Use the open source stream processing framework Flink to perform continuous operations on data, obtain the source data table and target data table of each operation link of the continuous operation, and the operator or operation description used by each operation link to operate the source data table into the target data table; Taking the job source data table and the job target data table as nodes, and the dependency relationship of the operations between the job source data table and the job target data table as edges, mark each edge with the operator it uses or mark each node of the job target data table with its operation description to establish the first or second data lineage relationship knowledge graph.

4. The method according to claim 3, characterized in that Use the open source stream processing framework Flink to perform continuous operations on data, obtain the source data table and target data table of each operation link of the continuous operation, and the operator or operation description used by each operation link to operate the source data table into the target data table, including: Use Flink's structured query language SQL to generate the external source data table of the continuous job and generate the target data table of each job link based on the external source data table and / or the target data table of the previous job link, and record the SQL table of the continuous job and / or use Flink's data flow graph StreamGraph to record the process topology diagram of the continuous job; Obtain the operator used by each job link to operate the job source data table into the job target data table from the SQL table and / or the process topology diagram, or use LLM to analyze the job source data table and the job target data table as well as the operator in the SQL table and / or the process topology diagram to obtain the operation description of each job link.

5. The method according to claim 4, characterized in that In response to the second data blood relationship knowledge graph being established, the method further includes: According to the upper limit of the minimum unit token of LLM processing text, the maximum number of nodes that can be included in a community is determined; According to the operation description and the maximum number of nodes, a plurality of nodes in the second data blood relationship knowledge graph are divided into the same community; The operation descriptions of several nodes in the same community are summarized using LLM to generate a community summary and added to the second data lineage relationship knowledge graph to form a third data lineage relationship knowledge graph.

6. The method according to any one of claims 2 to 5, characterized in that: Obtaining a data problem and determining that the data problem points to a first data table generated by a continuous job specifically includes: Obtain data questions from business personnel asking about the meaning of data or verifying data errors; Use LLM to identify the data table names that business personnel inquire or verify in data issues; The name of the data table to be queried or checked is matched with the name of the data table generated by the continuous job to determine the first data table to which the data problem points.

7. The method according to claim 6, characterized in that Use the large language model LLM to analyze the previous data kinship knowledge graph to obtain answers to data questions, including: In response to using LLM to identify that the data question is that the business personnel inquire about the meaning of the data table, and the previous data lineage relationship knowledge graph includes several communities, using LLM to summarize the community summaries of the several communities to obtain the meaning of the inquired data table as an answer to the data question; In response to using LLM to identify that the data problem is that the business personnel inquire about the meaning of the data item in the data table, the inquired data item is matched with the header of the data table recorded by each node in the previous data lineage relationship knowledge graph in turn to determine the second node where the inquired data item appears earliest, and the LLM is used to analyze the job source data table and operator descriptor or operation description of the second node of all directly connected previous edges of the second node to obtain the meaning of the inquired data item as the answer to the data problem; In response to using LLM to identify that the data problem is an erroneous data item in the data table checked by the business personnel, the checked data item is matched in turn with the header of the data table recorded in each node in the previous data lineage relationship knowledge graph to determine the third node where the checked data item is erroneous the earliest, and the third node is analyzed using LLM to obtain the reason why the checked data item is wrong as the answer to the data problem.

8. The method according to claim 7, characterized in that Use LLM to analyze the third node to obtain the reasons for the errors in the verified data items, including: In response to the third node being an external source data table node, the cause of the data item error checked using LLM generation is an external source data error; In response to the third node not being an external source data table node, LLM is used to analyze the operator descriptors of all directly connected preceding edges of the third node or the operation description of the third node to obtain the cause of the verified data item error being a data operation error of the third node.

9. A data question answering device, characterized in that: The device comprises: A problem module, used for obtaining a data problem and determining that the data problem points to a first data table generated by a continuous job; A positioning module, connected to the question module, for locating a first node corresponding to the first data table in the data kinship relationship knowledge graph of the continuous operation; A pre-order module, connected to the positioning module, for obtaining a pre-order data blood relationship knowledge graph including the first node and its pre-order data blood relationship; The answer module is connected to the previous module and is used to use the large language model LLM to analyze the previous data blood relationship knowledge graph to obtain answers to data questions; Among them, the data lineage knowledge graph records the dependencies between each data table in the continuous operation process and the operations performed on each data table in the continuous operation process.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data question answering method according to any one of claims 1 to 8 is implemented.

Images