Knowledge base construction method, data processing method, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a knowledge base construction method, a data processing method, equipment, a storage medium and a program product. The method comprises the following steps: generating table description information of a data table in database data and field description of the data table based on the database data and document data; identifying table relation information corresponding to the data table from historical query statements of the database data; therefore, the asset knowledge base can be constructed based on the table description information, the field description information and the table relation information, the table description information and the field description information are generated not only based on the database data, but also based on the document data, and the table relation information is recognized and obtained from the table description information and the field description information in combination with the historical query statements and the document data. Therefore, the information in the constructed asset knowledge base is rich and available, and the data quality of the asset knowledge base is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of large model technology, and in particular to a knowledge base construction method, a data processing method, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In some enterprise database application scenarios based on big models, such as querying data from the database, processing data, or converting text into NL2SQL for database SQL, building a high-quality asset knowledge base becomes a solution to improve the performance of big models.

[0003] The asset knowledge base is used to centrally store and manage organizational data assets, including data tables, data fields, etc. The asset knowledge base can be used to build training samples for large models to fine-tune the large models or as auxiliary information to enhance retrieval of the large models.

[0004] Therefore, how to build a high-quality asset knowledge base has become a technical problem that needs to be solved at present. Summary of the invention

[0005] The embodiments of the present application provide a knowledge base construction method, a data processing method, a device, a storage medium and a program product to solve the technical problem of how to build a high-quality asset knowledge base in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a method for constructing a knowledge base, comprising:

[0007] Obtaining database data and document data corresponding to the data provider; the database data includes data table data and metadata in the database;

[0008] Generate table description information of a data table in the database and field description information of the data table based on the database data and the document data;

[0009] Identifying table relationship information corresponding to the data table from historical query statements of the database data and the document data;

[0010] An asset knowledge base is constructed based on the table description information, the field description information, and the table relationship information.

[0011] In a second aspect, an embodiment of the present application provides a data processing method, including:

[0012] Identify the asset knowledge base;

[0013] Identifying target knowledge information matching the input data from the asset knowledge base; the target knowledge information includes target table description information, target field description information and / or target table relationship information;

[0014] Based on the input data and the target table description information, the target field description information and / or the target table relationship information, a target big model is input to obtain output data.

[0015] In a third aspect, an embodiment of the present application provides a data processing method, including:

[0016] Identify the asset knowledge base;

[0017] Constructing training samples based on table description information, field description information, and table relationship information in the asset database;

[0018] The target large model is trained using the training samples.

[0019] In a fourth aspect, an embodiment of the present application provides a data processing method, including:

[0020] Use the target large model to process query requests;

[0021] In the case where the query request involves database data, querying target knowledge information from an asset knowledge base corresponding to the database data; the target knowledge information includes table description information, target field description information and / or target table relationship information;

[0022] Based on the target knowledge information, generating a query result;

[0023] The query result is used as output data of the target large model.

[0024] In a fifth aspect, an embodiment of the present application provides a knowledge base construction device, including:

[0025] A data acquisition module is used to acquire database data and document data corresponding to a data provider; the database data includes data table data and metadata in a database;

[0026] A description information generation module, used to generate table description information of a data table in the database and field description information of the data table based on the database data and the document data;

[0027] A first information identification module, used to identify table relationship information corresponding to the data table from the historical query statements of the database data and the document data;

[0028] The knowledge base construction module is used to construct an asset knowledge base based on the table description information, the field description information and the table relationship information.

[0029] In a sixth aspect, an embodiment of the present application provides a data processing method, including:

[0030] A first knowledge base determination module, used to determine an asset knowledge base;

[0031] A second information identification module is used to identify target knowledge information matching the input data from the asset knowledge base; the target knowledge information includes target table description information, target field description information and / or target table relationship information;

[0032] The data generation module is used to input the target big model based on the input data and the target table description information, the target field description information and / or the target table relationship information to obtain output data.

[0033] In a seventh aspect, an embodiment of the present application provides a data processing method, including:

[0034] A second knowledge base determination module, used to determine an asset knowledge base;

[0035] A sample construction module, used to construct a training sample based on the table description information, field description information and table relationship information in the asset database;

[0036] The model training module is used to train the target large model using the training samples.

[0037] In an eighth aspect, an embodiment of the present application provides a data processing method, including:

[0038] A request processing module, used to process query requests using the target large model;

[0039] A query module, used for querying target knowledge information from an asset knowledge base corresponding to the database data when the query request involves database data; the target knowledge information includes table description information, target field description information and / or target table relationship information;

[0040] A result generation module, used to generate a query result based on the target knowledge information;

[0041] A data output module is used to use the query result as output data of the target large model.

[0042] In a ninth aspect, an embodiment of the present application provides a computing device, including a processing component and a storage component;

[0043] The storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the knowledge base construction method provided in the present application, or to implement the data processing method provided in the present application.

[0044] In the tenth aspect, a computer-readable storage medium is provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processing component, it implements the knowledge base construction method provided in the present application, or implements the data processing method provided in the present application.

[0045] In the eleventh aspect, a computer program product is provided in an embodiment of the present application, including a computer program / instruction, which, when executed by a processing component, implements the knowledge base construction method provided in the present application, or implements the data processing method provided in the present application.

[0046] In an embodiment of the present application, by performing data analysis on database data and document data, table description information of the data table in the database data and field description information of the data table are generated; and table relationship information corresponding to the data table is identified from historical query statements of the database data and document data; thereby, an asset knowledge base can be constructed based on the table description information, the field description information and the table relationship information. The embodiment of the present application not only generates table description information and field description information based on database data, but also identifies and obtains table relationship information therefrom in combination with historical query statements, thereby enriching the information in the constructed asset knowledge base and improving the data quality of the asset knowledge base.

[0047] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A flowchart showing an embodiment of a knowledge base construction method provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a knowledge base construction process in a practical application of an embodiment of the present application is shown;

[0051] Figure 3 A flowchart of an embodiment of a data processing method provided by an embodiment of the present application is shown;

[0052] Figure 4 A flowchart of another embodiment of a data processing method provided by an embodiment of the present application is shown;

[0053] Figure 5 A flowchart of another embodiment of a data processing method provided by an embodiment of the present application is shown;

[0054] Figure 6 A schematic diagram showing the structure of an embodiment of a knowledge base construction device provided in an embodiment of the present application is shown;

[0055] Figure 7 A schematic diagram showing the structure of an embodiment of a data processing device provided in an embodiment of the present application is shown;

[0056] Figure 8 A schematic structural diagram of another embodiment of a data processing device provided in an embodiment of the present application is shown;

[0057] Fig. 9 A schematic structural diagram of another embodiment of a data processing device provided in an embodiment of the present application is shown;

[0058] Fig.10 A schematic diagram of the structure of an embodiment of a computing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0060] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0061] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0062] In the current field of big model research, the focus is gradually shifting from pure model algorithm exploration to exploration that pays more attention to data quality. The driving force behind this shift is the increasing importance of data quality to model optimization. For example, in scenarios such as finding data, using data, or converting natural language (English: Natural Language, abbreviated as: NL) to SQL (Structured Query Language) statements in NL2SQL (NL-to-SQL) scenarios, the requirements for data accuracy are constantly increasing, and data quality has become particularly critical. Taking the NL2SQL scenario as an example, the big model is used to convert natural language query text into SQL statements. The big model needs to identify keywords in the natural language query text and associate them with table names, field names, etc. in the database. The challenge in this process is to perform semantic analysis on the database and understand the vocabulary and data relationships in the database.

[0063] In traditional solutions, large models often use metadata as auxiliary information. However, the inventors found in the process of implementing this application that some databases, such as ODPS (Open Data Processing Service) or Hadoop (an open source big data processing framework) ecological Metastore (a database used to store metadata in Hadoop), etc., cannot intuitively reflect the relationship between tables in the data model design stage. Therefore, the asset knowledge base built based on metadata is still relatively costly to understand for large models due to the lack of table relationship information. In addition, the table annotation information and field annotation information involved in the metadata are manually provided, and there are also problems such as non-standard annotations, which will also increase the cost of understanding large models.

[0064] Therefore, how to build a high-quality asset knowledge base to improve the performance of large models has become a technical problem that needs to be solved urgently. Based on this, the inventor has proposed a technical solution of the embodiment of the present application after multiple rounds of testing. In the embodiment of the present application, data analysis is performed on the database data and the document data to generate table description information of the data table in the database data and the field description information of the data table; and the table relationship information corresponding to the data table is identified from the historical query statements of the database data and the document data; thus, an asset knowledge base can be constructed based on the table description information, the field description information and the table relationship information. The embodiment of the present application not only generates table description information and field description information based on database data, but also combines historical query statements to identify and obtain table relationship information, thereby enriching the information in the constructed asset knowledge base and improving the data quality of the asset knowledge base.

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0066] Figure 1 A flow chart of a knowledge base construction method provided by an embodiment of the present application is shown as follows: Figure 1 As shown, the knowledge base construction method may specifically include the following steps:

[0067] 101: Obtain database data and document data corresponding to the data provider.

[0068] Database data may refer to database data corresponding to a data provider, and the asset knowledge base constructed by the embodiment of the present application may be used to connect to the target large model used by the data provider. The data provider may refer to an enterprise, for example. In actual applications, a lot of data may be generated during the production process of an enterprise, and these data may be stored in a database built by the enterprise. In addition, some data may exist in the form of text, video, or audio to form document data, such as text files such as plain text files (.txt), office documents, PDF files, web pages, emails, blog posts, academic papers, reports, etc.

[0069] Database data may include data table data and metadata, where metadata may include, for example, database schema information. In practical applications, enterprise databases may usually use ODPS and / or Hadoop's Metastore, etc. Such databases lack metadata information, often lacking table relationship information.

[0070] 102: Generate table description information of a data table in the database data and field description information of the data table based on the database data and the document data.

[0071] The table description information is the descriptive information of the data table, which may involve multiple table description types, for example, the table name, table type, table comment information, etc. The table description information may be extracted from the database data according to multiple table description types.

[0072] The field description information may include descriptive information of the field, and may involve multiple field description types, such as field name, field comment, table name of the data table to which it belongs, etc. The field description information may be extracted from database data according to multiple field description types.

[0073] The various table description types and the various field description types may be set in combination with actual data quality requirements for the asset knowledge base, so that the generated table description information and field description information meet the data quality requirements.

[0074] The specific generation method of the table description information and the field description information will be introduced in detail in the corresponding embodiments below.

[0075] In addition, since document data is also data generated during the production process of the data provider, it usually also contains entity-related information of entities associated with the data table. Such information can also be used to generate table description information, etc., which will be introduced in detail in the corresponding embodiments below.

[0076] Optionally, the large model may be used to generate table description information of a data table in the database data and field description information of the data table based on the database data and the document data.

[0077] 103: Identify table relationship information corresponding to the data table from historical query statements of database data and document data.

[0078] The historical query statements can be recorded in the historical query records in the metadata. The historical query statements often involve table relationship information. The embodiment of the present application can identify and obtain the table relationship information by analyzing and processing the historical query statements.

[0079] Among them, historical query statements may include query statements corresponding to historical data query operations or data cleaning operations on database data. Data cleaning operations refer to a series of operations on data to improve its quality, including but not limited to removing duplicate data, correcting erroneous data, filling missing values, standardizing data formats, etc. In a database environment, data cleaning and data query are usually completed using query statements, such as SQL statements, which usually reflect table relationships. For example, cross-table operations such as joins (JOIN) and subqueries.

[0080] In addition, since the document data may contain entity-related information related to entities associated with the data table, such as entity-related information of entities corresponding to the data table, such as entity relationship information, entity operation information, etc., the table relationship information can also be identified from the entity relationship information and entity operation information.

[0081] Optionally, the big model may be used to generate table description information and field description information of a data table in the database data based on the database data and the document data. The big model may be used to identify table relationship information corresponding to a data table from historical query statements of the database data and the document data, which will be described in detail in the following embodiments.

[0082] 104: Build an asset knowledge base based on the table description information, the field description information, and the table relationship information.

[0083] Among them, the asset knowledge base is used to connect to the target big model, for example, to generate training corpus to fine-tune the target big model or as auxiliary information to achieve retrieval enhancement of the big model or to help the big model perform semantic understanding during data query, etc., and to serve as the intellectual property knowledge base of the big model.

[0084] In one embodiment of the present application, the historical query statements may include all query statements historically executed in the database, but are not limited thereto, and may also refer to query statements executed within a preset historical time period. The query statements may refer to, for example, SQL statements.

[0085] Among them, table relationship information can refer to the relationship between a data table and other data tables stored in the database; table relationship information can include, for example, table association, table lineage, field lineage, lineage path information, etc. Table association refers to the connection between different data tables, such as a record in a data table is associated with a record in another data table. The association relationship is usually established by associating a foreign key in one data table with a primary key in another data table. For example, there is a customer ID (identification) field in the order table, which references the primary key ID in the customer table; table lineage refers to the lineage at the table granularity, indicating the source data table and target data table involved in the data flow process; field lineage refers to the lineage at the field granularity, indicating the fields in the source data table and the fields in the target data table involved in the data flow process; lineage path information can refer to the data tables involved in the data flow process, the flow order, the path depth, etc. In addition, table relationship information may also include inclusion relationships, transformation relationships, dependency relationships, aggregation relationships and / or derivation relationships of tables or fields. An inclusion relationship may refer to a field or record in a data table being part of a field or record in another data table. For example, an order table contains information in an order detail table, and each record in the order detail table corresponds to a specific order. A transformation relationship may refer to the change that occurs in the process of data from one data table or field to another. A dependency relationship describes the situation in which a field or data table depends on another field or data table. This dependency may be logical or formed during the data processing process. An aggregation relationship describes how to aggregate data from multiple data tables into one data table. For example, a sales overview table may aggregate data from multiple sales detail tables. A derivation relationship refers to the fact that data in a field or data table is calculated based on data in other fields or data tables.

[0086] Table relationship information can be analyzed and obtained from query statements. For example, in SQL statements, JOIN operations can describe the association relationship between tables; subqueries and CTEs (Common Table Expressions) define the flow of data between different tables, thereby determining the blood relationship of tables; in addition, field selection and calculations in SQL statements can also represent the dependency relationship between fields, and the definition of aggregate and calculated fields can display the derivative relationship of fields, thereby indicating the blood relationship of fields.

[0087] In an embodiment of the present application, after generating table description information, field description information and table relationship information, an asset knowledge base can be constructed based on the table description information, field description information and table relationship information. The asset knowledge base can be used to connect to the target big model, so that the asset knowledge base can provide data support for the training and reasoning of the target big model.

[0088] The asset knowledge base can provide a set of standardized interfaces to connect with the target big model, so that the target big model can easily query and access the data in the asset knowledge base.

[0089] In this embodiment, by extracting database data, generating table description information, field description information of the data tables in the database, and identifying table relationship information corresponding to the data tables, and building an asset knowledge base to connect to the target big model based on the table description information, field description information, and table relationship information, the asset knowledge base contains more comprehensive information related to the database assets, ensuring data quality, thereby helping the target big model to better understand and process data, and improving the interpretability and explainability of the big model.

[0090] In actual application scenarios, the accuracy, completeness, and validity of the data in the asset database have an important impact on the reasoning correctness and training results of the target large model. Therefore, before generating the table description information and field description information of the data table, the quality of the database data can be evaluated first.

[0091] In some embodiments, generating table description information of data tables in the database data and field description information of data tables based on database data and document data can be specifically implemented as follows: performing quality assessment on the database data and document data to screen database data and document data that meet data quality requirements; and generating table description information of data tables in the database data and field description information of data tables for database data that meet data quality requirements.

[0092] Among them, the quality assessment of the database data may include, for example, determining whether there are missing values ​​in the data table, whether the field data exceeds the value range, whether the data format of the field data meets the format requirements, whether there is duplicate data, etc. The data quality requirements may include the absence of missing values, the field data being within the value range, the data format of the field data meeting the format requirements, and / or the absence of duplicate data, etc. It should be noted that this is only an example of a possible quality assessment method, and the quality assessment method can be set in combination with the actual situation, or a large model can be used to screen database data that meets the data quality requirements from the database data, and this application does not limit this.

[0093] Similarly, quality assessment of document data may also include determining whether there are missing values ​​in the document data, whether the value range is exceeded, whether the data format meets the format requirements, whether there is duplicate data, etc. Of course, the document data that meets the data quality requirements can also be screened from the large model data document data.

[0094] Then, after the quality assessment of the database data and document data, the acquired database data and document data can be cleaned according to the inspection results, and the data tables and fields that do not meet the data quality requirements can be cleared to screen the database data and document data that meet the quality requirements. Thus, the table description information and the field description information of the data table can be generated for the database data and document data that meet the quality requirements after screening, so as to ensure the data quality of the asset knowledge base, improve the data accuracy, completeness and consistency of the asset database, and improve the value and reliability of the asset database.

[0095] In order to make full use of database data, when performing quality assessment, data that does not meet quality requirements can be maintained. In some embodiments, the knowledge base construction method may also include: outputting data maintenance prompt information for database data that does not meet data quality requirements; updating database data in response to a data adjustment request, and using the updated database data as database data that meets data quality requirements.

[0096] When performing quality assessment on database data, data that does not meet data quality requirements can be marked, and after the quality assessment is completed, data maintenance prompt information is output. The data maintenance prompt information may include specific data quality issues and suggested correction methods.

[0097] Among them, the data maintenance prompt information can be sent to relevant personnel, the data adjustment request can be triggered by relevant personnel, and the data adjustment request can include correction methods, etc., so that database data that does not meet the quality requirements can be updated, and the updated database data can be used as database data that meets the quality requirements.

[0098] For document data that does not meet the data quality requirements, data maintenance prompt information can also be output, so that the document data is updated in response to the data adjustment request, and the updated document data is used as database data that meets the symbol data quality requirements.

[0099] In one embodiment of the present application, after the database is updated, the quality of the updated database may be evaluated again to determine whether the updated database data meets the data quality requirements.

[0100] In addition, in the database, the data model has an important and direct impact on data quality. The data model is an abstract framework that describes the structure, organization, and management of data. It defines the data structure and storage method, as well as how to operate the data such as adding, deleting, accessing, maintaining, and updating, so that the database system can efficiently and accurately process and store large amounts of data.

[0101] Therefore, the data model can affect the accuracy, consistency, completeness, scalability and availability of the data. Therefore, in order to ensure the data quality of the database data and to ensure the data quality of the asset knowledge base built based on the database data, the quality assessment of the database data model can also be carried out on the basis of the quality assessment of the database data.

[0102] In some embodiments, the knowledge base construction method may further include: performing a quality assessment on a data model of database data to determine whether the data model meets the model quality requirements. If not, outputting a model maintenance prompt message; and updating the data model in response to a model adjustment request.

[0103] Wherein, when performing a quality assessment on a data model, for example, the quality assessment of the data model can be performed from at least one assessment dimension, so as to obtain a quality score of the data model according to the assessment score of at least one assessment dimension, and determine whether the data model meets the model quality requirements according to the quality. The at least one assessment dimension may include, for example, normativeness and / or validity, etc. For example, a normative assessment of a data model may be performed to assess whether the data structure and organization of the data model meet the normative requirements, such as whether the table usage instructions, annotations, naming, and fields meet the corresponding description requirements, etc., so that a normative assessment score can be obtained according to the judgment result; for example, a validity assessment of a data model may be performed to assess whether the data table defined by the data model meets the validity requirements, and the validity requirements may include, for example, the absence of an empty data table that is no longer updated, the absence of a data table that has not been accessed in the recent period, the absence of a data table with an incremental record of 0, the absence of a downstream application scenario, and at least one of the conditions of the downstream application being offline, etc., so that the corresponding assessment score can be determined according to the valid conditions met.

[0104] Optionally, the quality of the data model may be obtained by performing a weighted summation on the evaluation scores of at least one evaluation dimension, and the weight coefficient corresponding to each evaluation dimension may be set based on actual conditions, etc. This application does not impose any limitation on this.

[0105] Of course, the above-mentioned quality assessment of the data model can also be implemented using a large model, and this application does not limit this.

[0106] When it is determined through quality assessment that the data model does not meet the model quality requirements, in some embodiments, the method may further include: outputting model maintenance prompt information, which may include quality issues of the data model and suggested correction methods.

[0107] Among them, outputting the model maintenance prompt information may be sending the model maintenance prompt information to relevant personnel, and the model adjustment request may be triggered by the relevant personnel, and may include a correction method to update the data model according to the correction method.

[0108] Among them, the model quality requirements can be set based on actual conditions to evaluate the quality of the data model, and this application does not limit this.

[0109] In order to accurately and comprehensively determine table relationship information of data tables in database data, in some embodiments, the knowledge base construction method may further include: determining manually configured table relationship information in response to a relationship maintenance request for database data.

[0110] Manually configuring table relationships may refer to clarifying the table relationship information of data tables in a database by manually defining them, for example, defining primary key and foreign key relationships, specifying connection conditions, and describing the relationship between tables in logic to clarify the association relationship between database tables, etc. Therefore, the above-mentioned identifying the table relationship information corresponding to the data table from the historical query statements of the database data may be specifically implemented as follows: identifying the table relationship information corresponding to the data table from the historical query statements of the database data and the manually configured table relationship information.

[0111] In addition, the metadata may also include some table relationship information. In some embodiments, the above-mentioned identification of the table relationship information corresponding to the data table from the historical query statements of the database data and the document data can be specifically implemented as follows: identifying the table relationship information corresponding to the data table from the historical query statements of the database data, the document data, the table relationship information maintained in the metadata, and the manually configured table relationship information.

[0112] As mentioned above, the historical query statements of the database may contain table relationship information, such as JOIN, subquery, etc., and the table relationship information can be identified by parsing the query statements. In addition, the metadata of the database usually includes table structure information, primary key, foreign key, etc., which can also be used to determine the table relationship.

[0113] On this basis, when determining the table relationship information corresponding to the data table, the manually configured table relationship information can also be combined, so that the table relationship information can be identified more comprehensively and accurately.

[0114] In some embodiments, generating table description information and field description information of a data table in the database data based on database data and document data may include: generating table description information and field description information of a data table in the database data based on the database data; extracting entity-related information of an entity associated with the data table from the document data, and using the entity-related information as a table description information.

[0115] Among them, based on database data, generating table description information and field description information of a data table in the database data may include: normalizing the table annotation information of the data table to use the normalized table annotation information as a table description information; extracting table index keywords from the normalized table annotation information to use it as a table description information; using the table index keywords to match the table description information; normalizing the field annotation information of the data table to use the normalized field annotation information as a field description information; extracting field index keywords from the normalized field annotation information to use it as a field description information; using the field index keywords to match the field description information.

[0116] The entity-related information extracted from the document data may be used as a table description information of the data table, or may be incorporated into the table annotation information after normalization.

[0117] In the embodiment of the present application, the table description information may include table annotation information, which is a descriptive annotation of the database table, used to explain the structure, purpose, data source and other information of the data table. In the database, table annotations can be added to the data table through a specific syntax. When the asset database is connected to the target big model, the table annotation information can help the big model understand the structure and purpose of the data table.

[0118] However, combined with the previous description, it can be seen that the table annotations in the current database are not standardized, which is not conducive to the generalization and understanding of the target large model and increases the cost of understanding. Therefore, in order to normalize the table annotation information, the table annotation information can be converted into a more standard and unified form.

[0119] The normalization of the table annotation information may include format normalization and logic normalization, etc. Format normalization is used to standardize the text format of the table annotation information, and logic normalization can be processed from a semantic level to make the annotation content more logically unified and meaningful.

[0120] By performing format normalization and logic normalization on the table annotation information, the table annotation information can be effectively processed so that the table annotation information is consistent in form and semantics.

[0121] In addition, after normalizing the table annotation information, keywords can be extracted from the table annotation information, and the extracted table index keywords can be used as a kind of table description information. The table index keywords can be used to match the table description information, such as matching the input data with the table index keywords to determine the target table description information that the input data matches. The target table description information can be input into the target macro model as auxiliary information of the input data to achieve retrieval enhancement.

[0122] In addition, similar to the table description information, the field annotation information also needs to be normalized. The field annotation is a text used to describe the fields in the database table. It usually contains information such as descriptions of the fields, data types, length limits, constraints, etc. In the database, field annotations can be added to the data table through specific syntax. However, there are problems such as non-standard field annotations, which will increase the cost of understanding the target large model. Therefore, the embodiment of the present application can normalize the field annotation information. Among them, the normalization of the field annotation information may include aspects such as format normalization and logical normalization. Among them, format normalization is used to standardize the text format of the field annotation information, while logical normalization can be processed from a semantic level, so that the annotation content is more logically unified and meaningful.

[0123] Optionally, the format normalization may include, for example, converting uppercase and lowercase letters to a unified format, removing special symbols, removing redundant spaces, removing meaningless characters, unifying date formats, unifying numerical formats, correcting splicing errors, etc. The logic normalization may include, for example, adjusting the order of text, supplementing information, etc. The specific implementation of normalizing text is not limited in this application.

[0124] The entity-related information extracted from the document data and corresponding to the entity in the data table may include the data domain to which the entity belongs, the explanation information of the associated proper nouns, the application fields involved, and other information that can assist in understanding the data assets in the asset knowledge base. Optionally, the entity-related information of the entity associated with the data table may be extracted from the document data using a large model.

[0125] Among them, entity-related information such as the data field to which it belongs, the term explanation information of the associated proper nouns, and the application field involved can be used as a table description information of the corresponding data table or merged into the table annotation information after the data table is normalized.

[0126] In addition, entity-related information extracted from document data can also participate in normalization processing of table annotation information and field annotation information and keyword extraction.

[0127] In some embodiments, the above-mentioned normalization of the table annotation information of the data table to use the normalized table annotation information as a table description information, and extracting table index keywords from the normalized table annotation information as a table description information can be performed by using a large model in combination with entity-related information to normalize the table annotation information of the data table, and extracting table index keywords from the normalized table annotation information to obtain normalized table annotation information and table index keywords, which can be used as a table description information. Optionally, entity-related information and table annotation information can generate corresponding prompt information to be input into the large model, so that the large model obtains the normalized table annotation information and table index keywords, etc.

[0128] The above-mentioned normalization of the field annotation information of the data table to use the normalized field annotation information as a field description information, and extracting field index keywords from the normalized field annotation information as a field description information can be performed by using the big model in combination with entity-related information to normalize the field annotation information of the data table, and extracting field index keywords from the normalized field annotation information to obtain the normalized field annotation information and field index keywords, and the field annotation information and field index keywords can be used as a field description information. Optionally, the entity-related information and the field annotation information can generate corresponding prompt information to be input into the big model, so that the big model obtains the normalized table annotation information and table index keywords, etc.

[0129] In some embodiments, the knowledge base construction method may further include:

[0130] From the database data, the downstream application information of the data table is identified, and the downstream application information is used as a kind of table description information of the data table.

[0131] Among them, the downstream application may refer to an application or system that operates or further processes the data in the data table. The downstream application information may include the identification information, processing type, processing results, etc. of these applications or systems. The downstream application information is also important for the target large model to understand the database. Therefore, in the embodiment of the present application, the downstream application information is also used as a table description information.

[0132] Different downstream application information can reflect the different importance of data tables. For example, if a data table is used by core applications, such as order processing systems or customer relationship systems, then it can be said that this data table is more important; while a data table is only used by auxiliary applications, such as log analysis tools or statistical reporting systems, then this data table may be less important. By using the downstream application information of a data table as a table description information, it can help and deepen the target large model's understanding of the data table.

[0133] In addition, since each data packet in the database can represent an entity set, that is, a collection of entities of the same type, each row (record) in the data table represents a specific entity instance, and each column (field) represents an attribute of the entity. Therefore, the downstream application information of the entity corresponding to the data table can also be extracted from the document data, and the downstream application information of the entity can be converted into the downstream application information of the data table.

[0134] In some embodiments, the knowledge base construction method may further include: identifying downstream application information of the fields in the data table from the database data, and using the downstream application information as a kind of field description information.

[0135] According to the fields corresponding to the data table data used by the downstream application information, the downstream application information can be used as the downstream application information of the field. The downstream application information of the data table can include downstream application information of different fields.

[0136] In some embodiments, the knowledge base construction method may also include: identifying entities in a data table and the application fields corresponding to the entities from database data; using the entities and application fields as a kind of table description information of the data table or merging the entities and application fields into the table annotation information after normalization.

[0137] In the database, an entity can refer to an object in the database that is used to represent information that needs to be stored, such as "customer", "order", "product, etc." Entities are usually represented as data tables in the database. Each data table represents an entity set. Each row (data record) in the data table represents a specific entity instance, and each column (field) represents an attribute of the entity. Therefore, entities can be identified by analyzing the structure, fields, and table annotation information of the data table.

[0138] The application domain may refer to the role or use of an entity in the production process of the data provider. For example, an "order" entity may be related to the "sales management" domain, and a "customer" entity may be related to the "customer service" domain. The application domains mapped by different entities may be pre-configured to determine the application domains mapped by the identified entities.

[0139] In practical applications, entities and their application fields are also important for the target large model to understand the database. Therefore, in a possible implementation of the present application, the identified entities and the application fields corresponding to the entities can be used as a kind of table description information of the data table.

[0140] In another possible implementation of the present application, the entity and the corresponding application field may be merged into the normalized table annotation information to use the table annotation information as a kind of table description information.

[0141] By identifying the entities in the data table and their corresponding application areas, and using this information as table annotation information, the role and importance of the data table can be more comprehensively described, helping the target large model understand the database.

[0142] In addition, application fields of entities associated with data tables can also be extracted from document data.

[0143] In some embodiments, extracting table index keywords from the normalized table annotation information as a kind of table description information may be implemented as follows: extracting table index keywords from the normalized table annotation information and field annotation information using the first large model as a kind of table description information;

[0144] In some embodiments, extracting field index keywords from the field annotation information after normalization as a kind of field description information includes: using a first large model to extract field index keywords from the field annotation information after normalization as a kind of field description information.

[0145] In the embodiment of the present application, the ability of the big model can be used to extract keywords from the normalized table annotation information and field annotation information. Since the big model can understand the semantic relationship in natural language, it can more accurately identify important keywords and improve the accuracy and efficiency of keyword extraction.

[0146] When using the ability of the first large model to extract keywords from the table annotation information and field annotation information after normalization, you can first design a prompt (prompt word). Prompt can help the large model understand the input intent to guide the large model to generate specific output data. Prompt is a form of input used to prompt or guide the large voice model to give expected output. It is used to instruct the large language model what actions should be taken or what output should be generated when performing a specific task. Prompt is a natural language input, similar to a command or instruction, to let the large language model know what it needs to do.

[0147] Optionally, based on the normalized table annotation information and field annotation information, a first prompt word may be generated according to a first prompt template, and the first prompt word may be input into a first large model to extract table index keywords using the first large model.

[0148] Optionally, based on the normalized field annotation information, a second prompt word may be generated according to a second prompt template, and the second prompt word may be input into the first large model to extract field index keywords using the first large model.

[0149] In addition, since some specific professional terms may exist in the table annotation information or the field annotation information, these professional terms may be retained as keywords.

[0150] Therefore, the above-mentioned use of the first large model to extract table index keywords from the table annotation information and field annotation information after normalization processing can be to use the first large model to extract table index keywords from the table annotation information and field annotation information after normalization processing, and retain specific types of vocabulary as table index keywords.

[0151] The above-mentioned use of the first large model to extract field index keywords from the field annotation information after normalization as a kind of field description information can be to use the first large model to extract field index keywords from the table annotation information and field annotation information after normalization, and retain specific types of vocabulary as field index keywords.

[0152] The specific type of vocabulary may include, for example, time type vocabulary, behavior type vocabulary and / or proper nouns, etc.

[0153] For ease of understanding, the first prompt word may be, for example:

[0154] {

[0155] 'There is an existing physical table annotation \"{table}\", and the table contains a field annotation \"{filed}\", please extract the core keywords for the following text \"{table}, {filed}\"\

[0156] , if there are keywords such as p4p, AB, AB3, DUV, 2N, 4PL, 3PL, app, etc. (ignore capitalization), you need to keep \

[0157] If there are time-limited keywords such as the last 1 day, the last 7 days, the last 30 days, the last 90 days, and the fiscal year cumulative, you need to keep them.

[0158] , if there are keywords such as pending account, inquiry, business opportunity, ab, ab3 and other behavior-limited keywords, they need to be retained and the table annotation keyword modification needs to be added\

[0159] , the extracted keywords are required to be based on field annotations as the core content\

[0160] ,Keyword length should not exceed 20\

[0161] , finally please return information in json format only, keyword: keyword'

[0162] }

[0163] In addition, in some embodiments, basic table information such as table name, table type, etc. extracted from database data such as metadata can also be used as a table description information; basic field information such as field name, table name of the corresponding data table, etc. extracted from database data such as metadata can also be used as a table description information.

[0164] In addition, for some fields of a predetermined type, a specified number of field data can be selected as a field description information. The field data can help the big model understand the data output format, etc. The predetermined type can refer to a string type. For example, for a field of string type, if the field name is: Is it XX product, the field data is: Yes or No. "Yes or No" can help the target big model understand that the data output format of the field is a Chinese expression, rather than a letter expression such as "Y or N".

[0165] For example, for a field of a predetermined type, a distinct operation can be first performed to obtain different values ​​of the field, that is, field data, so that different enumeration values ​​of the field can be found to avoid duplication. For example, a field may have only a few fixed values ​​(for example, a status field may have only two values, "enabled" and "disabled"). Through the distinct operation, these enumeration values ​​can be obtained and all possible values ​​of the field can be identified.

[0166] After obtaining all the enumeration values ​​of the field, several values ​​with the highest frequency of occurrence may be selected, such as the first three values ​​with the highest frequency of occurrence, which are the field data, as a kind of field description information.

[0167] Of course, the field data of the predetermined type can also be merged into the field annotation information after normalization, so that the field annotation information not only contains concise core keywords, but also comes with the main enumeration values ​​of the field, making it more intuitive and easy to understand, which not only improves the quality of the data, but also enhances the comprehensibility and operability of the data.

[0168] In addition, the predetermined type of field data can also be directly used as a field index keyword and as field description information.

[0169] In some embodiments, identifying table relationship information corresponding to a data table from historical query statements of database data and document data can be specifically implemented as follows: using the second largest model to analyze historical query statements of database data to extract table relationship information corresponding to the data table; extracting entity-related information of entities associated with the data table from the document data; the entity-related information includes entity relationship information and entity operation information; and identifying the table relationship information corresponding to the data table from the entity relationship information and entity operation information.

[0170] In an embodiment of the present application, the capability of a large model may be utilized to analyze historical query statements of database data to extract table relationship information corresponding to a data table.

[0171] Optionally, a third prompt word may be generated based on the historical query statement, and the third prompt word may be input into the second large model to extract table relationship information corresponding to the data table.

[0172] For ease of understanding, the third prompt word may be, for example:

[0173] Please analyze the following SQL and find out the tables involved in the SQL and the corresponding relationships. Please return them in the following format:

[0174] {

[0175] 'table1':table_name1,

[0176] 'table2':table_name2,

[0177] "rel":table_name1.col=table2.col

[0178] }

[0179] SQL information:

[0180] select*from(select DISTINCT t1.channel_name, t1.domain_account, t1.nam, t1.work_no, t1.total_job_days, t1.regist_date, t1.manager_org_id, t1.manager_org_name, t1.super_show_name, t1.is_dimission, t1.inner_org_id from icbucdm.dim_en_sales_line_user t1 where t1.ds='${bizdate}'and t1.is_sales='Y')t1

[0181] left join(select distinct manager_org_id, manager_org_name, region_org_name, area_org_name, crm_new_zone from icbucdm.dim_en_crm_org where ds='${bizdate}'

[0182] )t2 on t1.manager_org_id=t2.manager_org_id

[0183] left outer join icbudwa.dwm_en_crm_sales_profile_d t3

[0184] on t1.domain_account=t3.crnt_owner_id

[0185] and t3.ds=max_pt('icbudwa.dwm_en_crm_sales_profile_d')

[0186] Among them, the above SQL information is also the historical query statement. The above SQL information describes extracting data from three data packets and connecting them. The above SQL information includes three data tables: data table icbucdm.dim_en_sales_line_user (alias t1); data table icbucdm.dim_en_crm_org (alias t2); data table icbudwa.dwm_en_crm_sales_profile_d (alias t3).

[0187] The main query logic of the above SQL information is: use the manager_org_id field to left join t1 and t2; use the domain_account field and the crnt_owner_id field to left join t1 and t3.

[0188] Therefore, the output data of the second largest model is:

[0189] {

[0190] 'table1':icbucdm.dim_en_sales_line_user

[0191] 'table2':icbucdm.dim_en_crm_org

[0192] "rel":t1.manager_org_id=t2.manager_org_id

[0193] }

[0194] {'

[0195] table1':icbucdm.dim_en_sales_line_user

[0196] 'table2':icbudwa.dwm_en_crm_sales_profile_d

[0197] "rel":t1.domain_account=t3.crnt_owner_id

[0198] }

[0199] The output data of the second largest model is in JSON format, and the extracted table relationship information is the table association relationship, which includes that data table t1 is associated with data table t2, and data table t1 is associated with the primary key manager_org_id of data table t2 through the foreign key manager_org_id; and data table t1 is associated with data table t2, and the primary key crnt_owner_id of data table t3 is associated with the foreign key domain_account.

[0200] The first large model and the second large model mentioned above can be the same large model or different large models, which refer to machine learning models with a large number of parameters and complex structures, which can process massive data and complete various complex tasks, such as natural language processing, computer vision, speech recognition, etc.

[0201] The first or second largest model can adopt the current open source large language model (Large Language Model, LLM), such as GPT-3 (Generative Pre-Trained Transformer-3, the third generation of generative pre-training model), GPT-4 (Generative Pre-Trained Transforme-4, the fourth generation of generative pre-training model), BERT (Bidirectional Encoder Representation from Transformers, a bidirectional encoder model based on Transformers), Turing NLG (Turing Natural language Generation), etc. This application does not limit this.

[0202] From the above description, we can know that entity-related information may include entity operation information and entity relationship information between entities, etc. In a database, entities are usually represented as data tables, each of which can represent an entity set, that is, a collection of entities of the same type. Each row (record) in the data table represents a specific entity instance, and each column (field) represents an attribute of the entity. Therefore, the table relationship information corresponding to the data table can also be identified from the entity operation information and entity relationship information.

[0203] Entity operation information may refer to the behavior operation performed on the entity. The association relationship information or lineage relationship information between multiple data tables or multiple fields involved in the entity operation information may be used, and the association relationship or lineage relationship between multiple data tables involved may be determined based on the entity relationship information. Since each entity may correspond to a data table, the entity relationship information may also represent the corresponding table relationship information.

[0204] Optionally, the big model may be used to extract entity operation information and entity relationship information from the document data, and corresponding prompt information may be generated based on the document data and input into the big model to obtain entity related information such as entity operation information and entity relationship information.

[0205] Optionally, the downstream application information of the above data table and the downstream application information of the field can also be identified and obtained from metadata, table annotation information and / or field information using a large model.

[0206] In some embodiments, generating table description information and field description information of data tables in the database data based on database data can be specifically implemented as follows: based on the table annotation information of the data tables in the database data, the fields and field annotation information contained in the data tables, and metadata, using the third model to generate table description information and field description information of the data tables in the database data.

[0207] In an embodiment of the present application, the big model capability can be used to generate table description information and field description information of the data table in the database data based on the table annotation information of the data table in the database data, the fields and field annotation information contained in the data table, and metadata.

[0208] Optionally, the third largest model can be used to generate table description information based on the table annotation information of the data table in the database data, the fields contained in the data table and the field annotation information, and the metadata; and then the third largest model can be used to generate field description information based on the fields contained in the data table and the field annotation information.

[0209] A fourth prompt word can be generated based on the table annotation information, field annotation information and metadata, and the fourth prompt word can be input into the third model to generate table description information; a fifth prompt word can be generated based on the field annotation information and metadata, and the fifth prompt word can be input into the third model to generate field description information.

[0210] For ease of understanding, the fourth prompt word may be, for example, (it should be noted that this is only an example and the present application is not limited thereto):

[0211] {

[0212] ”'

[0213] ####Expectation\n\

[0214] Please complete the Chinese description summary, keyword extraction, importance judgment and problem judgment of the data table according to the metadata of the given data table, and output it in Chinese according to the format in the output example. The output includes six parts: basic understanding, usage scenarios, keyword extraction, importance judgment, reasons for importance judgment, and table problems. Do not use * or bold. \n\

[0215] ####ROLE\n\

[0216] You are a data management specialist responsible for understanding and analyzing table information in the data warehouse to ensure data is effectively understood, utilized, managed, and found.\n\

[0217] ####Action(ACTION)\n\

[0218] 1. Understanding Metadata: Please carefully read and understand the results in the two columns of "Manually Extracted Keywords" and "Manually Judged Importance", and refer to the provided [Proper Nouns] to deeply understand the [Metadata Information] of each data table. Pay special attention to the content marked with "**Image Number**".

[0219] - For [Metadata]:

[0220] - It is necessary to comprehensively understand the metadata at the table and field levels;

[0221] - Focus on the information marked with "**Image Number**";

[0222] - It is necessary to learn the method of keyword extraction in "Manually Extracted Keywords" and the method of importance judgment with the judgment result of 'Y' in "Manually Judged Importance";

[0223] - The information at the field level should be used for further refinement and summary.

[0224] 2. Summary and Refinement of Chinese Descriptions in Tables:

[0225] - It is divided into two parts: "Basic Understanding" and "Usage Method":

[0226] - "Basic Understanding": Combining all the obtained metadata information, summarize the main content of the table, avoid repeating the Chinese annotations of the table, focus on referring to the table annotations, **Image Number** source table names, and information in the **Image Number** source table descriptions. Do not involve the full / partial judgment of the data table, and describe specifically rather than generally referring to "B2B e-commerce platforms";

[0227] - "Usage Method": Specifically explain how to use this table, referring to the measurement, qualification information, labels, and usage scenarios in the "**Image Number**". If not available, understand based on the fields and table descriptions.

[0228] 3. Keyword Extraction:

[0229] - Combining the summary and refinement of the table, extract keywords from the table annotation information, **Image Number** source table names, **Image Number** source table descriptions, attributed data domains, and table supplementary description information to facilitate users' quick positioning;

[0230] - If there is content in "Manually Extracted Keywords", these keywords need to be included, and learn the extraction methods of these keywords;

[0231] - Reference keywords: Merchant Portrait, Product Portrait, Buyer Portrait, Product Dimension Table (···· omitted here and set according to the actual situation);

[0232] -The keyword length must be greater than or equal to 2 characters, and multiple keywords must be separated by ",";\

[0233] -Do not use overly broad proper nouns such as "application scenario", "creator", "indicator value", "description" and so on; do not use common nouns such as "name" and "contact information" alone, these should be accompanied by the subject, such as "sales name", "sales contact information", "member contact information", etc.; do not use vague keywords such as "organizational relationship", "global sales organizational relationship" and "international organizational relationship" are acceptable. \n\

[0234] 4. Importance judgment:\n\

[0235] -Based on factors such as the table's assets in the "**symbol number**" (if there are assets, it is important), the number of recent visits (visits over 1000 are considered to be high), the priority (7 is the highest), the number of downstream nodes (the more the table, the more important it is), the table's supplementary description information (non-empty ones are relatively more important), the number of partitions (the more partitions, the more important they are), and the last modification time (the later the more important, now it is 2026), the importance of the table is evaluated using three levels: "important", "medium", and "unimportant";\

[0236] - Learn the judgment results of the importance of the column value "Is manual judgment important" equal to 'Y';\

[0237] - Provide reasons for your judgment.\n\

[0238] 5. Source table problem identification:\n\

[0239] - Determine whether there are any problems based on the table and field information, such as no longer recommended, abandoned, not updated for a long time, containing a lot of warehouse modeling professional terms that are difficult to understand, etc.;\

[0240] -If any illegal situation is found, it will be skipped and not output. \n\

[0241] 6. Understanding of proper nouns:\n\

[0242] - Understand the following terms:

[0243] -duv: detail page visit (detail unique visitor);\

[0244] -ab: active buyer;\

[0245] -ab pro: deeply active buyers (advanced version of ab, calculated by algorithm);\

[0246] (····Omitted here, can be set according to the actual situation)

[0247] 7. Output example:\n\

[0248] Take the table `icbucdm.dwd_en_crm_ctrct_ggs_ord_stat_di` as an example, and output it strictly in the following format. Do not bold the title, do not use the * symbol, and do not use the English name of the table:\n\

[0249] 1. Basic understanding: This table provides a light summary of the member contract order information of international merchants, including the merchant's latest order information, first order information, service information, etc.

[0250] 2. Usage scenario: Mainly used for querying order information of international merchants, identifying information such as customers, the latest order, and the earliest order in international services. \

[0251] 3. Keyword extraction: international order statistics, international merchant order information, international service customers. \

[0252] 4. Importance: Important.

[0253] 5. Reason for importance judgment: This table is configured with data quality inspection, has a high priority, and is frequently called downstream. \

[0254] 6. Table issues: The table annotations contain a lot of data warehouse modeling terminology, which is difficult to understand. \n\

[0255] 8. Metadata: {tableMsg}\n\

[0256] Please follow the above guidelines and examples to complete the corresponding tasks for each table in the database.\n\

[0257] ”'

[0258] }

[0259] The third large model may be the same large model as the first large model or the second large model or a different large model, and may also be implemented using an open source large language model.

[0260] In some embodiments, combined with the foregoing description, it can be known that the table relationship information may include table association relationship information, table lineage relationship information, field lineage relationship information, and lineage path information.

[0261] The knowledge base construction method may also include: extracting relationship index keywords corresponding to table association relationship information, table lineage relationship information, field lineage relationship information and lineage path information, respectively, and saving them in the asset knowledge base in correspondence with the table association relationship information, table lineage relationship information, field lineage relationship information and lineage path information; wherein the relationship index keywords are used for relationship identification.

[0262] After the relationship index keywords corresponding to the table association relationship information, table lineage relationship information, field lineage relationship information, and lineage path information are respectively saved in the asset knowledge base, the relationship index keywords can be used to match the table relationship information. For example, the input data is matched with the relationship index keywords to determine the target relationship information that matches the input data. The target relationship information can be input into the target large model as auxiliary information of the input data to achieve retrieval enhancement.

[0263] When the asset knowledge base is used in application scenarios such as retrieval enhancement, in order to facilitate data search, the table description information, field description information, and table relationship information can be converted into feature vectors and then saved in the asset knowledge base.

[0264] In addition, in actual applications, since the data structures of description information (including field description information and table description information) and table relationship information are not interoperable, if the description information and table relationship information are stored together when building an asset knowledge base, data pollution may occur. Therefore, in order to further improve data search and avoid data pollution, the table description information and field description information can be stored separately from the table relationship information. In some embodiments, building an asset knowledge base based on the table description information, field description information and table relationship information includes: building an asset vector library based on the table description information and field description information; building a relationship vector library based on the table relationship information.

[0265] Among them, after the table description information and the field description information are converted into feature vectors respectively, an asset vector library is formed; after the table relationship information is converted into a feature vector, a relationship vector library is formed.

[0266] The asset knowledge base includes the asset vector library and the relationship vector library.

[0267] In one embodiment of the present application, the vectorization capability of the large model can be utilized to construct an asset vector library and a relationship vector library based on the extracted table description information, field description information, and table relationship information.

[0268] By vectorizing and storing the description information and table relationship information separately, data quality can be ensured and data pollution can be avoided, which is conducive to improving the query and processing performance of large models and enhancing the availability and consistency of data.

[0269] In some embodiments, the knowledge base construction method may further include: outputting table description information, field description information, and table relationship information; in response to an information update request for the table description information, field description information, or table relationship information, updating the table description information, field description information, or table relationship information;

[0270] The above-mentioned construction of the asset knowledge base based on the table description information, the field description information and the table relationship information may be: in response to an information confirmation request for the table description information, the field description information and the table relationship information, the asset knowledge base is constructed based on the table description information, the field description information and the table relationship information.

[0271] Outputting the table description information, field description information and table relationship information may include sending the table description information, field description information and table relationship information to relevant personnel for manual review by relevant personnel to update or confirm the information. The above-mentioned information update request and information confirmation request may be triggered by relevant personnel.

[0272] Combined with the above description, it can be seen that the data generated by the data provider during the production process can also be stored in the document data, so the document data also contains a lot of knowledge, which also helps the target large model to understand the data. And the document data may also contain knowledge that is not related to the data table or field, which also helps the target large model to understand the data. Therefore, in some embodiments, the method may also include:

[0273] identifying an object and object-related information of the object from the document data;

[0274] The constructing of the asset knowledge base based on the table description information, the field description information and the table relationship information includes: constructing the asset knowledge base based on the table description information, the field description information, the table relationship information and the object related information.

[0275] Among them, the object can refer to the processing target in the production process of the data provider, such as "order", "commodity", etc.

[0276] The object-related information may include, for example, the application field to which the object belongs, the downstream application scenario of the object, the proper nouns involved in the object and their explanation information, the application field involved, object operation information, and object relationship information. The object operation information may include the behavior operation process performed on the object and the introduction information and execution strategy of an execution subject involved in the behavior operation process. The execution subject may refer to the system that performs a certain behavior operation on the object.

[0277] In combination with the above description, it can be seen that the table description information in the embodiment of the present application may include one or more of the following information:

[0278] Table name, table type, table index keywords, table annotation information, explanation information of proper nouns involved in the data table, and downstream application information of the data table.

[0279] The table type may be, for example, a fact table or a dimension table.

[0280] The field description information may include one or more of the following information:

[0281] The field name, the name of the data table to which it belongs, field annotation information, field index keywords, explanation information of the proper nouns involved in the field, and downstream application information of the field table.

[0282] The table relationship information may include one or more of the following information: table association relationship information, table lineage relationship information, field lineage relationship information, and field lineage path information.

[0283] The table association relationship information may include one or more of the following information:

[0284] Table name, associated table name of associated data table, foreign key field in data table, primary key field in associated data table, association condition, number of associations and relationship index keyword.

[0285] The association condition may refer to, for example, the association method between the foreign key field and the primary key field, such as the foreign key field is the same as the primary key field, or the foreign key field is greater than or less than the primary key field.

[0286] The blood relationship information may include one or more of the following information:

[0287] The table name of the source data table, the target table name of the target data table, the blood relationship type, the relationship creation time, the relationship modification time, and the relationship index keyword.

[0288] Field lineage key information may include one or more of the following information:

[0289] The table name of the source data table, the field name in the source data table, the target table name in the target data table, the target field name in the target data table, the blood relationship type, the relationship creation time, the relationship modification time, and the relationship index keyword.

[0290] Field lineage path information can include one or more of the following information:

[0291] The table name of the source data table, the field name in the source data table, the target table name of the target data table, the target field name in the target data table, the lineage path, the path depth, and the path index keyword.

[0292] The above table description information, field description information, table relationship information can be identified through data, and can be extracted from database data and document data with the help of a large model. The specific implementation can be detailed in the corresponding embodiments above and will not be repeated here.

[0293] The embodiment of the present application can convert all available data of the data provider, such as database data and document data, into the content of the asset knowledge base, and automatically parse and organize the asset knowledge base through data identification, relationship construction, large model understanding, etc., and with the help of prompt design, metadata design, product function design, etc., the asset knowledge base can be directly connected to the target large model, which can be any large model used by the data provider. The target large model can be obtained by pre-training in combination with the corresponding training data of the data provider, etc., and it can be obtained by fine-tuning the currently common large language model, etc.

[0294] To facilitate data search, table description information, field description information, and table relationship information may be stored in a structured manner, etc. Therefore, in some embodiments, building an asset knowledge base based on the table description information, field description information, and table relationship information may include: generating multiple data tables based on the table description information, field description information, and table relationship information according to their respective corresponding data structures; and building an asset knowledge base based on the multiple data tables.

[0295] The data structure can be defined to store table description information, field description information or table relationship information in table form to generate multiple storage tables, which can constitute an asset knowledge base. The storage tables can be stored in the database of the data provider such as ODPS or HDFS, and can also be stored in other databases such as MySQL or Hologres.

[0296] The data structure may define the table name of the storage table and the various fields included therein.

[0297] Optionally, the first storage table may be created according to the first data structure and based on the table description information. For example, in an actual application, the first storage table may be created according to the following SQL statement. Of course, the present application is not limited thereto:

[0298] CREATE TABLE IF NOT EXISTS dwm_en_meta_table_detail_d(

[0299] table_name STRING COMMENT'table name',

[0300] table_comment STRING COMMENT 'table comment',

[0301] table_type STRING COMMENT 'Table type [fact table, dimension table]',

[0302] keywords STRING COMMENT'Table index keywords',

[0303] proper_noun STRING COMMENT 'Explanation of proper nouns involved in the table',

[0304] downstream_used STRING COMMENT' downstream usage scenarios (including downstream execution systems, analysis reports, etc.)' )

[0306] COMMENT 'Details of table'

[0307] PARTITIONED BY (

[0309] ds STRING COMMENT 'Partition' )

[0311] LIFECYCLE 90;

[0312] This SQL statement can be used to create a table named dwm_en_meta_table_detail_d. This table is used to store description information about other tables. The storage table may include the following fields:

[0313] table_name: The field that stores the table name, the data type is STRING (string).

[0314] table_comment: stores table comment information. The data type is also STRING. It can specifically store table comment information after normalization.

[0315] table_type: stores the table type, such as "fact table" or "dimension table", and the data type is STRING.

[0316] keywords: stores table index keywords. Index keywords can help locate this table during search or intent recognition. The data type is STRING.

[0317] proper_noun: stores the explanation information of proper nouns, and the data type is STRING.

[0318] downstream_used: stores downstream application information, the data type is STRING.

[0319] In addition, the storage table has the following features:

[0320] The storage is partitioned by the ds field, where ds is a STRING type field, usually representing a date. This partitioning strategy can improve the query efficiency of data within a specific date range.

[0321] The storage table is set with LIFECYCLE 90, which means that the data partitions of the storage table will automatically expire and be cleaned up 90 days after creation.

[0322] Optionally, a second storage table may be created according to the second data structure and based on the field description information. For example, in an actual application, the second storage table may be created according to the following SQL statement:

[0323] CREATE TABLE IF NOT EXISTS dwm_en_meta_column_detail_d(

[0324] keywords STRING COMMENT'Keywords, used for intent identification similarity query',

[0325] column_name STRING COMMENT'field name',

[0326] column_comment STRING COMMENT'field original comment',

[0327] column_enum STRING COMMENT'field enumeration value',

[0328] table_name STRING COMMENT'table name',

[0329] proper_noun STRING COMMENT 'Proper noun explanation of the field',

[0330] downstream_used STRING COMMENT 'Field downstream usage scenario (including downstream execution system, analysis report, indicator configuration, etc.)' )

[0332] COMMENT 'Field Details'

[0333] PARTITIONED BY (

[0335] ds STRING COMMENT 'Partition' )

[0337] LIFECYCLE 90;

[0338] This SQL statement can be used to create a second storage table named dwm_en_meta_column_detail_d, which is used to store field face information in the database. The second storage table may include the following fields:

[0339] keywords: stores field index keywords. Field index keywords can be used for intent recognition and similarity queries to help understand the content and purpose of the column.

[0340] column_name: stores the field name, the data type is STRING.

[0341] column_comment: stores field comment information, the data type is STRING.

[0342] column_enum: If the column field has enumeration values, store these enumeration values, that is, field data, and the data type is STRING.

[0343] table_name: The name of the table that stores the data to which this field belongs. The data type is STRING.

[0344] proper_noun: stores the proper nouns and explanation information related to the field, and the data type is STRING.

[0345] downstream_used: stores downstream application information, the data type is STRING.

[0346] The second storage table may have the following characteristics:

[0347] Partitioning is done by the ds field, where ds is a STRING type field that usually represents a date. This partitioning strategy can improve the efficiency of querying data within a specific date range. The storage table is set with LIFECYCLE 90, which means that the data partitions of the storage table will automatically expire and be cleaned up 90 days after creation.

[0348] Optionally, the third storage table and the fourth storage table may be created according to the third data structure based on the table association relationship information. For example, in an actual application, the third storage table may be created according to the following SQL statement:

[0349] CREATE TABLE IF NOT EXISTS dwm_en_meta_primary_foreign_key_d(

[0350] keywords STRING COMMENT'Keywords, used for intent identification similarity query',

[0351] foreign_key_col_name STRING COMMENT 'foreign key field',

[0352] foreign_key_table_name STRING COMMENT'foreign key physical table',

[0353] primary_key_col_name STRING COMMENT'Primary key field',

[0354] primary_key_table_name STRING COMMENT'Primary key physical table', )

[0356] COMMENT 'Primary and foreign key information'

[0357] PARTITIONED BY (

[0359] ds STRING COMMENT 'Partition' )

[0361] LIFECYCLE 90;

[0362] This SQL statement can be used to create a storage table named dwm_en_meta_primary_foreign_key_d, that is, a third storage table, for storing table associations. The fields involved in the third storage table may include:

[0363] keywords: stores relational index keywords, which can be keywords related to primary keys or foreign keys, used for intent identification and similarity queries. The data type is STRING.

[0364] foreign_key_col_name: The field name that stores the foreign key field. The data type is STRING.

[0365] foreign_key_table_name: The table name of the data table to which the foreign key field belongs is stored. The data type is STRING.

[0366] primary_key_col_name: The field name that stores the primary key field. The data type is STRING.

[0367] primary_key_table_name: The table name of the data table to which the primary key field belongs is stored. The data type is STRING.

[0368] The third storage table can also have the following characteristics: partitioned by the ds field, where ds is a STRING type field, usually representing a date. This partitioning strategy can improve the efficiency of querying data in a specific date range. The storage table is set with LIFECYCLE 90, which means that the data partition of the storage table will automatically expire and be cleaned up 90 days after creation.

[0369] For example, in actual applications, the fourth storage table can be created according to the following SQL statement:

[0370] CREATE TABLE IF NOT EXISTS dwm_en_meta_table_associate_d(

[0371] keywords STRING COMMENT'Keywords, used for intent identification similarity query',

[0372] table_name STRING COMMENT'table name',

[0373] associate_table_name STRING COMMENT'Association table name',

[0374] associate_detail STRING COMMENT 'Association conditions',

[0375] associate_cnt BIGINT COMMENT'Number of associations', )

[0377] The relationship between the COMMENT tables'

[0378] PARTITIONED BY (

[0380] ds STRING COMMENT 'Partition' )

[0382] LIFECYCLE 90;

[0383] The SQL statement can be used to create a storage table named dwm_en_meta_table_associate_d, which is the fourth storage table. The fields involved in the fourth storage table may include:

[0384] keywords: stores the relationship keywords related to the table association information, which is used for intent recognition and similarity query. The data type is STRING.

[0385] table_name: The name of the table that stores data. The data type is STRING.

[0386] associate_table_name: The table name that stores the associated data table. The data type is STRING.

[0387] associate_detail: stores association conditions, the data type is STRING.

[0388] associate_cnt: stores the number of associations, the data type is BIGINT.

[0389] The fourth storage table can also have the following characteristics: partitioned by the ds field, where ds is a STRING type field, usually representing a date. This partitioning strategy can improve the efficiency of querying data in a specific date range. The storage table is set with LIFECYCLE 90, which means that the data partition of the storage table will automatically expire and be cleaned up 90 days after creation.

[0390] Optionally, a fifth storage table may be created according to the fifth data structure based on the table blood relationship information. For example, in an actual application, the fifth storage table may be created according to the following SQL statement:

[0391] CREATE TABLE IF NOT EXISTS dwm_en_meta_table_relation_d(

[0392] keywords STRING COMMENT'Keywords, used for intent identification similarity query',

[0393] src_table_name STRING COMMENT 'Source table name',

[0394] dest_table_name STRING COMMENT 'target table name',

[0395] relation_type STRING COMMENT'Relation source type: 1: direct mapping 2: function conversion; 3. join', create_time STRING COMMENT'Relation creation time',

[0396] modify_time STRING COMMENT 'Relationship modification time' )

[0398] COMMENT 'Table granularity lineage'

[0399] PARTITIONED BY (

[0401] ds STRING COMMENT 'Partition' )

[0403] LIFECYCLE 90;

[0404] This SQL statement can be used to create a storage table named dwm_en_meta_table_relation_d, which is the fifth storage table, to store table blood relationship information. The fields involved may include:

[0405] keywords: stores the relationship keywords of the table's blood relationship, used for intent recognition and similarity query, and the data type is STRING.

[0406] src_table_name: The table name of the source data table, the data type is STRING.

[0407] dest_table_name: The table name of the target data table. The data type is STRING.

[0408] relation_type: stores the source type of the relation, such as direct mapping, function conversion, or JOIN operation. The data type is STRING.

[0409] create_time: stores the relationship creation time, the data type is STRING.

[0410] modify_time: stores the last modification time of the relationship, the data type is STRING.

[0411] The fifth storage table has the following characteristics: partitioning is performed according to the ds field, where ds is a STRING type field, usually representing a date. This partitioning strategy can improve the efficiency of querying data within a specific date range. The storage table is set with LIFECYCLE 90, which means that the data partition of the storage table will automatically expire and be cleaned up 90 days after creation.

[0412] Optionally, a sixth storage table may be created according to the sixth data structure based on the field blood relationship information. For example, in an actual application, the sixth storage table may be created according to the following SQL statement:

[0413] CREATE TABLE IF NOT EXISTS dwm_en_meta_column_relation_d(

[0414] keywords STRING COMMENT'Keywords, used for intent identification similarity query',

[0415] src_table_name STRING COMMENT 'Source table name',

[0416] src_column_name STRING COMMENT 'Source field name',

[0417] dest_table_name STRING COMMENT 'target table name',

[0418] dest_column_name STRING COMMENT 'Destination field name',

[0419] relation_type STRING COMMENT'Relation source type: 1: direct mapping 2: function conversion; 3. join', create_time STRING COMMENT'Relation creation time',

[0420] modify_time STRING COMMENT 'Relationship modification time' )

[0422] COMMENT'Model metadata--field granularity lineage'

[0423] PARTITIONED BY (

[0425] ds STRING COMMENT 'Partition' )

[0427] LIFECYCLE 90;

[0428] This SQL statement can be used to create a storage table named dwm_en_meta_column_relation_d, which is the sixth storage table, to store field blood relationship information. The fields involved may include:

[0429] keywords: The relationship keywords that store the field blood relationship information are used for intent identification and similarity query. The data type is STRING.

[0430] src_table_name: The name of the table that stores the source data. The data type is STRING. The source data table is also the starting data table in the lineage path.

[0431] src_column_name: The field name that stores the source field. The data type is STRING.

[0432] dest_table_name: The name of the target data table, data type is STRING. The target data table is also the final data table in the lineage path.

[0433] dest_column_name: The field name of the target field. The data type is STRING.

[0434] relation_type: stores the relation source type, the data type is STRING.

[0435] create_time: stores the relationship creation time, the data type is STRING.

[0436] modify_time: stores the last modification time of the relationship, the data type is STRING.

[0437] The sixth storage table has the following characteristics: partitioning is performed according to the ds field, where ds is a STRING type field, usually representing a date. This partitioning strategy can improve the efficiency of querying data within a specific date range. The storage table is set with LIFECYCLE 90, which means that the data partition of the storage table will automatically expire and be cleaned up 90 days after creation.

[0438] Optionally, a seventh storage table may be created according to the seventh data structure and based on the lineage path information. For example, in an actual application, the seventh storage table may be created according to the following SQL statement:

[0439] CREATE TABLE IF NOT EXISTS dwm_en_meta_column_path_d(

[0440] keywords STRING COMMENT'Keywords, used for intent identification similarity query',

[0441] src_table_name STRING COMMENT 'Source table name',

[0442] src_column_name STRING COMMENT 'Source field name',

[0443] dest_table_name STRING COMMENT 'target table name',

[0444] dest_column_name STRING COMMENT 'Destination field name',

[0445] full_id_path STRING COMMENT'full path',

[0446] full_id_path_dept BIGINT COMMENT 'Path depth' )

[0448] COMMENT 'field path'

[0449] PARTITIONED BY (

[0451] ds STRING COMMENT 'Partition' )

[0453] LIFECYCLE 90;

[0454] This SQL statement can be used to create a storage table named dwm_en_meta_column_path_d, which is the seventh storage table, to store lineage path information. The fields involved may include:

[0455] keywords: store relationship keywords of bloodline path information, used for intent recognition and similarity query, data type is STRING.

[0456] src_table_name: The table name of the source data table, the data type is STRING.

[0457] src_column_name: The field name that stores the source field. The data type is STRING.

[0458] dest_table_name: The table name of the target data table. The data type is STRING.

[0459] dest_column_name: The field name of the target field. The data type is STRING.

[0460] full_id_path: The complete lineage path of the field, which can include the table names of each data table involved in the path and the field names of each field. The data type is STRING.

[0461] full_id_path_dept: path depth, data type is BIGINT. Path depth refers to, for example, the number of fields involved.

[0462] The seventh storage table also has the following characteristics: partitioning is performed according to the ds field, where ds is a STRING type field, usually representing a date. This partitioning strategy can improve the efficiency of querying data within a specific date range. The storage table is set with LIFECYCLE 90, which means that the data partition of the storage table will automatically expire and be cleaned up 90 days after creation.

[0463] It should be noted that the above is only an example of possible storage methods for table description information, field description information, and table relationship information, which can be in the form of a table, or in any other form such as text in json (a file format and data exchange format). The vector features after vectorization can also be stored in different formats as needed, or can refer to the characteristics of different selected vector libraries. In addition, the storage method can also be determined in combination with the characteristics of the target large model. The present application is not limited to this. In addition, the object-related information extracted from the document data is stored in different formats as needed, such as in the form of a table, or any other form, or can refer to the characteristics of different selected vector libraries. In addition, the storage method can also be determined in combination with the characteristics of the target large model. The object-related information can also be converted into vector features before storage, and the present application does not limit this.

[0464] For ease of understanding, Figure 2 The overall flow chart of the knowledge base construction process in a practical application according to an embodiment of the present application is shown.

[0465] like Figure 2 As shown, when building an asset knowledge base, the database data and document data corresponding to the data provider can be determined first, and the database data can include data table data and metadata.

[0466] Afterwards, the operation of step 201 may be performed to perform quality assessment on the database data and the document data, so as to screen the database data and the document data that meet the quality requirements and obtain high-quality data assets.

[0467] In one embodiment of the present application, a data asset screening model constructed based on a large model can be used to perform quality assessment, that is, the acquired data can be input into the data asset screening model, and after screening, the data asset screening model outputs high-quality data assets that meet the quality requirements.

[0468] High-quality data assets can be divided into: data table data, table relationship information maintained in metadata, historical query statements, and document data.

[0469] In addition, data assets that do not meet quality requirements can be corrected through manual maintenance to obtain manually maintained data assets (including data table data and document data, etc.). In addition, table relationship information can be provided through manual configuration to obtain manually configured table relationship information.

[0470] Afterwards, for these high-quality data assets, the operation of step 202 can be performed to perform entity recognition and application domain mapping, the operation of step 203 can be performed to perform table annotation information normalization processing and table index keyword extraction and field annotation information normalization processing and field keyword extraction, and the operation of step 204 can be performed to perform table relationship information recognition, etc. In addition, for document data, step 205 can also be performed to perform object recognition, thereby identifying objects and object-related information.

[0471] Thus, after steps 202 to 205, the knowledge base contents such as table description information, field description information, table relationship information and object related information can be obtained.

[0472] These table description information, field description information, table relationship information and object-related information can also be output to relevant personnel for manual sampling review, update and confirmation.

[0473] Afterwards, in step 206, the table description information and the field description information can be used to construct an asset vector library, the table relationship information can be used to construct a relationship vector library, and the object-related information can be used to construct a knowledge vector library.

[0474] The asset vector library, relationship vector library and knowledge vector library can be used as an asset knowledge library to connect to the target large model.

[0475] Among them, the asset knowledge base can be connected to the target big model in multiple application scenarios. The technical solutions of the embodiments of the present application will be introduced from the application perspective of the asset knowledge base respectively below:

[0476] Figure 3 A flowchart of an embodiment of a data processing method provided by an embodiment of the present application is shown. The method may include the following steps:

[0477] 301: Determine the asset knowledge base.

[0478] 302: Obtain target knowledge information matching the input data from the asset knowledge base;

[0479] The target knowledge information may include target table description information, target field description information and / or target table relationship information.

[0480] In addition, when the asset knowledge base includes object-related information, the target knowledge may also include target object information and the like.

[0481] 303: Input the input data and target knowledge information into the target large model to obtain output data.

[0482] Optionally, the asset knowledge base can be a general database. In order to adapt to different target big models, the knowledge information obtained from the asset knowledge base can be converted into a corresponding data format for processing. In this embodiment, the target knowledge information can be converted into index and metadata form and input into the target big model.

[0483] The construction process of the asset knowledge base can be found in Figure 1 The above is described in the illustrated embodiment and will not be described in detail here.

[0484] In this embodiment, the asset knowledge base can be connected to the target big model to realize RAG (Retrieval-augmented Generation), and target knowledge information such as target table description information, target field description information and / or target table relationship information can be used as auxiliary information of the input data to guide the generation of output data of the target big model.

[0485] Among them, the target knowledge information matching the input data can be obtained through similarity calculation, and the target table description information, target field description information, target table relationship information and / or target object information whose similarity is greater than a specified threshold or whose similarity is the largest are selected. Among them, the similarity calculation can be achieved through calculation methods such as Euclidean distance, cosine similarity, etc., and this application does not limit this.

[0486] The input data and target knowledge information can generate a specific prompt word (Prompt) according to a specific prompt template, and input into the target macro model, so that the target macro model generates target data based on the specific prompt word.

[0487] In actual applications, input data can be, for example, a number search request in a number search scenario, a number usage request in a number usage scenario, or a query text in NL2SQL, etc. Of course, in scenarios such as question-and-answer systems and intelligent assistants, it can refer to any data input by the user, etc., and this application does not limit this.

[0488] In this embodiment, the target knowledge information obtained from the asset knowledge base is used to combine the input data with this information to generate a structured input set. The generated structured input set is input into the target big model, providing the target big model with necessary context information, so that the reasoning ability of the target big model can be used to generate corresponding output data, thereby improving the reasoning effect of the target big model and ensuring the accuracy and efficiency of data processing.

[0489] Figure 4 A flowchart of another embodiment of a data processing method provided by an embodiment of the present application is shown. The method may include the following steps:

[0490] 401: Determine the asset knowledge base;

[0491] 402: Construct training samples based on the table description information, field description information, and table relationship information in the asset knowledge base.

[0492] Optionally, when the asset knowledge base also includes object-related information, the training samples may be constructed based on the table description information, field description information, table relationship information and object-related information in the asset knowledge base.

[0493] 403: Use the training samples to train the target large model.

[0494] The construction process of the asset knowledge base can be found in Figure 1 The knowledge base construction method shown is not repeated here.

[0495] In this embodiment, the asset knowledge base is connected to the target large model, which can be used to construct fine-tuning training samples to fine-tune the target large model. The target large model can be a pre-trained large model, or it can be a large model that has been fine-tuned. The technical solution of the embodiment of the present application can be further fine-tuned on the basis of the previous fine-tuned large model to make the effect of the target large model better and better.

[0496] Since the asset knowledge base may contain data in specific fields, such as proper nouns, industry terms, abbreviations, etc., by obtaining table description information, field description information and table relationship information from the asset knowledge base, and using this to construct training samples, and using the training samples to further train and fine-tune the target big model, the target big model can learn the domain knowledge contained in the asset knowledge base and improve the reasoning ability and knowledge reserve of the target big model in specific fields.

[0497] In some embodiments, constructing training samples based on table description information, field description information, and table relationship information in the asset knowledge base may be specifically implemented as follows:

[0498] Using the fourth model, a training sample is constructed from table description information, field description information and table relationship information in the asset knowledge base based on at least one sample example; wherein the sample example includes sample input data and sample training labels.

[0499] That is, the training samples of the target large model can also be obtained from the asset knowledge base with the help of the large model capabilities. Sample examples can be used as context information for the fourth large model. Through in-context learning (ICL), the fourth large model can infer and generate input data and output data based on some sample examples. The input data is used as the input data in the training sample, and the output data is used as the training label in the training sample.

[0500] Optionally, corresponding prompt words may be generated based on at least one sample example to be input into the fourth model, so that the fourth model learns the pattern of generating training samples from these sample examples and constructs training samples from the asset knowledge base. For ease of understanding, for example, the prompt words may be:

[0501] {Please refer to the following example to generate training samples:

[0502] "prompt":"ggs Revenue->","completion":"\n\nGGS is the abbreviation of overseas gold members, representing merchants who have purchased membership product packages.\n\nRevenue includes membership fee revenue, marketing revenue and other value-added revenue.\n\nTherefore, GGS revenue refers to the membership fee revenue plus marketing revenue plus other value-added revenue of overseas merchants"}

[0503] In the above example, "prompt" is the sample input data, and "completion" is the sample training label. It should be noted that the above example is only to illustrate the possible organization of sample examples, and the present application is not limited thereto.

[0504] Among them, the fourth largest model may refer to a machine learning model with a large number of parameters and complex structure, which can process massive data and complete various complex tasks, such as natural language processing, computer vision, speech recognition, etc. It can be implemented using any currently open source large language model, and this application does not limit this.

[0505] Optionally, the knowledge information obtained from the asset knowledge base can be converted into a corresponding data format for processing. In this embodiment, the knowledge information in the asset knowledge base can be converted into a json format, for example, so that the fourth large model generates training samples in a json format, and the training samples can be used to fine-tune the target large model. In this embodiment, by utilizing the reasoning, data analysis and processing capabilities of the large model, high-quality table description information, field description information and table relationship information can be extracted from the complex data in the asset knowledge base and used to construct training samples. Training the target large model with high-quality training samples can improve the training quality.

[0506] Figure 5A flowchart of another embodiment of a data processing method provided by the present application is shown. The method may include the following steps:

[0507] 501: Use the target large model to process the query request;

[0508] 502: When the query request involves database data, query the target knowledge information from the asset knowledge base corresponding to the database data.

[0509] Among them, the target knowledge information may include target table description information, target field description information and / or target table relationship information; when the asset knowledge base includes object-related information, the target knowledge information may include target table description information, target field description information, target table relationship information and / or target object information.

[0510] 503: Generate query results based on target knowledge information;

[0511] 504: Use the query result as output data of the target large model.

[0512] The construction process of the asset knowledge base can be found in Figure 1 The knowledge base construction method shown is not repeated here.

[0513] The query request initiated by the user may need to retrieve and analyze specific data from the database. In this case, the large model can use the asset knowledge base. The table description information, field description information and / or table relationship information in the asset knowledge base can be considered as metadata generated by reorganizing the database data. By querying the asset knowledge base, the target table description information, target field description information and / or target table relationship information can be combined to perform data query again. The target table description information, target field description information and / or target table relationship information can be used as index information to reduce the query scope or as auxiliary information to improve the query accuracy, etc. Therefore, the technical solution of the embodiment of the present application can improve the query effect and ensure the query efficiency and / or accuracy, etc.

[0514] The target table description information, target field description information and / or target table relationship information etc. matching the query request may be found by similarity calculation.

[0515] Optionally, the knowledge information obtained from the asset knowledge base can be converted into a corresponding data format for processing. In this embodiment, the asset knowledge base can be converted into a key-value format for query processing by the target large model. In this embodiment, the target large model accesses the asset knowledge base, because the asset knowledge base stores the table description information, field description information and table relationship information between tables of all data tables in the database. This information is essential for understanding the database structure and data content. The target large model can retrieve relevant table description information, field description information and table relationship information between tables from the asset knowledge base according to the specific content of the query request. With these detailed table description information, field description information and table relationship information between tables, the target large model can use its own intelligent algorithm to analyze this information, identify the connection between the data, filter out the key data that meets the query request, and may further aggregate, sort or process the data in other forms to obtain the query result.

[0516] Figure 6 A schematic diagram of a structure of an embodiment of a knowledge base construction device provided in an embodiment of the present application, the device may include:

[0517] The data acquisition module 601 is used to acquire the database data and document data corresponding to the data provider; the database data includes the data table data and metadata in the database;

[0518] A description information generating module 602 is used to generate table description information of a data table in the database and field description information of the data table based on the database data and the document data;

[0519] A first information identification module 603, used to identify table relationship information corresponding to the data table from the historical query statements of the database data and the document data;

[0520] The knowledge base construction module 604 is used to construct an asset knowledge base based on the table description information, the field description information and the table relationship information.

[0521] In some embodiments, the description information generating module 602 includes:

[0522] A first quality assessment submodule, used to perform quality assessment on the database data and the document data to screen the database data and the document data that meet the data quality requirements;

[0523] The description information submodule is used to generate table description information of the data table in the database data and field description information of the data table for the database data and document data that meet the data quality requirements.

[0524] In some embodiments, the knowledge base construction device further includes:

[0525] A first maintenance prompt information output module, used for outputting data maintenance prompt information for database data that does not meet data quality requirements;

[0526] The data updating module is used to update the database data in response to the data adjustment request, and use the updated database data as the database data that meets the quality requirements.

[0527] In some embodiments, the knowledge base construction device further includes:

[0528] The second quality assessment submodule performs a quality assessment on the data model of the database data to determine whether the data model meets the model quality requirements.

[0529] A second maintenance prompt information output module is used to output model maintenance prompt information when the data model does not meet the model quality requirements;

[0530] The model adjustment module is used to update the data model in response to a model adjustment request when the data model meets the model quality requirements.

[0531] In some embodiments, the knowledge base construction device further includes:

[0532] The data maintenance module is used to determine manually configured table relationship information in response to a relationship maintenance request for the database data.

[0533] In some embodiments, the first information identification module 603 includes:

[0534] The first information identification submodule is used to identify the table relationship information corresponding to the data table from the historical query statements of the database data, the document data, the table relationship information maintained in the metadata, and the manually configured table relationship information.

[0535] In some embodiments, the description information generating module 602 includes:

[0536] A first normalization submodule, configured to perform normalization processing on the table annotation information of the data table, so as to use the normalized table annotation information as a kind of table description information;

[0537] A first keyword extraction submodule is used to extract table index keywords from the normalized table annotation information as a kind of table description information; the table index keywords are used to match the table description information;

[0538] A second normalization submodule is used to normalize the field annotation information of the data table so as to use the normalized field annotation information as a kind of field description information;

[0539] The second keyword extraction submodule is used to extract field index keywords from the field annotation information after normalization as a kind of field description information; the field index keywords are used to match the field description information;

[0540] An entity information extraction submodule, used to extract entity related information of entities associated with the data table from the document data;

[0541] A table description information determination submodule is used to use the entity related information as a kind of table description information of the data table.

[0542] In some embodiments, the knowledge base construction device further includes:

[0543] A first downstream application identification module, used to identify downstream application information of the data table from the database data, and use the downstream application information as a kind of table description information of the data table;

[0544] A second downstream application identification module, used to identify downstream application information of a field in the data table from the database data, and use the downstream application information as a field description information;

[0545] An entity recognition module, used to identify entities in a data table and application fields corresponding to the entities from the database data;

[0546] The merging and normalizing module is used to use the entity and the application field as a kind of table description information of the data table or merge the entity and the application field into the table annotation information after normalization processing.

[0547] In some embodiments, the first keyword extraction submodule includes:

[0548] A first keyword extraction unit, used for extracting table index keywords from the normalized table annotation information and field annotation information using the first large model as a kind of table description information;

[0549] In some embodiments, the second keyword extraction submodule includes:

[0550] The second keyword extraction unit is used to extract field index keywords from the field annotation information after normalization using the first large model as a kind of field description information.

[0551] In some embodiments, the first information identification module 603 includes:

[0552] A table relationship information extraction submodule, used for analyzing the historical query statements of the database data using the second largest model to extract the table relationship information corresponding to the data table;

[0553] A related information extraction submodule, used to extract entity related information of entities associated with a data table from the document data; the entity related information includes entity relationship information and entity operation information;

[0554] The table relationship information identification submodule is used to identify the table relationship information corresponding to the data table from the entity relationship information and the entity operation information.

[0555] In some embodiments, the description information generating module 602 includes:

[0556] The description information generation submodule is used to generate the table description information of the data table in the database data and the field description information of the data table based on the table annotation information of the data table in the database data, the fields and field annotation information contained in the data table, and the metadata corresponding to the data table, using the third model.

[0557] In some embodiments, the table relationship information includes table association relationship information, table lineage relationship information, field lineage relationship information, and lineage path information.

[0558] In some embodiments, the knowledge base construction device further includes:

[0559] An information extraction module is used to extract relationship index keywords corresponding to table association relationship information, table lineage relationship information, field lineage relationship information and lineage path information, respectively, and save them in the asset knowledge base in correspondence with the table association relationship information, table lineage relationship information, field lineage relationship information and lineage path information; the relationship index keywords are used for relationship identification.

[0560] In some embodiments, the knowledge base construction module 604 includes:

[0561] A first vector conversion submodule, configured to convert the table description information and the field description information into vector features to construct an asset vector library;

[0562] The second vector conversion submodule is used to convert the table relationship information into vector features to construct a relationship vector library.

[0563] In some embodiments, the table description information includes table name, table type, table index keyword, table annotation information and downstream application information of the data table;

[0564] The field description information includes the field name, the table name of the data table to which it belongs, field annotation information, field index keywords, explanation information of the proper nouns involved in the field, and field downstream application information;

[0565] The table relationship information includes table association relationship information, table lineage relationship information, field lineage relationship information and field lineage path information;

[0566] The table association relationship information includes table name, associated table name of the associated data table, foreign key field in the data table, primary key field in the associated data table, association condition, association times and relationship index keyword;

[0567] The table blood relationship information includes the table name of the source data table, the target table name of the target data table, the blood relationship type, the relationship creation time, the relationship modification time and the relationship index keyword;

[0568] The field lineage key information includes the table name of the source data table, the field name in the source data table, the target table name of the target data table, the target field name in the target data table, the lineage relationship type, the relationship creation time, the relationship modification time and the relationship index keyword;

[0569] The field lineage path information includes the table name of the source data table, the field name in the source data table, the table name of the belonging data table, the target table name of the target data table, the target field name in the target data table, the lineage path, the path depth and the path index keyword.

[0570] In some embodiments, the knowledge base building module 604 includes:

[0571] A data table generation submodule, used to generate a plurality of data tables according to the corresponding data structures, based on the table description information, the field description information and the table relationship information;

[0572] The first knowledge base construction submodule is used to construct an asset knowledge base based on the multiple data tables.

[0573] In some embodiments, the knowledge base construction device further includes:

[0574] A document identification module, used for identifying an object and object-related information of the object from the document data;

[0575] In some embodiments, the knowledge base building module 604 includes:

[0576] The second knowledge base construction submodule is used to construct an asset knowledge base based on the table description information, the field description information, the table relationship information and the object related information.

[0577] Figure 7 A schematic diagram of a data processing device according to an embodiment of the present application is provided. The device may include:

[0578] A first knowledge base determination module 701 is used to determine an asset knowledge base;

[0579] The second information identification module 702 is used to identify target knowledge information matching the input data from the asset knowledge base; the target knowledge information includes target table description information, target field description information and / or target table relationship information;

[0580] The data generation module 703 is used to input the target big model based on the input data and the target table description information, the target field description information and / or the target table relationship information to obtain output data.

[0581] Figure 8 A schematic diagram of a data processing device according to an embodiment of the present application is provided. The device may include:

[0582] The second knowledge base determination module 801 is used to determine the asset knowledge base;

[0583] A sample construction module 802 is used to construct a training sample based on the table description information, field description information and table relationship information in the asset database;

[0584] The model training module 803 is used to train the target large model using the training samples.

[0585] In some embodiments, the sample construction module 802 includes:

[0586] The sample construction submodule is used to construct training samples from the table description information, field description information and table relationship information in the asset database based on sample examples using the fourth model; the sample examples include sample input data and sample training labels.

[0587] Fig. 9 A schematic diagram of a data processing device according to an embodiment of the present application is provided. The device may include:

[0588] The request processing module 901 is used to process the query request using the target large model;

[0589] Query module 902, used for querying target knowledge information from an asset knowledge base corresponding to the database data when the query request involves database data; the target knowledge information includes table description information, target field description information and / or target table relationship information;

[0590] A result generation module 903 is used to generate a query result based on the target knowledge information;

[0591] The data output module 904 is used to use the query result as output data of the target large model.

[0592] In one possible design, the apparatus provided in the embodiments of the present application may be implemented as a computing device, such as Fig.10 As shown, the computing device may include a storage component 1001 and a processing component 1002;

[0593] The storage component 1001 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 1002 to implement the knowledge base construction method and data processing method provided in the embodiments of the present application.

[0594] Of course, the computing device may also include other components, such as input / output interfaces, communication components, etc. The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0595] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0596] When the computing device is a physical device, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device.

[0597] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the knowledge base construction method and data processing method provided in the embodiment of the present application.

[0598] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a computer, can implement the knowledge base construction method and data processing method provided in the embodiment of the present application.

[0599] The processing components in the above corresponding embodiments may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing components may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0600] The storage component is configured to store various types of data to support operations in the device. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0601] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0602] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0603] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0604] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for constructing a knowledge base, characterized in that: include: Obtaining database data and document data corresponding to the data provider; the database data includes data table data and metadata in the database; Generate table description information of a data table in the database and field description information of the data table based on the database data and the document data; Identifying table relationship information corresponding to the data table from historical query statements of the database data and the document data; An asset knowledge base is constructed based on the table description information, the field description information, and the table relationship information.

2. The method according to claim 1, characterized in that The step of generating a table description of a data table in the database data and field description information of the data table based on the database data and the document data includes: Performing quality assessment on the database data and the document data to select database data and document data that meet data quality requirements; For database data and document data that meet data quality requirements, table description information of data tables in the database data and field description information of the data tables are generated.

3. The method according to claim 2, characterized in that Also includes: Output data maintenance prompt information for database data that does not meet data quality requirements; In response to the data adjustment request, the database data is updated, and the updated database data is used as database data that meets the quality requirements.

4. The method according to claim 2, characterized in that: Also includes: Performing a quality assessment on the data model of the database data to determine whether the data model meets the model quality requirements; If not, output model maintenance prompt information; In response to the model adjustment request, the data model is updated.

5. The method according to claim 1, characterized in that Also includes: In response to a relationship maintenance request for the database data, determining manually configured table relationship information; The identifying, from the historical query statements of the database data and the document data, the table relationship information corresponding to the data table comprises: The table relationship information corresponding to the data table is identified from the historical query statements of the database data, the document data, the table relationship information maintained in the metadata, and the manually configured table relationship information.

6. The method according to claim 1, characterized in that The step of generating table description information of a data table in the database data and field description information of the data table based on the database data and the document data includes: Normalizing the table annotation information of the data table so as to use the normalized table annotation information as a kind of table description information; Extracting table index keywords from the normalized table annotation information as a kind of table description information; the table index keywords are used for matching the table description information; Normalizing the field annotation information of the data table so as to use the normalized field annotation information as a kind of field description information; Extracting field index keywords from the normalized field annotation information as a kind of field description information; the field index keywords are used to match the field description information; Extracting entity related information of entities associated with the data table from the document data; The entity related information is used as a kind of table description information of the data table.

7. The method according to claim 6, characterized in that Also includes: Identifying downstream application information of the data table from the database data, and using the downstream application information as a kind of table description information of the data table; Identify downstream application information of a field in the data table from the database data, and use the downstream application information as a field description information; From the database data, identifying entities in a data table and application fields corresponding to the entities; The entity and the application field are used as a kind of table description information of the data table, or the entity and the application field are merged into the table annotation information after normalization processing.

8. The method according to claim 6, characterized in that The extracting of table index keywords from the normalized table annotation information as a kind of table description information includes: Using the first model, extracting table index keywords from the normalized table annotation information and field annotation information as a kind of table description information; The extracting of field index keywords from the field annotation information after normalization as a kind of field description information includes: The first large model is used to extract field index keywords from the field annotation information after normalization as a kind of field description information.

9. The method according to claim 1, characterized in that: The identifying, from the historical query statements of the database data and the document data, the table relationship information corresponding to the data table comprises: Analyzing the historical query statements of the database data using the second largest model to extract table relationship information corresponding to the data table; Extracting entity related information of entities associated with data tables from the document data; the entity related information includes entity relationship information and entity operation information; From the entity relationship information and the entity operation information, table relationship information corresponding to the data table is identified.

10. The method according to claim 1, characterized in that The step of generating table description information of a data table in the database data and field description information of the data table based on the database data and the document data includes: Based on the table annotation information of the data table in the database data, the fields and field annotation information contained in the data table, and the metadata corresponding to the data table, the third model is used to generate the table description information of the data table in the database data and the field description information of the data table.

11. The method according to claim 1, characterized in that: The table relationship information includes table association relationship information, table lineage relationship information, field lineage relationship information and lineage path information; The method further comprises: Relationship index keywords corresponding to table association relationship information, table lineage relationship information, field lineage relationship information and lineage path information are extracted, and saved in the asset knowledge base in correspondence with the table association relationship information, table lineage relationship information, field lineage relationship information and lineage path information; the relationship index keywords are used for relationship identification.

12. The method according to claim 1, characterized in that The constructing of the asset knowledge base based on the table description information, the field description information and the table relationship information includes: Converting the table description information and the field description information into vector features to construct an asset vector library; The table relationship information is converted into vector features to construct a relationship vector library.

13. The method according to claim 1, characterized in that The table description information includes table name, table type, table index keywords, table annotation information and downstream application information of the data table; The field description information includes the field name, the table name of the data table to which it belongs, field annotation information, field index keywords, explanation information of the proper nouns involved in the field, and field downstream application information; The table relationship information includes table association relationship information, table lineage relationship information, field lineage relationship information and field lineage path information; The table association relationship information includes table name, associated table name of the associated data table, foreign key field in the data table, primary key field in the associated data table, association condition, association times and relationship index keyword; The table blood relationship information includes the table name of the source data table, the target table name of the target data table, the blood relationship type, the relationship creation time, the relationship modification time and the relationship index keyword; The field lineage key information includes the table name of the source data table, the field name in the source data table, the target table name of the target data table, the target field name in the target data table, the lineage relationship type, the relationship creation time, the relationship modification time and the relationship index keyword; The field lineage path information includes the table name of the source data table, the field name in the source data table, the table name of the belonging data table, the target table name of the target data table, the target field name in the target data table, the lineage path, the path depth and the path index keyword.

14. The method according to claim 13, characterized in that The constructing of the asset knowledge base based on the table description information, the field description information and the table relationship information includes: Generate multiple data tables according to the corresponding data structures based on the table description information, the field description information and the table relationship information; An asset knowledge base is constructed based on the multiple data tables.

15. The method according to claim 1, characterized in that Also includes: identifying an object and object-related information of the object from the document data; The constructing of the asset knowledge base based on the table description information, the field description information and the table relationship information includes: An asset knowledge base is constructed based on the table description information, the field description information, the table relationship information, and the object related information.

16. A data processing method, characterized in that: include: Identify the asset knowledge base; Identify target knowledge information matching the input data from the asset knowledge base; The target knowledge information includes target table description information, target field description information and / or target table relationship information; Based on the input data and the target table description information, the target field description information and / or the target table relationship information, a target big model is input to obtain output data.

17. A data processing method, characterized in that: include: Identify the asset knowledge base; Constructing training samples based on table description information, field description information, and table relationship information in the asset database; The target large model is trained using the training samples.

18. The method according to claim 16, characterized in that The constructing of training samples based on the table description information, field description information and table relationship information in the asset database includes: The fourth model is used to construct training samples based on sample examples from table description information, field description information and table relationship information in the asset database; the sample examples include sample input data and sample training labels.

19. A data processing method, characterized in that: include: Use the target large model to process query requests; In the case where the query request involves database data, querying target knowledge information from an asset knowledge base corresponding to the database data; The target knowledge information includes table description information, target field description information and / or target table relationship information; Based on the target knowledge information, generating a query result; The query result is used as output data of the target large model.

20. A computing device, characterized in that including a processing component and a storage component; The storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the knowledge base construction method as described in any one of claims 1 to 15, or to implement the data processing method as described in claim 16, or to implement the data processing method as described in any one of claims 17 to 18, or to implement the data processing method as described in claim 19.

21. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by the processing component, it implements the knowledge base construction method as described in claims 1 to 15, or implements the data processing method as described in claim 16, or implements the data processing method as described in any one of claims 17 to 18, or implements the data processing method as described in claim 19.

22. A computer program product, characterized in that It includes computer programs / instructions, which, when executed by the processing component, implement the knowledge base construction method as described in any one of claims 1 to 15, or implement the data processing method as described in claim 16, or implement the data processing method as described in any one of claims 17 to 18, or implement the data processing method as described in claim 19.

Images