Evaluation method of data elements in model application, related hardware and program product

Abstract

The invention provides an evaluation method of data elements in a model application, related hardware and a program product. The method comprises the following steps: acquiring metadata knowledge and content knowledge of target data elements from a metadata knowledge base and a content knowledge base set by a big language model hitching data platform, and storing the target data elements in the data platform. Setting the metadata knowledge and the content knowledge of the target data element as cue words to be imported into a large language model so as to execute the following steps through the large language model: generating a query language order based on the metadata knowledge of the target business document, and querying a data platform according to the query language order to obtain a query result; and determining a matching degree between the query result and the content knowledge of the target data element. And based on the matching degree between the query result and the content knowledge of the target data element and the cost of applying the target data element to the large language model, determining the efficiency of applying the target data element to the large language model.

Description

Technical Field

[0001] The present application relates to the field of data management technology, and in particular to an evaluation method for data elements in model applications, related hardware and program products. Background Art

[0002] With the development of artificial intelligence, large language models have become the mainstream human-computer interaction mode. Whether a large language model can accurately provide query results that meet user expectations depends largely on the quality of the attached data elements.

[0003] Therefore, how to evaluate the effectiveness of data elements in model applications as a basis for optimizing low-quality data elements is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of this application is to provide a method for evaluating data elements in model applications, related hardware and program products, which can reasonably evaluate the effectiveness of data elements in large language model applications.

[0005] In order to achieve the above purpose, the embodiment of the present application is implemented as follows: In a first aspect, a method for evaluating data elements in a model application is provided, comprising: Obtaining metadata knowledge of a target data element from a metadata knowledge base set up on a data platform attached to a large language model, and obtaining content knowledge of the target data element from a content knowledge base set up on the data platform attached to the large language model; wherein the metadata knowledge base stores metadata knowledge of multiple data elements of the data platform, and the content knowledge base stores content knowledge of the multiple data elements; The metadata knowledge and content knowledge of the target data element are set as prompt words to be imported into the large language model, so as to execute through the large language model: generating a query command based on the metadata knowledge of the target business document, and querying the data platform according to the query command to obtain a query result; determining the matching degree between the query result and the content knowledge of the target data element; Based on the matching degree between the query result and the content knowledge of the target data element and the cost of applying the target data element to the large language model, the effectiveness of applying the target data element to the large language model is determined.

[0006] In a second aspect, a device for evaluating data elements in a model application is provided, comprising: An acquisition module, used to acquire metadata knowledge of a target data element from a metadata knowledge base set up by a large language model-attached data platform, and to acquire content knowledge of the target data element from a content knowledge base set up by the large language model-attached data platform; wherein the metadata knowledge base stores metadata knowledge of multiple data elements of the data platform, and the content knowledge base stores content knowledge of the multiple data elements; A query module, used to set the metadata knowledge and content knowledge of the target data element as prompt words to import into the large language model, so as to execute through the large language model: generating a query command based on the metadata knowledge of the target business document, and querying the data platform according to the query command to obtain a query result; determining the matching degree between the query result and the content knowledge of the target data element; A calculation module is used to determine the effectiveness of applying the target data element to the large language model based on the matching degree between the query result and the content knowledge of the target data element and the cost of applying the target data element to the large language model.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to execute the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute the method described in the first aspect.

[0009] The embodiment of the present application configures a corresponding metadata knowledge base and content knowledge base for the data platform attached to the large language model. Among them, the metadata knowledge base stores the metadata knowledge of each data element of the data platform; the content knowledge base stores the content knowledge of each data element of the data platform. When it is necessary to evaluate the effectiveness of the target data element applied to the large language model, the metadata knowledge and content knowledge of the target data element are set as prompt words to be imported into the large language model to guide the large language model to generate query commands according to the metadata knowledge, simulate a query task to query the data platform, and determine the match between the query result and the content knowledge of the target data element. It should be understood that a high match indicates that the target data element can be applied by the large language model to answer user questions. Correspondingly, according to the match between the query result and the content knowledge of the target data element, and the cost of the target data element applied to the large language model, the effectiveness of the target data element applied to the large language model can be effectively calculated, thereby reflecting the importance of the target data element to the large language model in the dimension of application effectiveness, and then providing a basis for optimizing the data elements attached to the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 This is a first flow chart of the method for evaluating data elements in model application according to an embodiment of the present application.

[0012] Figure 2 A schematic diagram of the application architecture of the method for evaluating data elements in a model application according to an embodiment of the present application.

[0013] Figure 3 A second flow chart of the method for evaluating data elements in model applications according to an embodiment of the present application.

[0014] Figure 4 A third flow chart of the method for evaluating data elements in model application according to an embodiment of the present application.

[0015] Figure 5 This is a fourth flow chart of the method for evaluating data elements in model applications according to an embodiment of the present application.

[0016] Figure 6 A schematic diagram of the structure of an evaluation device for data elements in a model application according to an embodiment of the present application.

[0017] Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] As mentioned above, whether a large language model can provide query results that meet user expectations depends largely on the quality of the attached data elements. Therefore, in the application of large language models, optimizing low-quality data elements is one of the important means to improve model performance.

[0019] Currently, the quality of data elements is measured by their information value, but large language models usually have the ability to answer everything. Measuring the quality by the information value of data elements cannot accurately reflect the importance of data elements to the application of large language models.

[0020] To this end, this application aims to propose an evaluation scheme for data elements in model applications, which can reasonably evaluate the effectiveness of data elements in large language model applications, thereby reflecting the quality of data elements in the dimension of application effectiveness.

[0021] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0022] An embodiment of the present application proposes a method for evaluating data elements in a model application. Figure 1 The following is a flow chart of the evaluation method, which includes the following steps: S102, obtaining metadata knowledge of the target data element from the metadata knowledge base set up on the large language model-attached data platform, and obtaining content knowledge of the target data element from the content knowledge base set up on the large language model-attached data platform; wherein the metadata knowledge base stores metadata knowledge of multiple data elements of the data platform, and the content knowledge base stores content knowledge of multiple data elements.

[0023] In this embodiment, the data platform stores multiple data elements, which can be extracted from existing knowledge documents. In the human-computer exchange scenario, after the large language model is connected to the data platform, data elements can be selected from the data platform as the reply content of the user's question. Therefore, the data elements are expressed in natural language descriptions so that they can be understood by users.

[0024] For a large language model, if you want to understand the semantics of a data element, you need to configure the corresponding machine language representation results. That is, you need to encode the content information of the data element based on semantics to obtain the corresponding semantic vector. The large language model understands the semantics of the data element through the semantic vector.

[0025] In this embodiment, the semantic vector is defined as content knowledge, and a content knowledge base is used to store the content knowledge of each data element in the data platform. That is, the big language model maps the data elements corresponding to the data platform through the content knowledge (semantic vector) in the content knowledge base, and understands the semantics of the corresponding data elements.

[0026] On the basis of the above, in order to reflect the effectiveness of data elements in the application of large language models, it is necessary to determine whether the data elements are effectively retrieved by the large language model. Here, this embodiment proposes a concept of metadata knowledge, which is a semantic vector obtained by semantically encoding the metadata information of the data requirement on the data platform.

[0027] The metadata information may include, but is not limited to, directories, descriptions, fields, shards, data query logs, and data lineage relationships. That is, the metadata knowledge of a data element is information that can be understood by the large language model and is used to retrieve information related to the data element. Similarly, this embodiment stores the metadata knowledge of each data element in the data platform through a metadata knowledge.

[0028] In practical applications, this embodiment can use an embedding model to implement semantic encoding of metadata information and content information (also called embedded encoding). It should be noted that the data types of knowledge documents are different, so the corresponding content information can be divided into text content information, image content information, and table content information, etc. Correspondingly, the content information can be extracted in a manner suitable for its data type for embedded encoding. For example: For table-type data elements, a deep learning model can be used to identify the table format, table text content, and the mapping relationship between table key-value pairs, and the above content can be converted into text content information. For picture-type data elements, a large language model can be called to identify key information and convert it into text content information.

[0029] Among them, the shards corresponding to the target data elements are obtained by dividing multiple data elements of the data platform based on at least one of the data table, database and data warehouse subject domains as dimensions; or, the shards corresponding to the target data elements are determined based on the classification results of the data table to which the target data elements belong.

[0030] Specifically, a clustering method or a model prediction method may be used to determine the classification of the data table.

[0031] Taking clustering as an example: This embodiment can cluster data tables based on the query relationship between multiple data tables of the data platform to determine the classification results of each data table. Graph clustering is used here to illustrate that this embodiment can construct a data table query relationship graph, in which the nodes in the data table query relationship graph represent data tables, and the edges connecting the nodes represent the query relationship between data tables. The weight of the edge can be the query frequency determined from the data query log. Through a clustering algorithm (such as K-means clustering algorithm, hierarchical clustering algorithm), the nodes of the data table query relationship graph are clustered with query frequency as the dimension to determine the classification results of each data table.

[0032] Taking model prediction as an example: In this embodiment, the classification result of the data table to which the target data element belongs can be obtained by inputting the query relationship features between the data table to which the target data element belongs and other data tables into the deep learning model and predicting it by the deep learning model. The deep learning model can train the prediction ability of the classification result based on the query relationship features in the sample data table and the shard labels annotated in the sample data table.

[0033] In addition, the present embodiment may also appropriately expand the metadata information of the data element. For example, the fields in the metadata information include the original fields of the target data element and the extended fields expanded on the basis of the original fields. As an exemplary introduction, the extended fields may include: 1) The statistical field of the specified calculation method is obtained by calculating the amount information in the original field according to the specified calculation method; For example, if the original field includes an amount field, it can be expanded to obtain a total amount statistics field, an average amount statistics field, and the like.

[0034] 2) A specified enumeration type statistical field that performs statistics on the amount information in the original field according to the specified enumeration type; For example, the original field includes a transaction amount field, which can be expanded to obtain a credit card transaction amount statistics field, an APP payment transaction amount statistics field, and a cash transaction amount statistics field, etc.

[0035] 3) The specified unit time series statistics field is calculated based on the specified unit time series window for the time series information in the original field.

[0036] For example, the original field includes a transaction amount field, which can be expanded to obtain a daily transaction amount statistics field and a monthly transaction amount statistics field.

[0037] 4) An information quantity statistics field for counting the quantity of information in the original field.

[0038] For example, the original field includes the signal card number, and then the total number of signal card numbers can be expanded to obtain a statistical field.

[0039] Here we take the original metadata information shown in the following table as an example:

[0040] The metadata information that can be expanded can eventually include: Total number of people: The total number of rows recorded in the table.

[0041] Number of Males: The number of records marked as Male in the Gender field.

[0042] Number of Females: The number of records marked as Female in the Gender field.

[0043] Number of people in a specific age group: Filter the number of people in a specific age group (such as 20-30 years old) based on the age field.

[0044] Distribution by age group: Statistics of the number of people in different age groups (such as 10-year-old groups) in the age field.

[0045] Population distribution by province / city / district: Population statistics by province, city, and district.

[0046] Average Age: The average value of the age field for all records.

[0047] Median Age: The median of the age field for all records.

[0048] Age Standard Deviation: The standard deviation of the age field for all records.

[0049] Number of people with the highest educational level: Statistics of the number of people with different educational levels (such as bachelor's degree, master's degree) in the educational level field.

[0050] Number of people with the lowest education level: Statistics of the number of people with the lowest education level (such as primary school) in the Education level field.

[0051] Annual income distribution: statistics of the number of people with different income ranges in the annual income field (such as less than 100,000, 100,000-200,000, etc.).

[0052] Average Annual Income: The average value of the annual income field for all records.

[0053] Median Annual Income: The median of the annual income field for all records.

[0054] Number of people with the highest / lowest annual income: The number of records with the highest and lowest annual income.

[0055] Phone Number Format: Format consistency check for phone number fields.

[0056] Gender distribution of specific educational attainment: Cross-analysis of educational attainment and gender fields.

[0057] Age distribution at a specific income level: Cross-analysis of annual income and age fields.

[0058] Highest / lowest income earners in a specific area: cross-analysis of the province, city, and district of residence fields and the annual income field.

[0059] Population density: The population of a province / city / region divided by the area of ​​the region.

[0060] Annual income distribution by educational background: cross-analysis of educational background and annual income fields.

[0061] Distribution of annual income by gender: Cross-analysis of gender and annual income fields.

[0062] Distribution of annual income by age: Cross analysis of age and annual income fields.

[0063] Educational background distribution by place of residence: cross-analysis of the province, city, district of residence fields and the educational background field.

[0064] That is, this embodiment can use the semantic vector of "original metadata information+extended metadata information" as metadata knowledge to store it in the metadata knowledge base.

[0065] Based on the above-mentioned content knowledge base and metadata knowledge base, this embodiment defines the target data element as the data element whose effect is to be evaluated in the data platform. Correspondingly, it is necessary to obtain the content knowledge and metadata knowledge of the target data element from the metadata knowledge base and the content knowledge base, so as to perform a reverse query on the data platform according to the metadata knowledge through the large language model, and analyze the query value of the target data element based on the matching degree between the query result of the reverse query and the content knowledge of the target data element, so as to use the query value of the target data element as the basis for evaluating its effectiveness in the application of the large language model.

[0066] S104, setting the metadata knowledge and content knowledge of the target data element as prompt words to be imported into the large language model, so as to execute through the large language model: generating query commands based on the metadata knowledge of the target business document, and querying the data platform according to the query commands to obtain query results; determining the matching degree between the query results and the content knowledge of the target data element.

[0067] Prompt is an important configuration of the large language model, which is used to guide the large language model to complete the specified task.

[0068] This embodiment sets the metadata knowledge and content knowledge of the target data element as prompt words in order to guide the large language model to generate query commands, such as Structured Query Language (SQL) commands, based on the metadata knowledge of the target business document, so as to perform a reverse query on the data platform according to the query commands and determine the matching degree between the query results and the content knowledge of the target data element (i.e., the similarity distance of the semantic vector).

[0069] It should be understood that if the matching degree is low, it reflects that there is a large gap between the content knowledge of the target data element and the actual semantic content, and its performance in large language model applications is subsequently poor.

[0070] On the basis of the above, in order to simulate a more realistic query example, the present embodiment can also add the prompt words to the query description information for querying the target data element constructed based on the Cot thinking chain. Among them, the query description information of the Cot thinking chain can guide the large language model to try to query the content matching the target data element through thinking with logical relationships.

[0071] For ease of understanding, here is a simple example: Assuming that the target data element is "120,000 5G base stations are configured in North China", the query description information of the Cot thinking chain can be: Currently, Beijing, Tianjin, Hebei, Shanxi, and Inner Mongolia are equipped with a total of multiple 5G base stations. That is, the North China region is composed of Beijing, Tianjin, Hebei, Shanxi, and Inner Mongolia. By asking side questions, the large language model is guided to query the target data elements. Obviously, if the query result is inconsistent with 120,000, it means that the knowledge of the target data element attached to the large language model has application defects (it cannot correctly answer the query), and the corresponding performance should be at a low level.

[0072] Based on the inquiry description information of the Cot thinking chain, this embodiment can also introduce some complete question and answer examples containing inquiry description information and correct answer information in the prompt words, so as to serve as a reference for the large language model. It should be noted that introducing examples in the prompt words belongs to the prior art and will not be described in detail in this article.

[0073] In addition, in order to more accurately evaluate the effectiveness of the target data elements, the present embodiment may also add supplementary data elements that meet preset similarity requirements with the target data elements in the prompt word data platform, for example, selecting the top N data elements with similarity to the target data elements as supplementary data elements; correspondingly, under the guidance of the prompt words, the large language model also determines the matching degree between the query results and the content knowledge of the supplementary data elements as a basis for measuring the application effectiveness of the target data elements for the large language model.

[0074] S106, based on the matching degree between the query result and the content knowledge of the target data element, and the cost of applying the target data element to the large language model, determine the effectiveness of applying the target data element to the large language model.

[0075] The effectiveness is used to measure the proportional relationship between benefits and costs. If the matching degree between the query result and the content knowledge of the target data element is taken as the benefit, the cost of applying the target data element to the large language model needs to be determined when calculating the effectiveness.

[0076] As a feasible solution, the cost of applying the target data element to the large language model is determined based on the storage cost and computing cost of the target data element on the data platform.

[0077] The storage cost of the target data element in the data platform is determined based on the table cost of the data table to which the target data element belongs and the number of data elements; the table cost of any target data table is determined based on at least one of the following: 1) Storage cost of the target data table; 2) The cost of generating the target data table; 3) The storage cost of the parent table allocated to the target data table (ignore this item if there is no parent table); 4) The generation cost of the parent table allocated to the target data table (this item is ignored if there is no parent table).

[0078] That is, in addition to the storage cost of the target data element itself, this embodiment also takes into account the table cost of the parent table on which the target data element depends. The table cost of the parent table is shared by all data elements in the data table to which the target data element belongs.

[0079] When calculating the specific performance, a benefit score can be quantified based on the matching degree between the query result and the content knowledge of the target data element, and the matching degree between the query result and the content knowledge of the supplementary data element (if there is a supplementary data element). That is, the higher the matching degree, the higher the corresponding benefit score. By calculating the benefit score and the cost of applying the target data element to the large language model, the performance of the target data element applied to the large language model can be determined.

[0080] In summary, the method of this embodiment configures a corresponding metadata knowledge base and content knowledge base for the data platform attached to the large language model. Among them, the metadata knowledge base stores the metadata knowledge of each data element of the data platform; the content knowledge base stores the content knowledge of each data element of the data platform. When it is necessary to evaluate the effectiveness of the target data element applied to the large language model, the metadata knowledge and content knowledge of the target data element are set as prompt words to be imported into the large language model to guide the large language model to generate query commands according to the metadata knowledge, simulate a query task to query the data platform, and determine the match between the query result and the content knowledge of the target data element. It should be understood that a high match indicates that the target data element can be applied by the large language model to answer user questions. Correspondingly, according to the match between the query result and the content knowledge of the target data element, and the cost of the target data element applied to the large language model, the effectiveness of the target data element applied to the large language model can be effectively calculated, so as to reflect the importance of the target data element to the large language model in the dimension of application effectiveness, and then provide a basis for optimizing the data elements attached to the large language model.

[0081] The application of the method in this embodiment is described in detail below.

[0082] Figure 2 It is the application architecture of the method of this embodiment, including: content information extraction module, parsing module, data point extraction module, metadata information extraction module, metadata information enhancement module, model management module, knowledge base management module, and performance evaluation module.

[0083] 1. Content information extraction module: This module connects to the data platform, periodically pulls data elements from it, and uses a deep learning model to perform layout analysis to identify the text, table and picture parts. The picture part only identifies data-related pictures.

[0084] 2. Parsing module: parse text, tables and pictures separately.

[0085] (1) The table content uses a deep learning model to identify the table format, table text content, and the mapping relationship between table key-value pairs, and converts the above content into text content.

[0086] (2) The image content uses a large language model to identify key information and convert it into text content.

[0087] 3. Data point extraction module: Use the large language model to filter the data point related content in the text generated in 2.

[0088] It should be noted that the data points described in this article refer to data further extracted from the data elements of the data platform. Data points can be understood as data elements with smaller granularity. Therefore, the data points described below are all regarded as data elements.

[0089] 4. Metadata information extraction module: This module connects to the data platform and periodically pulls metadata information, including dataset name, description, data table name, data definition statement, description information, data field name, description, data lineage, data query log, etc.

[0090] 5. Metadata information enhancement module: (1) Rule-based metadata information enhancement: Generate metadata for data points that may be calculated based on the data sets, data tables, fields, and other information pulled from the metadata information extraction module. For example: ① Rule 1: If there is an amount field in the table, generate data points based on different calculation methods such as total amount and average amount ② Rule 2: If there are date fields and amount fields in the table, generate data points such as daily amount, monthly amount, quarterly amount, and annual amount.

[0091] ③Rule 3: If there is a varchar type field in the table, a data point of field name + "quantity" is automatically generated, such as the number of transaction types, the number of card numbers, etc.

[0092] ④ Rule 4: Arrange and combine the enumeration value text type fields and decimal and other numeric type fields in the data table to generate all possible data point names. For example, if the transaction type is an enumeration value (credit card, WeChat payment, cash), then generate the credit card transaction amount, WeChat payment transaction amount, and cash transaction amount.

[0093] (2) Metadata information enhancement based on large language models.

[0094] (3) The enhanced metadata is stored in the form of structured information + data points. The structured information includes database connection information, directly pulled metadata, etc., and the data points include data points generated by the above two methods.

[0095] 6. Model management module: mainly responsible for model registration, switching, and operation. It covers three types of models: text embedding, reordering, and large language models.

[0096] (1) Embedding model, which is responsible for embedding and encoding the content information output by the data point extraction module and the metadata information output by the element information enhancement module to obtain content knowledge and metadata knowledge. (2) The re-ranking model is used to retrieve supplementary data elements. That is, the similarity between other data elements in the data platform and the target data elements is calculated and sorted according to the similarity to find the supplementary data elements.

[0097] (3) The large language model is divided into four parts, including the image content recognition optimization model, the data point enhancement optimization model, the data point matching optimization model, and the SQL generation optimization model, which are respectively used in the parsing module (2), the metadata information enhancement module (2), and the performance evaluation modules (3) and (4) below.

[0098] 7. Knowledge base management module (1) Content knowledge base, which receives and stores the content knowledge output by the embedding model.

[0099] (2) Metadata knowledge base, which receives and stores metadata knowledge output by the embedding model. In addition, metadata can be filled in according to the source dataset of the data point. The metadata includes the institution to which the dataset belongs, the institution level, the dataset type, the dataset level, the dataset name, the dataset size, the dataset connection information, etc.

[0100] 8. Performance evaluation module (1) Knowledge acquisition unit: selects the target data elements to be matched. The target data elements can be the newly added data elements of the data platform, that is, the data elements of the newly added knowledge of the content knowledge base and metadata knowledge base. In addition, supplementary data elements that match the target data elements can also be selected according to the re-ranking model.

[0101] (2) The prompt word generation unit generates prompt words for the large language model based on the target data elements (including metadata knowledge and content knowledge) and supplementary data elements (including content knowledge) selected by the knowledge acquisition unit, combined with the query description information and complete question and answer examples constructed by the Cot thinking chain introduced in the previous article for querying the target data elements.

[0102] (3) The large language model reasoning unit inputs the prompt word provided by the prompt word generation unit, calls the data point matching optimization large model for query through a unified reasoning interface, and determines the matching degree between the query result and the content knowledge of the target data element, and obtains three types: complete match, possible match, and mismatch. If the output is a complete match or a possible match, the matched knowledge fragment link and the metadata of the knowledge fragment are also output.

[0103] (4) A model result parsing unit, which takes as input the matching results and related metadata information outputted from the model management module (3), parses the database connection information and field information therein according to the metadata information, and again calls the SQL generated by the model management module (3) to generate the optimized large language model to generate data query SQL.

[0104] (5) Data query unit, which takes as input the SQL generated in the performance evaluation module (4), connects to the database, performs data query, and outputs the data query results.

[0105] (6) Cost query unit, ① The cost CT of each data table itself consists of storage cost S and generation cost G. S consists of storage bytes Sv and storage byte unit price Ps, and generation cost G consists of generation task calculation bytes Gv and calculation byte unit price Pg.

[0106] ② Each data table needs to share the cost CTi of parent table i. If the parent table has N child tables, each child table needs to share the parent table cost CTi / N. If there are m parent tables with costs CT1..CTm, and one child table is generated, then the child table needs to share all the parent table costs. .

[0107] ③The cost C of each data point is composed of the table cost CT of the data table where it is located plus the data point calculation cost Cg.

[0108] ④When a data table has o data points, multiple data points can share the cost CT of the same data table, and the cost of each data point is CT / o + Cg.

[0109] ⑤ With the metadata in the performance evaluation module (3) as input, the basic storage space occupancy is obtained from the metadata, the number of calculation bytes is generated according to the SQL in the performance evaluation module (5), the storage space occupancy of each parent node is queried according to the data lineage relationship in the metadata, the number of calculation bytes of each task is queried according to the task log generated by each level of data, and finally the cost of the data point is generated.

[0110]

[0111] (7) Performance calculation unit, the input is the matching result in 8-(3), ① If it is a complete match, the basic score is x, if it is a partial match, the basic score is y, if it is no match, the basic score is 0. ② If it is a complete match, the matching coefficient is 1. If it is a partial match, a second judgment is required in combination with the SQL query result in the performance evaluation module (4), and the matching degree r is generated based on the matching degree between the number in the query result and the number in the business document data point.

[0112] ③ Evaluate importance based on application document metadata, including document citation count, data point repetition count, reading count, classification, and grading.

[0113] ④ Obtain the data cost C in the performance evaluation module (6). Calculate the effect score of the data. The formula is: in is a function that is expressed by the weight coefficient ( ) to standardize the scores. Cmax is the total cost of the data, which is used to standardize the cost C.

[0114]

[0115] based on Figure 2 As shown in the application architecture, the method of this embodiment is mainly divided into: an application document knowledge base construction process, a metadata knowledge base construction process and a data efficiency evaluation process.

[0116] Figure 3 A diagram of the process of building a knowledge base for application documentation, including: A1, call the interface to obtain data elements from the data platform.

[0117] A2, parse the content information of the data elements.

[0118] In this step, the text part, table part and picture part are extracted respectively through the deep learning model.

[0119] The text part generates structured text by performing text recognition; the table part generates structured text by performing table recognition, table key-value relationship recognition, and table content recognition; the image part uses a large language model optimized for data table and chart recognition to convert image information into text information describing the data. The training data is generated using a commercial large language model, covering a variety of task types such as chart extraction, chart description, text to chart, chart to chart, etc., as well as a variety of chart types, including pie charts, line charts, funnel charts, Gantt charts, etc. Based on these data, the open source model is fine-tuned to make the model have better performance than other models in chart interpretation.

[0120] A3, calling the embedding model to embed the content information of the parsed data elements.

[0121] The embedding model can be fine-tuned based on the pre-training. The fine-tuning training can be generated in the following semi-automatic way: ① Provide background information such as the name, description, and column definition of the format of the file or data table containing the data.

[0122] ② Use role-playing to let the model play different roles (such as government officials, data analysts, traders, machine learning experts, etc.) to generate problems.

[0123] ③ Use prompt templates and examples to guide question generation and ensure the diversity of questions.

[0124] ④ Adjust the number of relevant documents or data tables required for each question.

[0125] The specific fine-tuning method includes two steps. The first step is to initialize the basic embedding model using "new word embedding". This method can effectively expand the vocabulary of the model while avoiding the problem of drastic changes in probability distribution after the vocabulary is expanded. The second step is to fine-tune the model using the dataset we generated.

[0126] A4, add the content knowledge generated by embedded coding to the content knowledge base.

[0127] Among them, the content knowledge in the content knowledge base is mapped and aligned with the data elements of the data platform.

[0128] Figure 4 A schematic diagram of the process of building a metadata repository, including: B1, metadata information of data elements obtained by calling the interface data platform.

[0129] Metadata information includes dataset name, description, data table name, data definition statement, description information, data field name, description, data lineage, data query log, etc.

[0130] B2, parsing metadata information.

[0131] This step parses metadata information through rules. For specific methods, refer to the function (1) of the metadata information enhancement module. In addition, metadata information points are enriched through a large language model. For specific methods, refer to the function (2) of the metadata information enhancement module.

[0132] B3, calling this embedding model to embed and encode the parsed metadata information to obtain metadata knowledge.

[0133] B5, storing the metadata knowledge in the metadata knowledge base.

[0134] Among them, the metadata knowledge in the metadata knowledge base is mapped and aligned with the corresponding data elements, thereby forming a triple relationship structure of data element-metadata knowledge-content knowledge.

[0135] Figure 5 The following is a schematic diagram of the data performance evaluation process, including: C1, after receiving the performance evaluation request for the target data element, obtains relevant metadata knowledge and content knowledge from the metadata knowledge base and the content knowledge base.

[0136] Among them, the performance evaluation request of the target data element can be automatically initiated when the metadata knowledge of the target data element is added to the metadata knowledge base, or automatically initiated when the content knowledge of the target data element is added to the content knowledge base, or manually initiated.

[0137] C2, retrieves the supplementary data elements corresponding to the target data elements.

[0138] Here, the content knowledge similarity between the target data element and other data elements can be calculated, and other data elements whose similarity reaches a certain standard, such as other data elements in the TopN, can be determined as supplementary data elements.

[0139] C3, fill in the prompt word template based on the target data and the corresponding supplementary data elements to generate the prompt word.

[0140] C4, sends the generated prompt words to the large language model for query, and generates the matching degree between the query results and the content knowledge of the target data elements.

[0141] Among them, the matching degree can be divided into three types: complete match, no match, and partial match.

[0142] It should be noted that C3 and C4 can be executed multiple times according to the matching results. For example, if it is determined that there is no match or a partial match, the prompt word can be readjusted and the query can be performed again to ensure the rigor of the result.

[0143] C5, calculates the cost of the target data element.

[0144] The cost of the target data element includes the storage cost and the calculation cost of the target data element on the data platform. The corresponding calculation method is described above.

[0145] C6, based on the matching degree between the query result and the content knowledge of the target data element, and the cost of applying the target data element to the large language model, calculate the effectiveness of applying the target data element to the large language model.

[0146] The specific performance calculation method is described in the previous article.

[0147] In addition, corresponding to Figure 1 The method shown, another embodiment of this embodiment also provides an evaluation device for data elements in model applications. Figure 6 It is a schematic diagram of the structure of the evaluation device, including: The acquisition module 610 is used to acquire metadata knowledge of the target data element from the metadata knowledge base set by the large language model-attached data platform, and to acquire content knowledge of the target data element from the content knowledge base set by the large language model-attached data platform; wherein the metadata knowledge base stores metadata knowledge of multiple data elements of the data platform, and the content knowledge base stores content knowledge of the multiple data elements; The query module 620 is used to set the metadata knowledge and content knowledge of the target data element as prompt words to import into the large language model, so as to execute through the large language model: generating a query command based on the metadata knowledge of the target business document, and querying the data platform according to the query command to obtain a query result; determining the matching degree between the query result and the content knowledge of the target data element; The calculation module 630 is used to determine the effectiveness of applying the target data element to the large language model based on the matching degree between the query result and the content knowledge of the target data element and the cost of applying the target data element to the large language model.

[0148] The device of this embodiment configures a corresponding metadata knowledge base and content knowledge base for the data platform attached to the large language model. Among them, the metadata knowledge base stores the metadata knowledge of each data element of the data platform; the content knowledge base stores the content knowledge of each data element of the data platform. When it is necessary to evaluate the effectiveness of the target data element applied to the large language model, the metadata knowledge and content knowledge of the target data element are set as prompt words to be imported into the large language model to guide the large language model to generate query commands according to the metadata knowledge, simulate a query task to query the data platform, and determine the matching degree between the query result and the content knowledge of the target data element. It should be understood that a high matching degree indicates that the target data element can be applied by the large language model to answer user questions. Correspondingly, according to the matching degree between the query result and the content knowledge of the target data element, and the cost of the target data element applied to the large language model, the effectiveness of the target data element applied to the large language model can be effectively calculated, so as to reflect the importance of the target data element to the large language model in the dimension of application effectiveness, and then provide a basis for optimizing the data elements attached to the large language model.

[0149] Optionally, the cost of applying the target data element to the large language model is determined based on the storage cost and computing cost of the target data element on the data platform.

[0150] Optionally, the storage cost of the target data element on the data platform is determined based on the table cost of the data table to which the target data element belongs and the number of data elements; wherein the table cost of any target data table is determined based on at least one of the following: The storage cost of the target data table; The generation cost of the target data table; The storage cost of the parent table allocated to the target data table The generation cost of the parent table allocated to the target data table.

[0151] Optionally, the metadata knowledge of the target data element is obtained by embedding and encoding the metadata information of the target data element; the metadata information includes at least one of the directory, description, field, shard, data query log and data lineage relationship corresponding to the target data element on the data platform; the content knowledge of the target data element is obtained by embedding and encoding the content information of the target data element; the content information includes at least one of text content information, image content information and table content information.

[0152] Optionally, the field corresponding to the target data element includes the original field of the target data element, and at least one of the following: A statistical field of a specified calculation method obtained by calculating the amount information in the original field according to a specified calculation method; A specified enumeration type statistical field for performing statistics on the amount information in the original field according to the specified enumeration type; A specified unit time series statistics field obtained by counting the time series information in the original field according to a specified unit time series window; An information quantity statistics field for counting the quantity of information in the original field.

[0153] Optionally, the shards corresponding to the target data element are obtained by dividing multiple data elements of the data platform based on at least one of the data table, database and data warehouse subject domains as dimensions; or, the shards corresponding to the target data element are determined based on the classification result of the data table to which the target data element belongs, and the classification result of the data table to which the target data element belongs is determined by clustering the data tables based on the query relationship between multiple data tables of the data platform; or, the classification result of the data table to which the target data element belongs is input into a deep learning model, and the query relationship features between the data table to which the target data element belongs and other data tables are input into the deep learning model, and the deep learning model is trained based on the query relationship features in the sample data table and the shard labels annotated on the sample data table.

[0154] Optionally, the prompt word also includes query description information constructed based on the Cot thinking chain for inquiring about the target data element; and generates query commands based on the metadata knowledge of the target business document of the large language model, including: determining a query path based on the metadata knowledge of the target business document, and generating query commands that conform to the query path according to the query description information.

[0155] Optionally, the prompt word also includes a supplementary data element in the data platform that meets a preset similarity requirement with the target data element; the large language model further performs, based on the prompt word: determining a matching degree between the query result and content knowledge of the supplementary data element; The calculation module 630 determines the effectiveness of the target data element applied to the large language model based on the matching degree between the query result and the content knowledge of the target data element, and the cost of applying the target data element to the large language model, including: determining the effectiveness of the target data element applied to the large language model based on the matching degree between the query result and the content knowledge of the target data element, the matching degree between the query result and the content knowledge of the supplementary data element, and the cost of applying the target data element to the large language model.

[0156] It should be noted that the evaluation device of this embodiment can be used as Figure 1 The execution subject of the method shown can thus realize Figure 1 The steps and functions in the method shown.

[0157] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0158] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0159] The memory is used to store the computer program. Specifically, the computer program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides the computer program to the processor.

[0160] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming the above-mentioned Figure 6 The evaluation device of the data elements shown in the model application. Correspondingly, the processor executes the program stored in the memory and is specifically used to perform the following operations: The metadata knowledge of the target data element is obtained from the metadata knowledge base set up on the data platform attached to the large language model, and the content knowledge of the target data element is obtained from the content knowledge base set up on the data platform attached to the large language model; wherein the metadata knowledge base stores metadata knowledge of multiple data elements of the data platform, and the content knowledge base stores content knowledge of the multiple data elements.

[0161] The metadata knowledge and content knowledge of the target data element are set as prompt words to be imported into the large language model, so as to execute through the large language model: generating query commands based on the metadata knowledge of the target business document, and querying the data platform according to the query commands to obtain query results; and determining the degree of match between the query results and the content knowledge of the target data element.

[0162] Based on the matching degree between the query result and the content knowledge of the target data element and the cost of applying the target data element to the large language model, the effectiveness of applying the target data element to the large language model is determined.

[0163] The above is as in this manual Figure 1The method disclosed in the illustrated embodiment can be applied to a processor and implemented by the processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0164] Of course, in addition to software implementation, the electronic device of this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0165] In addition, an embodiment of the present application provides a computer program product, the computer program product comprising a computer-readable storage medium storing a computer program. The computer program is operable to cause a computer to execute Figure 1 The steps in the method shown include: The metadata knowledge of the target data element is obtained from the metadata knowledge base set up on the data platform attached to the large language model, and the content knowledge of the target data element is obtained from the content knowledge base set up on the data platform attached to the large language model; wherein the metadata knowledge base stores metadata knowledge of multiple data elements of the data platform, and the content knowledge base stores content knowledge of the multiple data elements.

[0166] The metadata knowledge and content knowledge of the target data element are set as prompt words to be imported into the large language model, so as to execute through the large language model: generating query commands based on the metadata knowledge of the target business document, and querying the data platform according to the query commands to obtain query results; and determining the degree of match between the query results and the content knowledge of the target data element.

[0167] Based on the matching degree between the query result and the content knowledge of the target data element and the cost of applying the target data element to the large language model, the effectiveness of applying the target data element to the large language model is determined.

[0168] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0170] The above are only embodiments of this specification and are not intended to limit this specification. For those skilled in the art, this specification may be subject to various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the scope of the claims of this specification. In addition, all other embodiments obtained by ordinary technicians in this field without creative work shall fall within the scope of protection of this document.

Claims

1. A method for evaluating data elements in model applications, characterized in that: include: Obtaining metadata knowledge of a target data element from a metadata knowledge base set up on a data platform attached to a large language model, and obtaining content knowledge of the target data element from a content knowledge base set up on the data platform attached to the large language model; wherein the metadata knowledge base stores metadata knowledge of multiple data elements of the data platform, and the content knowledge base stores content knowledge of the multiple data elements; The metadata knowledge and content knowledge of the target data element are set as prompt words to be imported into the large language model, so as to execute through the large language model: generating a query command based on the metadata knowledge of the target business document, and querying the data platform according to the query command to obtain a query result; determining the matching degree between the query result and the content knowledge of the target data element; Based on the matching degree between the query result and the content knowledge of the target data element and the cost of applying the target data element to the large language model, the effectiveness of applying the target data element to the large language model is determined.

2. The method according to claim 1, characterized in that The cost of applying the target data element to the large language model is determined based on the storage cost and the computational cost of the target data element on the data platform.

3. The method according to claim 2, characterized in that The storage cost of the target data element in the data platform is determined based on the table cost of the data table to which the target data element belongs and the number of data elements; wherein the table cost of any target data table is determined based on at least one of the following: The storage cost of the target data table; The generation cost of the target data table; The storage cost of the parent table allocated to the target data table The generation cost of the parent table allocated to the target data table.

4. The method according to claim 1, characterized in that: The metadata knowledge of the target data element is obtained by embedding and encoding the metadata information of the target data element; the metadata information includes at least one of the directory, description, field, shard, data query log and data lineage relationship corresponding to the target data element on the data platform; The content knowledge of the target data element is obtained by embedding and encoding the content information of the target data element; the content information includes at least one of text content information, picture content information and table content information.

5. The method according to claim 4, characterized in that The fields corresponding to the target data element include the original fields of the target data element and at least one of the following: A statistical field of a specified calculation method obtained by calculating the amount information in the original field according to a specified calculation method; A specified enumeration type statistical field for performing statistics on the amount information in the original field according to the specified enumeration type; A specified unit time series statistics field obtained by counting the time series information in the original field according to a specified unit time series window; An information quantity statistics field for counting the quantity of information in the original field.

6. The method according to claim 4, characterized in that The shards corresponding to the target data elements are obtained by dividing the multiple data elements of the data platform based on at least one of the data table, database and data warehouse subject domains as a dimension; or, The shard corresponding to the target data element is determined based on the classification result of the data table to which the target data element belongs, and the classification result of the data table is determined by clustering the data tables based on the query relationship between multiple data tables of the data platform; or, the classification result of the data table is input into a deep learning model to predict the query relationship features between the data table to which the target data element belongs and other data tables, and the deep learning model is trained based on the query relationship features in the sample data table and the shard labels annotated on the sample data table.

7. The method according to any one of claims 1 to 6, characterized in that: The prompt words also include inquiry description information constructed based on the Cot thinking chain for inquiring about the target data element; Generating a query command based on the metadata knowledge of the target business document includes: A query path is determined based on the metadata knowledge of the target business document, and a query command that conforms to the query path is generated according to the query description information.

8. The method according to any one of claims 1 to 6, characterized in that: The prompt word also includes a supplementary data element in the data platform that meets a preset similarity requirement with the target data element; the large language model further performs, based on the prompt word: determining a matching degree between the query result and the content knowledge of the supplementary data element; Determining the effectiveness of applying the target data element to the large language model based on the matching degree between the query result and the content knowledge of the target data element and the cost of applying the target data element to the large language model includes: Based on the matching degree between the query result and the content knowledge of the target data element, the matching degree between the query result and the content knowledge of the supplementary data element, and the cost of applying the target data element to the large language model, the effectiveness of applying the target data element to the large language model is determined.

9. A device for evaluating data elements in model application, characterized in that: include: An acquisition module, used to acquire metadata knowledge of a target data element from a metadata knowledge base set up by a large language model-attached data platform, and to acquire content knowledge of the target data element from a content knowledge base set up by the large language model-attached data platform; wherein the metadata knowledge base stores metadata knowledge of multiple data elements of the data platform, and the content knowledge base stores content knowledge of the multiple data elements; A query module, used to set the metadata knowledge and content knowledge of the target data element as prompt words to import into the large language model, so as to execute through the large language model: generating a query command based on the metadata knowledge of the target business document, and querying the data platform according to the query command to obtain a query result; determining the matching degree between the query result and the content knowledge of the target data element; A calculation module is used to determine the effectiveness of applying the target data element to the large language model based on the matching degree between the query result and the content knowledge of the target data element and the cost of applying the target data element to the large language model.

10. An electronic device comprising: processor; and a memory arranged to store computer executable instructions which, when executed, cause the processor to perform a method as claimed in any one of claims 1 to 8.

11. A computer program product, comprising a computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute the method according to any one of claims 1 to 8.

Images