Metadata generation method and device

Abstract

The invention relates to the field of data analysis, and discloses a metadata generation method and device, and the method comprises the steps: receiving a metadata generation task for a target data table; based on the task type of the metadata generation task, determining the category of context information matched with the task type; based on the category of the context information, collecting the context information of the corresponding category and taking the context information as dependent context information of the target data table; generating a cue word of the metadata generation task through a cue word template matched with the task type and the dependency context information; and obtaining metadata of the target data table through the large language model and the cue word. In the scheme, the different types of context information strongly related to the task type is beneficial to accurately generating the metadata, and the generated cue word is based on the matched task type and the dependent context information, so that the cue word used for generating the metadata is automatic and highly accurate, and the accuracy of the generated cue word is improved. The accuracy of metadata generation can also be improved.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, and in particular to a method and apparatus for generating metadata. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial sector, and the traditional financial industry is gradually transforming into Fintech. However, due to the security and real-time requirements of the financial industry, higher demands are also being placed on technology.

[0003] In enterprise-level data governance, the metadata of database tables and fields is the foundation for data understanding and use. Metadata describes "what this table does" and "what this field means," directly impacting data discoverability, understandability, and usability.

[0004] Currently, regarding metadata generation, the Chinese patent with publication number CN119740653A, "Metadata Management Method and System Based on Large Model," discloses a method that first collects labeled metadata as training data, then fine-tunes the base large model using the LORA algorithm; next, the fine-tuned model is integrated into the metadata management process, and business system metadata is periodically collected through a metadata collection engine. When metadata changes, the fine-tuned model is invoked to complete the changes; finally, the completed results are written into a metadata knowledge graph. However, this approach mainly relies on basic metadata such as table names and field names, lacking a deep understanding of data sources, processing logic, and business scenarios. For example, the field "cust_id" may have different meanings in different business scenarios, such as customer ID, customer number, or customer unique identifier. Based solely on the field name, it is difficult to accurately determine its business meaning. Summary of the Invention

[0005] This application provides a metadata generation method for accurately generating metadata related to data tables, and the generated metadata can closely match real business scenarios.

[0006] In a first aspect, embodiments of this application provide a metadata generation method, the method comprising: receiving a metadata generation task for a target data table; determining a category of context information matching the task type based on the task type of the metadata generation task; collecting context information of the corresponding category based on the category of the context information and using it as dependent context information of the target data table; generating prompt words for the metadata generation task using prompt word templates matching the task type and the dependent context information; and obtaining the metadata of the target data table using a large language model and the prompt words.

[0007] In the above scheme, as the task type of metadata generation for the target data table varies, the corresponding categories of context information also differ. Therefore, collecting different categories of context information and using them as dependent context information for the target data table helps to accurately generate metadata. In addition, prompt words for the metadata generation task can be obtained by matching prompt word templates with the task type of the metadata generation task and dependent context information. Finally, the metadata of the target data table can be obtained through a large language model and prompt words. Because the generated prompt words are based on the matched task type and dependent context information, the prompt words used to generate metadata will be automated and highly accurate, which can also improve the accuracy of metadata generation.

[0008] In one possible implementation method, the prompt word template matching the task type is obtained by: determining a general prompt word template corresponding to the task type; the general prompt word template includes a first context marker and a corresponding first parameter; generating a business scope associated with the task based on the metadata, and determining a scope template matching the business scope; the scope template includes a second context marker and a corresponding second parameter; updating the first parameter of the context marker with the same name in the general prompt word template through the second parameter, and adding the second context marker and the corresponding second parameter not present in the general prompt word template to the general prompt word template, thereby obtaining a prompt word template matching the task type.

[0009] In the above scheme, a corresponding general prompt template is generally configured for each task type. The general prompt template contains a first context marker and a corresponding first parameter. However, for certain specific metadata generation tasks, there is also a scope template that matches the business scope under its associated business scope. This scope template contains a second context marker and a corresponding second parameter. In this application, for such specific metadata generation tasks, the priority of the scope template is set higher than that of the general prompt template. To this end, this application can update the first parameter of the context marker with the same name in the general prompt template with the second parameter in the scope template, and add the second context marker and the corresponding second parameter that are contained in the scope template but not in the general prompt template to the general prompt template. The final template formed is the prompt template that matches the specific task type.

[0010] In one possible implementation, the context marker includes an explicit context marker and an implicit context marker; the explicit context marker is used to directly indicate that the resource usage of the context marker does not exceed a first preset threshold; the implicit context marker is used to indirectly indicate that the resource usage of the context marker does not exceed a second preset threshold; the indirect indication is achieved through parameter settings of the context marker.

[0011] In the above scheme, by adding explicit context tags and implicit context tags to the context tags, both tags are used to restrict the amount of resources used during the generation of metadata to not exceed the set threshold. Therefore, lightweight metadata generation can be achieved in this way.

[0012] In one possible implementation, generating prompts for the metadata generation task using a prompt template that matches the task type and the dependency context information includes: extracting instantiation data required for the prompt template from the context information of any category in the dependency context information and obtaining prompts with instantiation data based on the prompt template.

[0013] In one possible implementation, the context information for each category is collected by the corresponding collector; the instantiation data for each category is processed by the corresponding processor.

[0014] In one possible implementation, the categories of the context information include at least one or more of basic metadata, lineage summary, business reference, and scenario context; the step of collecting context information of the corresponding category based on the categories of the context information and using it as the dependency context information of the target data table includes: collecting basic information of the target data table and using it as basic metadata in the dependency context information of the target data table; determining the lineage summary as the dependency context information of the target data table based on the upstream data table of the target data table and the SQL statement containing the target data table and the upstream data table; collecting business-level information of the target data table and using it as business reference in the dependency context information of the target data table; and collecting downstream applications and operational characteristics of the target data table and using them as scenario context in the dependency context information of the target data table.

[0015] In one possible implementation method, determining the lineage summary in the dependency context information of the target data table based on the upstream data table of the target data table and the SQL statement containing the target data table and the upstream data table includes: using any field in the basic information of the target data table as an initial tracing field, determining the tracing field from the SQL statement containing the target data table and the upstream data table, and performing the step of determining the tracing field from the SQL statement containing the target data table and the upstream data table on the tracing field until the tracing termination condition is met, thereby obtaining the lineage summary.

[0016] The above scheme describes in detail the formation process of the lineage summary. It is not difficult to see from the described formation process that this application starts from the fields of basic information in the target data table and traces upward layer by layer until the tracing termination condition is met, highlighting the rigor of the lineage summary formation.

[0017] In one possible implementation, before determining the tracing fields from the SQL statement containing the target data table and the upstream data table, the method further includes: determining whether to initiate a multi-level tracing operation based on the task type and the number of upstream data tables of the target data table; wherein, the multi-level tracing operation represents tracing upstream data tables that are relied upon, starting from the target data table; if it is determined to initiate a multi-level tracing operation, the multi-level tracing operation is stopped under one or more tracing termination conditions, namely, the maximum tracing level, the maximum tracing resource consumption, and the absence of any further upstream data tables.

[0018] The above scheme provides specific details on the initiation and termination conditions for tracing the lineage, which enhances the understandability of this application when generating a lineage summary.

[0019] In one possible implementation, determining the source field from the SQL statements containing the target data table and the upstream data table includes: determining the category of the SQL statements containing the target data table and the upstream data table; the category includes long SQL statements and short SQL statements; if it is determined that the SQL statements containing the target data table and the upstream data table belong to short SQL statements, then deduplicating each short SQL statement to obtain deduplicated short SQL statements; determining a second field from the deduplicated short SQL statements that is related to a first field in the first target data table and belongs to the upstream data table, and using the second field as the source field.

[0020] The above scheme provides a process for determining the traceability field. It is easy to see from this process that this application considers the use of resources when determining the traceability field. That is, this application filters long SQL statements and duplicate SQL statements and uses the deduplicated short SQL statements to determine the traceability field, thus avoiding the resource consumption caused by processing long SQL statements or duplicate SQL statements when determining the traceability field.

[0021] In one possible implementation method, the deduplication of each short SQL statement includes: sequentially calculating the hash value of each short SQL statement, and adding short SQL statements with unique hash values ​​to a unique hash set; for each short SQL statement in the unique hash set, calculating the similarity between any two short SQL statements, and taking short SQL statements with similarity values ​​lower than a third set threshold as deduplicated short SQL statements.

[0022] Secondly, embodiments of this application provide a metadata generation apparatus, comprising: a receiving unit for receiving a metadata generation task for a target data table; a determining unit for determining a category of context information matching the task type based on the task type of the metadata generation task; a collecting unit for collecting context information of the corresponding category based on the category of the context information and using it as dependent context information of the target data table; a generating unit for generating prompts for the metadata generation task using prompt templates matching the task type and the dependent context information; the generating unit is further configured to obtain the metadata of the target data table using a large language model and the prompts.

[0023] Thirdly, embodiments of this application provide a computing device, including: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute any implementation method of the first aspect according to the obtained program.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform any implementation method as described in the first aspect.

[0025] Fifthly, embodiments of this application provide a computer program product, the computer program product including computer-executable instructions, the computer-executable instructions being used to cause a computer to perform any implementation method as described in the first aspect. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic diagram of a system architecture for metadata generation provided in an embodiment of this application; Figure 2 A schematic diagram illustrating a metadata generation method provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the collection of a bloodline summary as provided in an embodiment of this application; Figure 4 This is a schematic diagram of an SQL deduplication process provided in an embodiment of this application; Figure 5 A schematic diagram of a metadata generation apparatus provided in an embodiment of this application; Figure 6 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] refer to Figure 1 This is a system architecture diagram for metadata generation applicable to the embodiments of this application. The system architecture diagram for metadata generation includes at least a terminal device 101 and a metadata generation system 102.

[0030] Terminal device 101 has a target application installed for metadata generation. This target application can be a pre-installed client, a web application, or a mini-program embedded in other applications. Terminal device 101 can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these.

[0031] The metadata generation system 102 serves as the backend server for the target application, providing services to it. The metadata generation system 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0032] Terminal device 101 and metadata generation system 102 can be connected directly or indirectly through wired or wireless communication, and this application does not impose any restrictions.

[0033] Terminal device 101 responds to the user's metadata generation operation by sending a metadata generation instruction to metadata generation system 102. Metadata generation system 102 receives a metadata generation task for a target data table; based on the task type of the metadata generation task, it determines the category of context information matching the task type; based on the category of the context information, it collects the corresponding category of context information and uses it as the dependency context information of the target data table; using a prompt word template matching the task type and the dependency context information, it generates prompt words for the metadata generation task; and using a large language model and the prompt words, it obtains the metadata of the target data table.

[0034] based on Figure 1 The system architecture diagram described above, and the embodiments of this application provide a flow chart for a metadata generation method, such as... Figure 2 As shown, the process of this method is as follows: Figure 1 The metadata generation system 102 shown executes the following steps: Step 201: Receive the metadata generation task for the target data table.

[0035] In this step, the target data table can be any table in any database. Metadata generation tasks can include low-complexity tasks and high-complexity tasks. Low-complexity tasks can include field descriptions (field_comment) and table descriptions (table_comment), while high-complexity tasks can include field processing rules (field_rule) and table business descriptions (business_meaning).

[0036] For example, a database has a table named `customer_order` containing the fields: `order_id` (order number, primary key), `customer_id` (customer ID), `product_id` (product ID), `order_date` (order date), and `amount` (order amount). This table is known to be derived from upstream tables `customer_info` (customer information table) and `product_info` (product information table) through a SQL statement join.

[0037] It should be noted that the preceding paragraph provides a general description of the customer_order data table from a third-party perspective, to enhance user understanding. This content can be obtained by executing the metadata generation method proposed in this application; details can be found in the embodiments below.

[0038] This application describes a method for a metadata generation task, specifically a task requiring the generation of a table description (business_meaning) for the customer_order data table. Here, customer_order is the target data table.

[0039] For example, suppose there exists a receiving module that can receive a request from a user to generate metadata for the target data table customer_order, specifically the table business description business_meaning. The request content is as follows: (1) Target table identifier: dataset_id=12345 (corresponding to the customer_order table); (2) Task type: business_meaning (table business description); (3) Request timestamp: 2025-01-20 10:00:00.

[0040] Upon receiving the metadata generation task, this application may further verify the validity of the parameters: (1) Query the metadata database to confirm that the table customer_order corresponding to dataset_id=12345 exists; (2) Confirm that the task type table_comment is a system-supported task type.

[0041] After confirming the validity of the parameters, this application can generate a unique task identifier for the metadata generation task, for example, the generated unique task identifier is: task_id=TASK_20250120_100000_12345.

[0042] Step 202: Generate the task type of the task based on the metadata, and determine the category of context information that matches the task type.

[0043] In this step, the field description (field_comment), table description (table_comment), field processing rule (field_rule), and table business description (business_meaning), as exemplified earlier, are individual task types.

[0044] For different task types, business personnel will pre-configure the categories of context information that match each task type.

[0045] For example, taking the metadata generation task task_id=TASK_20250120_100000_12345 mentioned earlier as an example, its task type has already been specified in the user request. Its task type is business_meaning (table business description). Suppose that for this task type, the business personnel have already configured the matching context information category. At this time, we only need to retrieve the corresponding context information category for this task type from the already configured information according to the task type.

[0046] Step 203: Based on the category of the context information, collect the context information of the corresponding category and use it as the dependent context information of the target data table.

[0047] In this step, for each category of context information identified in step 202, context information for these categories can be collected separately, and all collected information can be used as the dependency context information for the target data table. This dependency context information serves as the basis for subsequently determining the metadata to be generated.

[0048] Step 204: Generate prompts for the metadata generation task using prompt templates that match the task type and the dependency context information.

[0049] In this step, business personnel will configure not only the categories of context information matching each task type, but also the prompt word templates matching each task type. After obtaining the dependency context information of the target data table in step 203, this step can continue to obtain the prompt word templates matching the task type of the metadata generation task of the target data table from the configuration information. With these two pieces of information, this step can generate the prompt words for the metadata generation task.

[0050] Step 205: Obtain the metadata of the target data table using the large language model and the prompt words.

[0051] In this step, after identifying the prompt words, the metadata of the target data can be output using a large language model.

[0052] In the above scheme, as the task type of metadata generation for the target data table varies, the corresponding categories of context information also differ. Therefore, collecting different categories of context information and using them as dependent context information for the target data table helps to accurately generate metadata. In addition, prompt words for the metadata generation task can be obtained by matching prompt word templates with the task type of the metadata generation task and dependent context information. Finally, the metadata of the target data table can be obtained through a large language model and prompt words. Because the generated prompt words are based on the matched task type and dependent context information, the prompt words used to generate metadata will be automated and highly accurate, which can also improve the accuracy of metadata generation.

[0053] Optionally, the categories of the context information include at least one or more of basic metadata, lineage summary, business reference, and scenario context; the step of collecting context information of the corresponding category based on the categories of the context information and using it as the dependency context information of the target data table includes: collecting basic information of the target data table and using it as basic metadata in the dependency context information of the target data table; determining the lineage summary as the dependency context information of the target data table based on the upstream data table of the target data table and the SQL statement containing the target data table and the upstream data table; collecting business-level information of the target data table and using it as business reference in the dependency context information of the target data table; and collecting downstream applications and operational characteristics of the target data table and using them as scenario context in the dependency context information of the target data table.

[0054] Based on the metadata generation task mentioned earlier, task_id=TASK_20250120_100000_12345, assuming that the business personnel have configured four categories of context information for it, namely basic metadata, lineage summary, business reference, and scenario context, the following metadata generation task will be used to describe how to obtain the context information of each category.

[0055] It should be noted that this application assumes that the context information of the metadata generation task with task_id=TASK_20250120_100000_12345 falls into one of these four categories in order to illustrate how the context information of the above four categories should be collected. However, the actual context information of the metadata generation task with task_id=TASK_20250120_100000_12345 may not fall into these categories. In other words, this application is merely providing an example.

[0056] Optionally, contextual information for each category can be collected using the corresponding collector.

[0057] For example, the technical solution of this application will be described below by how to use the collector to obtain context information of four categories of the target data table customer_order.

[0058] (1) For the category of basic metadata, the basic metadata collector can be invoked to collect the context information of this category. For example, the basic metadata collector can be invoked to query the basic information of the target data table, customer_order, from the database containing the target data table as the basic metadata in the dependency context information of the target data table. An example of basic information is as follows: Table name: customer_order; Table name (Chinese): Customer Orders; Creation time: 2024-06-01; Field list: 5 fields (order_id, customer_id, product_id, order_date, amount); Primary key: order_id; Data type: order_id(BIGINT), customer_id(BIGINT), product_id(INT), order_date(DATE), amount(DECIMAL).

[0059] For ease of description below, this application forms a dictionary of basic metadata for the basic information of the target data table customer_order, and names this dictionary table_metadata.

[0060] In addition, this application can also perform resource consumption statistics on the behavior of collecting basic metadata of the target data table mentioned above, such as consuming 500 tokens.

[0061] (2) For the category of business reference, since the information that can be referenced at the business level includes at least one or more of the following: fields with the same name, the product category in which the data table is located, and the subsystem, this application can call a business reference collector to collect the context information of this category. For example, for each field in the target data table, the business reference collector can be called to obtain the manually labeled information of the fields with the same name in other data tables in the database in which the target data table is located, as well as to obtain the product category information and subsystem information of the target data table, and use the obtained information as the business reference in the dependent context information of the target data table. An example of business reference is as follows: Suppose that in the target data table customer_order, there is a field with the same name in the data table customer_info, and the manual label of customer_id in the data table customer_info is "customer unique identifier, associated with CRM system customer ID"; and that there is a field with the same name in the data table order_detail, and the manual label of customer_id in the data table order_detail is "ordering customer ID"; Suppose that the product category information and subsystem information of the target data table customer_order are collected separately. The collected product category of the target data table customer_order is e-commerce platform, and the subsystem of the target data table customer_order is order management system.

[0062] For ease of description below, this application creates a list of fields with the same name as the target data table `customer_order`, named `database_other_fields`; and creates a product dictionary named `product_info` for the product category information of the target data table `customer_order`, and a subsystem dictionary named `subsystem_info` for the subsystem information of the target data table `customer_order`. The `database_other_fields` list, the `product_info` dictionary, and the `subsystem_info` dictionary together serve as business references in the dependency context information of the target data table `customer_order`. It should be noted that the business references are not limited to the three items of information mentioned above: fields with the same name, product categories, and subsystems.

[0063] In addition, this application can also perform resource consumption statistics on the business reference behavior of the above-mentioned target data table, such as consuming 1200 tokens.

[0064] (3) For the scenario context category, call the scenario context collector to query the downstream references and runtime characteristics of the target data table's customer_order. For example, the information retrieved is as follows: The target data table customer_order will be referenced by 15 downstream data tables, such as dws_customer_order_daily and app_customer_behavior_model. The target data table customer_order has 10 million records, and the data update frequency of the target data table customer_order is T+0, that is, daily.

[0065] For ease of description below, this application uses the downstream data table information, data volume, and data update frequency of the target data table customer_order described above to form a scenario context dictionary, named scenario_context. It should be noted that the scenario context is not limited to the three items of downstream data table information, data volume, and data update frequency mentioned in the example.

[0066] In addition, this application can also perform resource consumption statistics on the behavior of the scene context of the above-mentioned target data table, such as consuming 700 tokens.

[0067] (4) For the category of bloodline summary, since the bloodline relationships in the data table are generally quite complex, this category can be performed in at least two steps: first, determine the bloodline relationships, and then determine the bloodline summary from the bloodline relationships. Specifically: (4.1) The lineage summary collector can be called to collect the upstream data tables of the target data table customer_order and the SQL statements containing the target data table and at least one upstream data table within a set time period. The collected upstream data tables and SQL statements are used as the lineage relationship in the dependency context information of the target data table.

[0068] For example, the lineage summary collector might collect data from the upstream tables of the target data table `customer_order`, which include `customer_info` (customer information table) and `product_info` (product information table). It's also assumed that the lineage summary collector collects SQL statements from the last three days that contain the target data table `customer_order` and at least one upstream data table, and that the following five SQL statements are collected: SQL1: INSERT INTO customer_order SELECT c.customer_id, p.product_id,NOW(), p.price FROM customer_info c JOIN product_info p ON c.region=p.regionWHERE c.status='active' (short SQL) SQL2: INSERT INTO ... (1500 characters of complex ETL logic omitted here) ... (Long SQL) SQL3: INSERT INTO customer_order ... WHERE c.status='active' (Completely duplicated from SQL1) SQL4: INSERT INTO customer_order ... WHERE c.status='active' AND order_date='2025-01-19' (Highly similar to SQL1) SQL5: INSERT INTO customer_info SELECT ... FROM raw_customer (short SQL) For ease of description below, this application forms a lineage list of the two upstream data tables and the five SQL statements mentioned above, and names this list lineage_info.

[0069] In addition, this application can also perform resource consumption statistics on the bloodline relationship behavior of the above-mentioned target data table, such as consuming 6000 tokens.

[0070] (4.2) Regarding the blood relationship obtained in (4.1), this application may obtain a blood relationship summary in the following manner: Optionally, determining the lineage summary in the dependency context information of the target data table based on the upstream data table of the target data table and the SQL statement containing the target data table and the upstream data table includes: using any field in the basic information of the target data table as the initial tracing field, determining the tracing field from the SQL statement containing the target data table and the upstream data table, and performing the step of determining the tracing field from the SQL statement containing the target data table and the upstream data table on the tracing field until the tracing termination condition is met, thereby obtaining the lineage summary.

[0071] For example, taking the `customer_id` field in the target table `customer_order` as the initial source field, by matching five SQL statements in the lineage relationship, if the `customer_id` field appears in any SQL statement, then all other fields in that SQL statement that are not in the target table `customer_order` are used as source fields. For the sake of convenience, let's call one of these other fields `field1`. Then, determine which upstream tables the table containing `field1` is. Here, we assume that the table containing `field1` is `table1`, and that `table1`'s upstream tables are `table2` and `table3`. At the same time, find SQL statements containing `table1`, `table2`, and / or `table3`. Let's assume there are 10 SQL statements. Then, match the `field1` field against these 10 SQL statements. If the `field1` field appears in any SQL statement, then all other fields in that SQL statement that are not in `table1` are used as source fields. Repeat this step until the source tracing termination condition is met, thus obtaining the lineage summary.

[0072] Before describing the above scheme for determining the lineage summary of the target data table from the lineage relationship, optionally, before determining the tracing field from the SQL statement containing the target data table and the upstream data table, the method further includes: determining whether to initiate a multi-level tracing operation based on the task type and the number of upstream data tables of the target data table; wherein, the multi-level tracing operation represents tracing the upstream data tables that the target data table depends on, starting from the lineage tracing, layer by layer upwards; if it is determined to initiate a multi-level tracing operation, the multi-level tracing operation is stopped under one or more of the following tracing termination conditions: maximum tracing level, maximum tracing resource consumption, and no more upstream data tables.

[0073] For example, before using fields from the target table as initial source fields and executing a method to determine source fields from SQL statements containing both the target and upstream tables, it's necessary to examine whether multi-level traversal is required. As the name suggests, multi-level traversal refers to using the target table as the starting point for tracing the lineage, and then tracing upwards layer by layer to the upstream tables it depends on.

[0074] As for multi-level penetration operations, business personnel had previously configured methods for whether to enable multi-level penetration operations. For example, for complex tasks and / or tasks with many upstream data tables, they were configured to enable multi-level penetration, while for simple tasks and / or tasks with few upstream data tables, they were configured not to enable multi-level penetration.

[0075] Furthermore, for tasks requiring multi-level trace-through operations, business personnel will also configure the stage at which further multi-level trace-through operations can be stopped. In other words, business personnel need to configure the trace-tracing termination conditions in advance. These conditions can include reaching the maximum trace-tracing level, consuming the maximum resources required for trace-tracing, and no further upstream data items being found.

[0076] Optionally, determining the source field from the SQL statements containing the target data table and the upstream data table includes: determining the category of the SQL statements containing the target data table and the upstream data table; the category includes long SQL statements and short SQL statements; if the SQL statements containing the target data table and the upstream data table are determined to be short SQL statements, then deduplication is performed on each short SQL statement to obtain deduplicated short SQL statements; from the deduplicated short SQL statements, a second field related to the first field in the first target data table and belonging to the upstream data table is determined, and the second field is used as the source field.

[0077] Optionally, the deduplication process for each short SQL statement includes: calculating the hash value of each short SQL statement in sequence, and adding short SQL statements with unique hash values ​​to a unique hash set; for each short SQL statement in the unique hash set, calculating the similarity between any two short SQL statements, and taking short SQL statements with similarity values ​​lower than a third set threshold as deduplicated short SQL statements.

[0078] The following example illustrates how the lineage summary of (4.2) can be collected using the target data table customer_order, its two upstream data tables customer_info and product_info, and five SQL statements.

[0079] like Figure 3 The diagram shown is a schematic diagram of a bloodline summary provided in an embodiment of this application.

[0080] Step 301: Determining the bloodline penetration procedure.

[0081] If the metadata generation task has low task type complexity and the number of upstream data tables of the target data table is 0, then the lineage lookup operation is not required; if the metadata generation task has high task type complexity and the number of upstream data tables of the target data table is greater than 0, then the lineage lookup operation is required.

[0082] Because the task type of this application is table business description (business_meaning), which is a high-complexity task type, and the target data table customer_order has two upstream data tables with a value greater than 0, a lineage penetration operation needs to be performed.

[0083] Step 302: Determining the SQL statement deduplication process.

[0084] If the number of SQL statements containing the target data table and the upstream data table meets the deduplication condition, then the SQL statements containing the target data table and the upstream data table are deduplicated.

[0085] In the example of the target data table customer_order, assuming the deduplication condition is set to no less than 5 records, since the number of SQL statements containing the target data table customer_order and the upstream data table in this application is 5, the deduplication condition is met. Therefore, deduplication processing needs to be performed on 5 SQL statements.

[0086] the following Figure 4 This application provides a schematic diagram of SQL deduplication processing as an embodiment: Step 401: Initialize the deduplication and splitting structure Initialize the short_sqls list: [] (used to store short SQL statements) Initialize the long_sqls list: [] (for storing long SQL statements) Initialize the `seen_hashes` set: {} (for hash deduplication) Step 402: Set the shunt threshold The system determines the threshold for splitting long and short SQL queries based on preset configurations.

[0087] Example of a traffic splitting threshold: In this application, it is set to MAX_SQL_LENGTH = 1000 characters (or MAX_SQL_TOKENS = 1500).

[0088] Traffic splitting criteria: If the character length of the SQL or the estimated token value exceeds this threshold, it is judged as "long SQL".

[0089] It should be noted that the traffic splitting threshold is a configurable item used to balance the amount of context information and the cost of tokens.

[0090] Step 403: Traverse the SQL list and perform splitting.

[0091] (1) Traverse the list of SQL queries to be processed: {SQL1, SQL_LONG, SQL3, SQL4, SQL5} (2) Processing SQL1: If the length (assuming 150 characters) is less than 1000, it is considered a short SQL. Add it to the short_sqls list.

[0092] (3) Processing SQL_LONG: If the length (1500 characters) > 1000, it is determined to be a long SQL. Add it to the long_sqls list.

[0093] (4) Processing SQL3: If the length (150 characters) is less than 1000, it is considered a short SQL. Add it to the short_sqls list.

[0094] (5) Processing SQL4: If the length (180 characters) is less than 1000, it is considered a short SQL statement. Add it to the short_sqls list.

[0095] (6) Processing SQL5: If the length (120 characters) is less than 1000, it is considered a short SQL statement. Add it to the short_sqls list.

[0096] Traffic splitting results:

[0097] long_sqls: { SQL_LONG}

[0098] short_sqls: { SQL1, SQL3, SQL4, SQL5}

[0099] Step 404: Perform compound deduplication on the short SQL list

[0100] Phase 1: Complete Duplicate Removal

[0101] Iterate through the short_sqls list and calculate the MD5 hash:

[0102] Process SQL1: Calculate hash H(SQL1) and store it in seen_hashes.

[0103] Processing SQL3: Calculate the hash H(SQL3). It is found that H(SQL3) == H(SQL1), meaning it already exists in `seen_hashes`. Perform deduplication, removing SQL3 from `short_sqls`.

[0104] Process SQL4: Calculate hash H(SQL4) and store it in seen_hashes.

[0105] Processing SQL5: Calculate hash H(SQL5) and store it in seen_hashes.

[0106] This stage outputs short_sqls: { SQL1, SQL4, SQL5}

[0107] Phase Two: Similarity Clustering (SimHash or N-gram)

[0108] Calculate the similarity of the short_sqls list output from stage one.

[0109] Calculate Sim(SQL1, SQL4). Assume the similarity threshold is 90% and the calculated result Sim(SQL1, SQL4) > 90%.

[0110] Perform clustering and merge SQL4

[0111] This stage outputs short_sqls: { SQL1, SQL5}

[0112] Step 405: Output deduplication and splitting results and placeholder strategy

[0113] Final output: long_sqls: 1 (SQL_LONG).

[0114] short_sqls: 2 rows (SQL1, SQL5).

[0115] Deduplication results: Successfully split 1 long SQL statement, removed 1 completely duplicate SQL statement (SQL3), and merged 1 highly similar SQL statement (SQL4).

[0116] Positioning strategy: If short_sqls is ultimately empty and long_sqls is not empty, a placeholder message will be returned: "Excessively long SQL exists, has been split and processed, manual review is recommended."

[0117] If both short_sqls and long_sqls are empty, a placeholder message will be returned: "No lineage SQL was obtained. Please check the lineage collection configuration."

[0118] At this point, a context dictionary can be output regarding the SQL statement containing the target data table and the upstream data table: context["filtered_sqls"] = short_sqls context["long_sqls"] = long_sqls(Additional warning message: "The following SQL is too long, manual review is recommended") Step 303: Initialize the BFS queue and the visited_tables collection.

[0119] BFS queue: (customer_order, 0)

[0120] visited_tables: {customer_order}

[0121] Initial traceability fields: {order_id, customer_id, product_id, order_date, amount}

[0122] Step 304, BFS loop

[0123] First cycle:

[0124] Retrieved from the BFS queue: (customer_order, 0)

[0125] Query its direct upstream data tables: customer_info, product_info

[0126] Add the retrieved upstream data table to the BFS queue: [(customer_info, 1), (product_info,1)]

[0127] Pass in the short_sqls list output from stage two in step 302, namely SQL1 and SQL5.

[0128] Field-level filtering: Extract field dependencies from the SQL to determine that customer_id and product_id originate from customer_info and product_info respectively. Update the source fields: Customer_info requires attention to the following fields: {customer_id, region, status}; product_info requires attention to the following fields: {product_id, price, region}.

[0129] Summarize Level 0 SQL using the Large Language Model (LLM):

[0130] Prompt template:

[0131] The following is the processing SQL for the table customer_order. Please summarize its field mapping relationships:\nSQL1: ...\nSQLN: ...

[0132] LLM returned:

[0133] The customer_order table is joined with the customer_info and product_info tables to extract the customer_id, product_id, order date, and amount fields. The filter criteria are that the customer status is 'active' and the region matches.

[0134] The LLM summary above indicates a token consumption of approximately 100 tokens.

[0135] Hierarchical summary storage: level_summaries[0] = “Level 0 (target data table): customer_order table linked with customer_info and product_info tables…”

[0136] Second cycle:

[0137] Retrieved from the BFS queue: (customer_info, 1)

[0138] Determine the level: 1 ≤ max_level(5), then continue.

[0139] Check if visited: customer_info is not in visited_tables, continue.

[0140] Query the direct upstream data table of customer_info: raw_customer (original customer table)

[0141] Add the upstream data table to the queue: [(product_info, 1), (raw_customer, 2)]

[0142] SQL to query customer_info: SQL6: INSERT INTO customer_info SELECT customer_id, customer_name, region, status FROM raw_customer WHERE import_date > '2024-01-01'

[0143] Length: 120 characters, Estimated Tokens: 150

[0144] Field-level filtering: Based on the source fields {customer_id, region, status}, only focus on the source of these three fields in raw_customer.

[0145] Summarize the Level 1 SQL (customer_info part) using LLM:

[0146] LLM returned:

[0147] The customer_info table extracts customer ID, region, and status from the raw_customer table, with the filter condition being import date > 2024-01-01.

[0148] Summary of token consumption: Approximately 80 tokens

[0149] Hierarchical summary storage: level_summaries[1] = “Level 1 (upstream table): customer_info table is extracted from raw_customer table…”

[0150] Add customer_info to visited_tables

[0151] 3rd cycle:

[0152] Retrieved from the queue: (product_info, 1)

[0153] (The processing logic is similar to step 304, and a detailed description is omitted.)

[0154] After field-level filtering, only focus on the {product_id, price, region} fields.

[0155] 4th cycle:

[0156] Retrieved from the queue: (raw_customer, 2)

[0157] Determine the level: 2 ≤ max_level(5), continue.

[0158] Query the upstream table of raw_customer: None (original table, imported from external source).

[0159] Marked as the terminating node (exported table boundary).

[0160] Add raw_customer to visited_tables

[0161] Step 305: Determine the termination conditions and rollback strategy for source tracing.

[0162] The queue is empty: Yes

[0163] Terminate BFS loop

[0164] Field tracing failure rollback: If the field mapping relationship cannot be reliably identified, rollback to table-level tracing (analyzing the necessary context at that table level) to ensure process robustness.

[0165] Suppose that in step 304, if SQL1 is an extremely complex dynamic SQL or stored procedure, causing the system to be unable to reliably parse the customer_id field from the customer_info table.

[0166] Triggering rollback: The system will trigger a "field tracing failure rollback" at this time, stopping the precise tracing of the customer_id field.

[0167] Execute rollback: Switch to table-level tracing. The system will include the entire customer_info table (not just {customer_id, region, status}) and its upstream tables (such as raw_customer) in the context to ensure the integrity of the lineage, although this will sacrifice some context precision.

[0168] Step 306, Hierarchical Assembly

[0169] Assemble the bloodline chains by hierarchical order:

[0170] Level 0 (Target Data Table): The target data table is linked with multiple upstream data tables to extract key fields and filter them according to business conditions.

[0171] Level 1 (Upstream Data Table): The upstream data table extracts the required fields from the upstream original / detail data table and applies the necessary filtering conditions.

[0172] Layer 2 (Original Data Table): The original data table is collected / imported from external sources and serves as the starting point for lineage.

[0173] For ease of description below, this application forms a dictionary of lineage summaries from the output of the lineage chain, which is named lineage_sql_chained_summary (approximately 500 tokens).

[0174] Furthermore, this application can also perform time-consuming statistics on the aforementioned process of collecting the lineage summary of the target data table. When performing time-consuming statistics, considering the complexity of the process of collecting the lineage summary of the target data table and the numerous steps involved, this application can calculate the time consumption of each step in the process of collecting the lineage summary of the target data table separately. For example, this application consumed 6000 tokens when determining the lineage_info list, 505 tokens in the first BFS loop, and 735 tokens in the second BFS loop.

[0175] The advantage of calculating this time consumption is that the amount of resources required for subsequent operations can be dynamically adjusted based on the current execution result, so that the overall resource consumption of the scheme during execution is within an acceptable range.

[0176] After obtaining the dependency context information of the target data table, this application can further obtain a prompt word template that matches the task type of the metadata generation task of the target data table. Optionally, the prompt word template that matches the task type is obtained in the following way: determining a general prompt word template corresponding to the task type; the general prompt word template includes a first context marker and a corresponding first parameter; determining a scope template that matches the business scope based on the business scope associated with the metadata generation task; the scope template includes a second context marker and a corresponding second parameter; updating the first parameter of the context marker with the same name in the general prompt word template through the second parameter, and adding the second context marker and the corresponding second parameter that are not present in the general prompt word template to the general prompt word template, thereby obtaining a prompt word template that matches the task type.

[0177] For example, business personnel might configure corresponding generic prompt templates for different task types. For instance, for the task type "business_meaning" in the target data table "customer_order", the configured generic prompt template might be as follows: {{database_other_fields max_fields=5}} (By default, a maximum of 5 fields with the same name can be displayed for reference) {{lineage_sql max_sql_tokens=5000}} (Default: SQL accumulated tokens cannot exceed 5000) Taking the first record in the template, {{database_other_fields max_fields=5}}, as an example, the database_other_fields max_fields before the equals sign is called the context marker, and the 5 after the equals sign is called the parameter.

[0178] If the business scope associated with the metadata generation task does not have a matching scope template, the general prompt word template of the metadata generation task can be used directly as the final prompt word template.

[0179] However, for some metadata generation tasks, there might be a scope template for the business scope they are associated with. For the target data table `customer_order` mentioned earlier, it belongs to the e-commerce platform's business scope, and we assume that the business personnel have configured its scope template for this e-commerce platform business scope, as follows: {{database_other_fields max_fields=10}} (Maximum of 10 fields with the same name can be displayed for reference) {{lineage_sql max_sql_tokens=5000}} (Default: SQL accumulated tokens cannot exceed 5000) {{business_terms("SKU")}} (Knowledge related to SKUs in e-commerce business) Therefore, for the metadata generation task of business_meaning in the target data table customer_order, since it has both a general prompt template and a range template, this application can replace the parameters of the same-named context marker in the general prompt template with the parameters in the range template, and then add the records that are not in the general prompt template to the general prompt template, thereby obtaining the prompt template for the metadata generation task.

[0180] For example, for the context marker `database_other_fields max_fields`, parameter 10 in the range template can be used to replace parameter 5 in the general prompt template. For the context marker `lineage_sql max_sql_tokens`, since its parameters are consistent in both templates, it does not need to be updated. However, for the record `business_terms("SKU")`, it appears in the range template but not in the general prompt template, so this record needs to be added to the general prompt template. Therefore, it can be understood that the prompt template for the metadata generation task of `business_meaning` in the target data table `customer_order` will be consistent with its range template.

[0181] Optionally, the context marker includes an explicit context marker and an implicit context marker; the explicit context marker is used to directly indicate that the resource usage of the context marker does not exceed a first set threshold; the implicit context marker is used to indirectly indicate that the resource usage of the context marker does not exceed a second set threshold; the indirect indication is achieved through the parameter settings of the context marker.

[0182] For example, in the first record of the general prompt word template mentioned earlier, database_other_fieldsmax_fields=5, its context tag is an implicit context tag because it indirectly controls the amount of resources used by controlling the number of fields with the same name collected. For example, the fewer fields with the same name collected, the less resources are used.

[0183] For the second record in the general prompt word template mentioned earlier, lineage_sql max_sql_tokens=5000, its context marker is an explicit context marker because it directly controls the resource usage of the lineage summary during collection to not exceed 5000 tokens.

[0184] After obtaining the dependency context information of the target data table and acquiring a prompt word template that matches the task type of the metadata generation task for the target data table, this application can generate prompt words for the metadata generation task. Optionally, generating prompt words for the metadata generation task using the prompt word template that matches the task type and the dependency context information includes: for any category of context information in the dependency context information, extracting the instantiation data required for the prompt word template from the context information of the category and obtaining prompt words with instantiation data based on the prompt word template.

[0185] Optionally, the instantiated data for each category is processed by the corresponding processor.

[0186] For example, business personnel can pre-establish the relationship between each context marker in the prompt word template and its corresponding processor. For instance, the context marker "database_other_fields max_fields" in the prompt word template means that a maximum of 5 field references with the same name will be displayed. Therefore, the processor corresponding to this context marker can be named the "Field Reference Processor". This Field Reference Processor can be used to extract information about the same-named fields in the "Business Reference" category of the various categories of context information mentioned above. The extracted information is the instantiated data. Based on the extracted instantiated data and the prompt word template, prompt words with instantiated data can be obtained.

[0187] Furthermore, based on the context information of any of the various categories of context information described above, if a certain context tag contained in the prompt word template can be extracted from the context information of that category, then the extracted instantiated data can be used to construct prompt words with instantiated data.

[0188] Here are some examples of constructing prompt words with instantiated data: For context markers of {{metadata}}, the metadata_handler(context) is called. The handler returns formatted text such as "table name, primary key, fields and types". This formatted text can be used to replace the "basic metadata" item in the prompt word template.

[0189] For context markers of {{lineage_sql_chained_summary}}, the handler lineage_summary_handler(context) is called. The handler returns the context "lineage_sql_chained_summary", which can be used to replace the item "multi-layered lineage summary" in the prompt word template.

[0190] By replacing all contextual tags in the prompt template with instantiated data, the complete prompt can be output. The resulting prompt can then be used to output metadata from a large language model.

[0191] For large language models, when outputting metadata, the output results also need to be parsed. First, the response status is checked. If the status code is 200, the response is successful, and the result parsing process begins. The result parsing process includes: first attempting structured parsing, such as extracting key fields like table business descriptions from JSON format. If extraction is successful, the extracted information is directly output; this extracted information is the metadata of the target data table. If structured parsing fails, the JSON string within the Markdown code block "json..." is extracted and parsed. If both fail, the cause is investigated, and an alarm log is recorded.

[0192] Based on the same concept, embodiments of this application also provide a metadata generation apparatus, such as... Figure 5 As shown, the device includes: The receiving unit 501 is used to receive metadata generation tasks for the target data table.

[0193] The determining unit 502 is used to generate a task type based on the metadata and determine the category of context information that matches the task type.

[0194] The acquisition unit 503 is used to acquire context information of the corresponding category based on the category of the context information and use it as the dependent context information of the target data table.

[0195] The generation unit 504 is used to generate prompts for the metadata generation task using prompt templates that match the task type and the dependency context information.

[0196] The generation unit 504 is further configured to obtain the metadata of the target data table through the large language model and the prompt words.

[0197] Furthermore, for this device, the generation unit 504 is also used to determine a general prompt template corresponding to the task type; the general prompt template includes a first context marker and a corresponding first parameter; generate the business scope associated with the task based on the metadata, and determine a scope template that matches the business scope; the scope template includes a second context marker and a corresponding second parameter; update the first parameter of the context marker with the same name in the general prompt template through the second parameter, and add the second context marker and the corresponding second parameter that are not present in the general prompt template to the general prompt template, thereby obtaining a prompt template that matches the task type.

[0198] Furthermore, for this device, the context marker includes an explicit context marker and an implicit context marker; the explicit context marker is used to directly indicate that the resource usage of the context marker does not exceed a first preset threshold; the implicit context marker is used to indirectly indicate that the resource usage of the context marker does not exceed a second preset threshold; the indirect indication is achieved through the parameter settings of the context marker.

[0199] Furthermore, for this device, the generation unit 504 is specifically used to extract the instantiation data required for the prompt word template from the context information of any category in the dependent context information and obtain a prompt word with instantiation data based on the prompt word template.

[0200] Furthermore, for this device, the context information of each category is collected by the corresponding collector; the instantiation data of each category is processed by the corresponding processor.

[0201] Furthermore, for this device, the categories of the context information include at least one or more of basic metadata, lineage summary, business reference, and scenario context; the acquisition unit 503 is specifically used to acquire the basic information of the target data table and use it as the basic metadata in the dependency context information of the target data table; determine the lineage summary as the dependency context information of the target data table based on the upstream data table of the target data table and the SQL statement containing the target data table and the upstream data table; acquire the business-level information of the target data table and use it as the business reference in the dependency context information of the target data table; and acquire the downstream application and operational characteristics of the target data table and use them as the scenario context in the dependency context information of the target data table.

[0202] Furthermore, for this device, the acquisition unit 503 is specifically used to determine the tracing field from the SQL statement containing the target data table and the upstream data table by using any field in the basic information of the target data table as the initial tracing field, and to perform the step of determining the tracing field from the SQL statement containing the target data table and the upstream data table on the tracing field until the tracing termination condition is met, thereby obtaining the lineage summary.

[0203] Furthermore, for this device, the acquisition unit 503 is also used to determine whether to initiate a multi-level tracing operation based on the task type and the number of upstream data tables of the target data table; wherein, the multi-level tracing operation represents tracing upstream data tables that are relied upon, starting from the target data table as the ancestry tracing starting point; if it is determined to initiate a multi-level tracing operation, the multi-level tracing operation is stopped under one or more of the following tracing termination conditions: maximum tracing level, maximum tracing resource consumption, and no more upstream data tables.

[0204] Furthermore, for this device, the acquisition unit 503 is also used to determine the category of the SQL statements containing the target data table and the upstream data table; the category includes long SQL statements and short SQL statements; if it is determined that the SQL statements containing the target data table and the upstream data table belong to short SQL statements, then each short SQL statement is deduplicated to obtain deduplicated short SQL statements; from the deduplicated short SQL statements, a second field related to the first field in the first target data table and belonging to the upstream data table is determined, and the second field is used as the tracing field.

[0205] Furthermore, for this device, the acquisition unit 503 is also used to sequentially calculate the hash value of each short SQL statement, and add short SQL statements with unique hash values ​​to the unique hash set; for each short SQL statement in the unique hash set, calculate the similarity between any two short SQL statements, and take each short SQL statement with a similarity lower than a third set threshold as the deduplicated short SQL statement.

[0206] This application also provides a computing device, which may specifically be a desktop computer, portable computer, smartphone, tablet computer, personal digital assistant (PDA), etc. The computing device may include a central processing unit (CPU), memory, input / output devices, etc. Input devices may include a keyboard, mouse, touchscreen, etc., and output devices may include display devices, such as liquid crystal displays (LCDs) and cathode ray tubes (CRTs).

[0207] The memory may include read-only memory (ROM) and random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In this embodiment, the memory may be used to store program instructions for a metadata generation method; The processor is used to call program instructions stored in the memory and execute the metadata generation method according to the obtained program.

[0208] like Figure 6 The diagram shown is a schematic representation of a computing device provided in an embodiment of this application. The computing device includes: The processor 601, memory 602, transceiver 603, and bus interface 604 are included; wherein the processor 601, memory 602, and transceiver 603 are connected via bus 605. The processor 601 is used to read the program in the memory 602 and execute the above-described metadata generation method; Processor 601 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. It can also be a hardware chip. The aforementioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0209] The memory 602 is used to store one or more executable programs and can store data used by the processor 601 when performing operations.

[0210] Specifically, the program may include program code, which includes computer operation instructions. Memory 602 may include volatile memory, such as random-access memory (RAM); memory 602 may also include non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 602 may also include combinations of the above types of memory.

[0211] Memory 602 stores the following elements: executable modules or data structures, or subsets thereof, or extended sets thereof: Operation instructions: This includes various operation instructions used to perform various operations.

[0212] Operating system: includes various system programs used to implement various basic business functions and handle hardware-based tasks.

[0213] The 605 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0214] Bus interface 604 can be a wired communication interface, a wireless bus interface, or a combination thereof. The wired bus interface can be, for example, an Ethernet interface. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. The wireless bus interface can be a WLAN interface.

[0215] This application also provides a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a metadata generation method.

[0216] This application also provides a computer program product, which includes computer-executable instructions for causing a computer to execute a metadata generation method.

[0217] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0218] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0219] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0220] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0221] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A metadata generation method characterized by comprising: The method comprises the steps of: receiving a metadata generation task for a target data table; determining a category of context information matched with the task type of the metadata generation task based on the task type; collecting context information of the corresponding category as dependent context information of the target data table based on the category of context information; generating a prompt word of the metadata generation task through a prompt word template matched with the task type and the dependent context information; obtaining metadata of the target data table through a large language model and the prompt word.

2. The method of claim 1, wherein the prompt word template matched with the task type is obtained by the following method, comprising: determining a prompt word general template corresponding to the task type; the prompt word general template comprises a first context mark and a corresponding first parameter; determining a range template matched with a business range associated with the metadata generation task based on the business range; the range template comprises a second context mark and a corresponding second parameter; updating the first parameter of the context mark with the same name in the prompt word general template through the second parameter, and adding the second context mark and the corresponding second parameter not contained in the prompt word general template to the prompt word general template, thereby obtaining the prompt word template matched with the task type.

3. The method of claim 2, wherein the context mark comprises an explicit context mark and an implicit context mark; the explicit context mark is used to directly indicate that the resource usage of the context mark does not exceed a first set threshold; the implicit context mark is used to indirectly indicate that the resource usage of the context mark does not exceed a second set threshold; the indirect indication is realized through parameter setting of the context mark.

4. The method of claim 1, wherein the generation of the prompt word of the metadata generation task through the prompt word template matched with the task type and the dependent context information comprises: for any category of context information in the dependent context information, extracting instantiation data required by the prompt word template from the context information of the category and obtaining a prompt word with instantiation data based on the prompt word template.

5. The method of claim 4, wherein each category of context information is collected by a corresponding collector; each category of instantiation data is processed by a corresponding processor. The category of context information comprises one or more of basic metadata, blood relation summary, business reference and scene context; the collection of context information of the corresponding category as the dependent context information of the target data table based on the category of context information comprises: collecting basic information of the target data table as the basic metadata in the dependent context information of the target data table; determining a blood relation summary as the dependent context information of the target data table according to an upstream data table of the target data table and a SQL statement containing the target data table and the upstream data table.

6. The method according to any one of claims 1 to 5, wherein, ​ ​ ​ ​ collecting information of the target data table at a business level and taking the information as a business reference in the dependent context information of the target data table; collecting downstream application and running features of the target data table and taking the downstream application and running features as a scenario context in the dependent context information of the target data table.

7. The method of claim 6, wherein the determining of the blood relation summary in the dependent context information of the target data table according to the upstream data table of the target data table and the SQL statement containing the target data table and the upstream data table comprises: taking any field in the basic information of the target data table as an initial tracing field, determining a tracing field from the SQL statement containing the target data table and the upstream data table, and performing the step of determining the tracing field from the SQL statement containing the target data table and the upstream data table on the tracing field until a tracing termination condition is met, thereby obtaining the blood relation summary.

8. The method of claim 7, wherein before the determining of the tracing field from the SQL statement containing the target data table and the upstream data table, the method further comprises: determining whether to start a multi-layer penetration operation according to the task type and the number of the upstream data table of the target data table, wherein the multi-layer penetration operation represents a layer-by-layer upward tracing of the dependent upstream data table with the target data table as a blood relation tracing starting point; if it is determined to start the multi-layer penetration operation, stopping the multi-layer penetration operation when one or more of a tracing maximum layer, a tracing maximum resource consumption, and no more upstream data table are met.

9. The method of claim 7, wherein the determining of the tracing field from the SQL statement containing the target data table and the upstream data table comprises: determining a category of the SQL statement containing the target data table and the upstream data table, wherein the category comprises a long SQL statement and a short SQL statement; if it is determined that the SQL statement containing the target data table and the upstream data table belongs to the short SQL statement, performing a deduplication processing on each short SQL statement to obtain a deduplicated short SQL statement; determining a second field related to a first field in the first target data table and belonging to the upstream data table from the deduplicated short SQL statement, and taking the second field as the tracing field.

10. The method of claim 9, wherein the deduplication processing on each short SQL statement comprises: sequentially calculating a hash value of each short SQL statement, and adding a short SQL statement with a non-repeated hash value to a non-repeated hash set; for each short SQL statement in the non-repeated hash set, respectively calculating a similarity between any two short SQL statements, and taking each short SQL statement with a similarity lower than a third set threshold as a deduplicated short SQL statement. ​ ​ ​ ​

Images