Government affair data quality checking rule generation method and system based on large language model

Abstract

The present application relates to a kind of government affair data quality check rule generation method and system based on large language model, belong to data quality check technical field, solve the existing data quality check rule needs technician to participate, cannot automatically and quickly generate and self-adapting adjustment problem.Method includes: obtaining the target table to be checked and the metadata information of each target field therein;Each target field is taken out in turn, according to the metadata information of target table and target field, obtain field semantic vector, joint semantic vector, field structure vector and / or enumeration value vector, and then retrieve the knowledge fragment of the most similar historical field from government affair knowledge graph;When knowledge fragment is not empty, according to the metadata information of target table and target field and knowledge fragment, construct dynamic prompt word, input into large language model to generate the data quality check rule of target field.It realizes the automatic and quick generation of data quality check rule.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data quality checking, and in particular to a government data quality checking rule generation method and system based on a large language model. BACKGROUND

[0002] Under the background of digital economy, government data has the characteristics of wide range, authority and real-time. Ensuring the quality of government data is crucial for improving decision-making accuracy, improving public service level, promoting digital transformation of government affairs, and promoting the development of data economy.

[0003] Traditional data quality checking rules mainly rely on manual writing, which is inefficient and prone to errors. Usually, data quality checking rules are configured as templates to complete some automated processing, but if the requirements change, technical personnel still need to understand the relationship between table structures and fields in the data warehouse or data lake in order to write accurate checking rules to check data quality. This results in a high technical threshold for the entire process and requires a lot of time and manpower, resulting in low efficiency of data processing and analysis.

[0004] With the strong language understanding and generation capabilities of large language models, technical solutions have emerged that use large language models to convert natural language descriptions into structured SQL retrieval statements to provide users with the required data records. Existing solutions focus on understanding natural language intent and do not consider the standardization of database table structure and the accuracy of data. Therefore, there is still a lack of solutions that use large language models to generate SQL statements for checking data quality. SUMMARY

[0005] In view of the above analysis, the embodiments of the present application aim to provide a government data quality checking rule generation method and system based on a large language model to solve the problem that existing data quality checking rules require the participation of technical personnel and cannot automatically and quickly generate and adaptively adjust.

[0006] In one aspect, the embodiments of the present application provide a government data quality checking rule generation method based on a large language model, comprising the following steps:

[0007] Obtain the metadata information of the target table and each target field to be checked;

[0008] Take out each target field in turn, obtain the field semantic vector, joint semantic vector, field structure vector and / or enumeration value vector according to the metadata information of the target table and the target field, and then retrieve the knowledge fragment of the historical field with the largest similarity from the government knowledge graph; the knowledge fragment includes: metadata information of the historical field, data quality checking rules associated with the historical field, data standards and data quality checking templates;

[0009] When the knowledge fragment is not empty, a dynamic prompt word is constructed according to the metadata information of the target table and the target field and the knowledge fragment, and the data quality inspection rule of the target field is generated by inputting the dynamic prompt word into a large language model.

[0010] Based on the further improvement of the above method, the field semantic vector is obtained by using an embedding model to obtain the embedding vector of the target field annotation; the joint semantic vector is obtained by using the embedding model to obtain the embedding vector of the spliced target table annotation and target field annotation; the field structure vector is obtained by using the embedding model to obtain the embedding vector of the spliced multiple metadata of the target field; and the enumeration value vector is obtained by using the embedding model to obtain the embedding vector of the enumeration value list if the target field is an enumeration type.

[0011] Based on the further improvement of the above method, the metadata information of the target table and each target field thereof includes: target table name, target table annotation, each target field name, each target field annotation, basic attribute and constraint condition of each target field; the basic attribute includes: data type, length, precision and enumeration value list; and the constraint condition includes: whether it is a primary key, whether it is a foreign key, whether it is allowed to be empty, unique constraint and value range constraint.

[0012] Based on the further improvement of the above method, each historical field in the government affair knowledge graph has multiple indexes, wherein the first index is a weighted fusion vector calculated according to the joint semantic vector, the field structure vector and / or the enumeration value vector of the historical field and the respective weights, the second index is the joint semantic vector of the historical field, and the third index is the field semantic vector of the historical field.

[0013] Based on the further improvement of the above method, the knowledge fragment of the historical field with the maximum similarity is retrieved from the government affair knowledge graph, including:

[0014] According to the joint semantic vector, the field structure vector and / or the enumeration value vector of the target field and the respective weights, a weighted fusion vector of the target field is calculated;

[0015] The first similarity, the second similarity and the third similarity between the weighted fusion vector, the joint semantic vector and the field semantic vector of the target field and the first index, the second index and the third index of each historical field in the government affair knowledge graph are calculated in sequence, and as long as the maximum value of the calculated similarity is greater than the similarity threshold, the corresponding knowledge fragment is obtained and the retrieval is exited.

[0016] Based on the further improvement of the above method, the weight of the field structure vector and the initial weight of the enumeration value vector are dynamically adjusted according to the metadata information of the field, and the formula is as follows:

[0017]

[0018] wherein, W s and respectively represent the adjusted weight and initial weight of the field structure vector, W e and respectively represent the adjusted weight and initial weight of the enumeration value vector, k represents the enumeration coefficient; r represents the weight growth coefficient, and m represents the number of non-zero values in the field metadata information.

[0019] Further improvement based on the above method, if the maximum value of the first similarity is greater than the similarity threshold, the corresponding knowledge fragment includes: the metadata information of the historical field corresponding to the maximum value of the first similarity, and the associated historical data quality inspection rule; if the maximum value of the second similarity is greater than the similarity threshold, the corresponding knowledge fragment includes: the metadata information of the historical field corresponding to the maximum value of the second similarity, and the field attribute consistency inspection template in the data quality inspection template; if the maximum value of the third similarity is greater than the similarity threshold, the corresponding knowledge fragment includes: the data standard followed by the historical field corresponding to the maximum value of the third similarity, and the data quality inspection template related to the data standard.

[0020] Further improvement based on the above method, constructing a dynamic prompt word according to the metadata information of the target table and the target field and the knowledge fragment, including:

[0021] Setting the role as a data quality inspection rule generation expert;

[0022] If there is a historical data quality inspection rule in the knowledge fragment, the task target set for the role is: referring to the historical field and the historical data quality inspection rule, generating a data quality inspection rule for the current field in the table to be inspected; if there is a data standard in the knowledge fragment, the task target set for the role is: referring to the data standard and the related data quality inspection template, generating an inspection rule for the current field in the table to be inspected on whether it is consistent with the data standard; otherwise, the task target set for the role is: referring to the field attribute consistency inspection template, generating an inspection rule for the current field in the table to be inspected on whether it is consistent with the historical field attribute;

[0023] Setting the table to be inspected and the current field in the prompt word according to the metadata information of the target table and the target field; setting the reference information required by the task target in the prompt word according to the knowledge fragment.

[0024] Based on the further improvement of the above method, the large language model is obtained by supervised fine-tuning training based on the base large model; the loss function of the large language model includes SQL generation loss and SQL syntax loss; the SQL generation loss is calculated by using the cross-entropy loss function, and the SQL syntax loss is calculated according to the number and weight of each type of syntax error in the syntax detection result by performing syntax error detection on the data quality inspection rule output by the large language model.

[0025] In another aspect, the embodiment of the present application provides a government affair data quality inspection rule generation system based on a large language model, comprising:

[0026] A metadata extraction module is configured to obtain metadata information of a target table and each target field therein to be inspected.

[0027] A knowledge fragment retrieval module is configured to sequentially take out each target field, obtain a field semantic vector, a joint semantic vector, a field structure vector and / or an enumeration value vector according to the metadata information of the target table and the target field, and then retrieve a knowledge fragment of a historical field with the largest similarity from a government affair knowledge graph; the knowledge fragment comprises metadata information of the historical field, a data quality inspection rule associated with the historical field, a data standard and a data quality inspection template.

[0028] An inspection rule generation module is configured to, when the knowledge fragment is not empty, construct a dynamic prompt word according to the metadata information of the target table and the target field and the knowledge fragment, and input the dynamic prompt word into a large language model to generate a data quality inspection rule of the target field.

[0029] Compared with the prior art, the present application can achieve at least one of the following beneficial effects:

[0030] 1. The feature vector of the field is extracted from different dimensions and levels based on the metadata information, and then the related knowledge fragment is adaptively retrieved from the government affair knowledge graph, so that the retrieval strategy is automatically adjusted and the knowledge fragment is flexibly selected, and finally the data quality inspection rule is quickly generated by using the large language model, forming a complete and automatic data quality guarantee system, which can comprehensively and deeply detect and improve the data quality, effectively solve the problems of data consistency, accuracy and completeness, and ensure the reliability of the data.

[0031] 2. The dynamic adjustment of the weight of different feature vectors is realized according to the metadata information of the field, the semantic, structure and enumeration value features of the field are weighted and fused, the ability of deep understanding and accurate analysis of the field is improved, the generated statement is closely matched with the actual structure of the database, and the accuracy of the data quality rule is improved.

[0032] 3. According to the metadata information and the knowledge fragment of adaptive retrieval, a dynamic and efficient prompt word is constructed to improve the generation ability of the large language model; the syntax detection result of the data quality inspection rule is introduced into the loss function to guide the model to continuously optimize the identification and correction ability of the syntax error in the learning process, thereby improving the robustness and adaptability of the model when facing various complex syntax errors.

[0033] The above technical solutions can be combined with each other in the present application to realize more preferred combination solutions. Other features and advantages of the present application will be described in the subsequent specification, and some advantages will become apparent from the specification, or will be understood by implementing the present application. The purposes and other advantages of the present application can be realized and obtained from the contents specifically pointed out in the specification and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated herein and constitute a part of the specification. The drawings illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0035] Figure 1 A flow chart of a government data quality inspection rule generation method based on a large language model in Embodiment 1 of the present application;

[0036] Figure 2 A structure schematic diagram of a government data quality inspection rule generation system based on a large language model in Embodiment 2 of the present application. DETAILED DESCRIPTION

[0037] The preferred embodiments of the present application will be specifically described below in conjunction with the drawings, wherein the drawings constitute a part of the present application and serve to explain the principles of the embodiments of the present application, but are not used to limit the scope of the present application.

[0038] Embodiment 1

[0039] One specific embodiment of the present application discloses a government data quality inspection rule generation method based on a large language model, as shown in Figure 1 The method comprises the following steps:

[0040] S1, obtaining metadata information of a target table to be inspected and each target field therein.

[0041] In this embodiment, the selection operation of the target table is provided through the visual interface, and the metadata information of the target table and each target field therein is obtained according to the selected target table.

[0042] The embodiment is not limited to the method of obtaining metadata information, which can be obtained by querying database system tables or views, such as querying sys.tables, sys.columns, sys.foreign_keys and other system tables for SQL Server databases; querying information_schema.tables, information_schema.columns, information_schema.table_constraints and other views for MySQL databases; obtaining various metadata information through the DatabaseMetaData interface of JDBC; if a metadata management platform is constructed, the metadata information can also be directly obtained from the metadata management platform.

[0043] Further, the obtained metadata information of the target table and each target field thereof includes: a target table name, a target table annotation, each target field name, each target field annotation, basic attributes and constraint conditions of each target field. The basic attributes include: a data type, a length, a precision and an enumeration value list. The constraint conditions include: whether a primary key, whether a foreign key, whether to allow null, a unique constraint and a value range constraint. The target table annotation represents a Chinese name of the target table, and each target field annotation represents a Chinese name of each target field. When the target field is a string type, the length represents the maximum number of characters that the field can store. When the target field is a numerical type, the precision represents the number of digits after the decimal point. When the target field is an enumeration type, the obtained data type contains an enumeration value list, such as a gender field of the enumeration type, which corresponds to a field type of ENUM('male', 'female', 'unknown'). The 'ENUM' represents an enumeration type, and the information in the brackets is an enumeration value list.

[0044] S2, sequentially taking out each target field, obtaining a field semantic vector, a joint semantic vector, a field structure vector and / or an enumeration value vector according to the metadata information of the target table and the target field, and further retrieving a knowledge fragment of a historical field with the maximum similarity from the government affair knowledge graph. The knowledge fragment includes: metadata information of the historical field, data quality inspection rules associated with the historical field, data standards and data quality inspection templates.

[0045] It should be noted that, since there are multiple target fields in the target table, this step obtains the knowledge fragments that can be referred to by each target field from the government affair knowledge graph that has been constructed, so as to fully utilize historical experience, enrich the context information of the target field, facilitate the large language model to more comprehensively understand the business meaning and use scenario of the field, and generate data quality checking rules that are more in line with actual needs. The data quality checking rules in this embodiment are SQL (Structured Query Language) statements of the database, which are used to check the standardization, accuracy and consistency of the database table structure and database table records.

[0046] It should be noted that the data in the government affair knowledge graph is derived from the metadata of the database tables and fields of various business systems, historical data quality checking rules and explanations, data standard files, and the constructed entities include business systems, historical tables, historical fields and data standards; the relationships between entities include containing, associating and complying with. For example, the CRM system contains the order table and the customer table, the order table is associated with the customer table, the customer table contains the customer name field, the customer ID field and the customer age field, and the customer ID field complies with the ID card data standard.

[0047] Further, different types of entities include multiple attributes, wherein the attributes of the business system include but are not limited to the business system name, the business system description and the business system creation time; the attributes of the historical table include the table annotation and the table creation time; the attributes of the historical field include but are not limited to the data type, the length, the precision, the enumeration value list, whether it is a primary key, whether it is a foreign key, whether it is allowed to be empty, whether it is unique, and whether it has a value range constraint; and the attributes of the data standard include but are not limited to the standard data type, the standard data format, the standard value range, the standard basis, the data definer, the data manager and the data user.

[0048] In order to obtain accurate knowledge fragments, the entities corresponding to each historical field in the government affair knowledge graph are indexed based on multiple semantic vectors, which are used to capture the characteristics of the historical field in multiple dimensions.

[0049] It should be noted that the first index is a weighted fusion vector calculated according to the joint semantic vector, the field structure vector and / or the enumeration value vector of the historical field, the second index is the joint semantic vector of the historical field, and the third index is the field semantic vector of the historical field.

[0050] Specifically, the field semantic vector is obtained by using an embedding model to obtain the embedding vector of the field annotation, and is only used to represent the business semantic characteristics of the field itself; the joint semantic vector is obtained by using an embedding model to obtain the embedding vector of the spliced target table annotation and target field annotation, and is used to represent the business semantic characteristics of the field in a specific table.

[0051] The field structure vector is obtained by using an embedding model to obtain the embedding vector of the spliced metadata of the target field, and is used to represent the structural semantic features of the field. Since the primary key in the data table generally implements the uniqueness of the primary key by the database mechanism, the primary key generally does not have business meaning. The foreign key is used to establish the correlation between tables, and generally the data quality of the numerical value of the primary key and the foreign key is not checked. Therefore, the field structure vector is generated according to the data type, length, precision, whether to allow null, unique constraint and value range constraint of the metadata of the target field. Among them, the data type is standardized, that is, the same type is represented by the same English character, for example, the string type is Varchar; the length and precision are represented by actual numerical values. If it is a string type, the precision is represented by 0; whether to allow null is represented by 0 and 1 respectively to allow null and not to allow null; the unique constraint is represented by 1 and 0 respectively to represent unique and not unique; if there is a value range constraint, the upper limit and lower limit of the range are used to represent, otherwise 0 is used to represent that there is no value range constraint. The metadata after the above processing is spliced and input into the embedding model to obtain the field structure vector.

[0052] The enumeration value vector is obtained by using an embedding model to obtain the embedding vector of the enumeration value information if the target field is an enumeration type. That is, only the fields of the enumeration type have the enumeration value vector, and the fields of other types do not have the enumeration value vector.

[0053] It should be noted that the type of the embedding model is not limited in the embodiment, including but not limited to the trained model based on word2vec, Bert, etc.

[0054] Further, the weighted fusion vector calculated according to the joint semantic vector, the field structure vector and / or the enumeration value vector and the respective weights is used as the first index, which is used to represent the comprehensive data features of the historical field.

[0055] It should be noted that the initial weights of the joint semantic vector, the field structure vector and the enumeration value vector are dynamically adjusted according to the metadata information of the field in the embodiment, and then the three weights are normalized to obtain the final weights of the three vectors respectively.

[0056] Specifically, when it is identified according to the metadata information of the field that the field is an enumeration type, the length, whether to allow null, unique constraint and value range constraint of the field do not need to be considered, and therefore the weight of the structure semantic vector is smaller than the weight of the enumeration value vector. At this time, on the basis of the initial weight, the embodiment increases the enumeration value vector and reduces the weight of the field structure vector by setting an enumeration coefficient, and the formula is as follows:

[0057]

[0058] Among them, Ws and respectively represent the adjusted weight and initial weight of the field structure vector, W e and respectively represent the adjusted weight and initial weight of the enumeration value vector, and k represents a preset enumeration coefficient.

[0059] When the field is identified as an enumeration type according to the metadata information of the field, the weight W e of the enumeration value vector is set to 0, and the weight of the field structure vector is dynamically adjusted according to the number of metadata that is not 0 in the metadata information of the field structure vector. If the metadata information is not 0, a data quality check rule can be generated for it, and the structural characteristics of the field are highlighted. The formula is as follows:

[0060]

[0061] wherein r represents a preset weight growth coefficient, and m represents the number of metadata that is not 0 in the metadata information of the field.

[0062] For example, the data type of field A is Varchar, the length is 50, the precision is 0, “whether to allow null” is 0, that is, it is allowed to be null, “unique constraint” is 0, that is, it is not unique, and “value range constraint” is 0, that is, there is no value range constraint. Therefore, the number of metadata that is not 0 is 2. The data type of field B is Decimal, the length is 10, the precision is 2, “whether to allow null” is 1, that is, it is not allowed to be null, “unique constraint” is 0, that is, it is not unique, and “value range constraint” is 0, that is, there is no value range constraint. Therefore, the number of metadata that is not 0 is 4.

[0063] For the current target field, the method for calculating the plurality of indexes of the historical field in the government affair knowledge graph is used to calculate the field semantic vector, the joint semantic vector, the field structure vector and / or the enumeration value vector of the target field, and the weighted fusion vector of the target field is calculated according to the joint semantic vector, the field structure vector and / or the enumeration value vector of the target field according to the above method for calculating the weight.

[0064] In the embodiment, the weighted fusion vector, the joint semantic vector and the field semantic vector represent feature representations of different depths, respectively. The historical field with the maximum similarity under different conditions is obtained from the government affair knowledge graph in order, and the knowledge fragments of different dimensions are selected from the data associated with the historical field. This way not only selects the historical field that matches the target field highly, but also selects the most relevant and applicable knowledge fragments in the matching historical field, and simultaneously realizes the adaptive adjustment of the retrieval strategy and the knowledge fragments, thereby improving the accuracy and relevance of the retrieval result.

[0065] Specifically, the knowledge fragment of the historical field with the largest similarity is retrieved from the government affair knowledge graph, including:

[0066] The first similarity, the second similarity and the third similarity between the weighted fusion vector, the joint semantic vector and the field semantic vector of the target field and the first index, the second index and the third index of each historical field in the government affair knowledge graph are calculated in sequence. As long as the maximum value of the calculated similarity is greater than the similarity threshold, the corresponding knowledge fragment is obtained and the retrieval is exited.

[0067] That is, the first similarity between the weighted fusion vector of the target field and the first index of each historical field in the government affair knowledge graph is calculated first. If the maximum value of the first similarity is greater than the similarity threshold, the first knowledge fragment of the historical field corresponding to the maximum value of the first similarity is obtained, and the retrieval is not performed again. Otherwise, the second similarity between the joint semantic vector of the target field and the second index of each historical field in the government affair knowledge graph is calculated. If the maximum value of the second similarity is greater than the similarity threshold, the second knowledge fragment of the historical field corresponding to the maximum value of the second similarity is obtained, and the retrieval is not performed again. Otherwise, the third similarity between the field semantic vector of the target field and the third index of each historical field in the government affair knowledge graph is calculated. If the maximum value of the third similarity is greater than the similarity threshold, the third knowledge fragment of the historical field corresponding to the maximum value of the third similarity is obtained.

[0068] It should be noted that the similarity in the embodiment is obtained by calculating the cosine similarity.

[0069] Further, if the maximum value of the first similarity is greater than the similarity threshold, it indicates that each dimension of the target field is very similar to the historical field corresponding to the maximum value of the first similarity, and thus the data quality inspection rule of the historical field can be directly referred to. Therefore, the first knowledge fragment obtained includes the metadata information of the historical field and the associated historical data quality inspection rule.

[0070] If the maximum value of the second similarity is greater than the similarity threshold, it indicates that the table name and the field name of the target field are very similar to the historical field corresponding to the maximum value of the second similarity, and thus it is necessary to further check whether the other field-level metadata of the target field and the historical field are consistent, so as to ensure that the fields with the same business meaning are unified in structure with the existing historical field. Therefore, the second knowledge fragment obtained includes the metadata information of the historical field and the field attribute consistency inspection template in the data quality inspection template. The field attribute consistency inspection template includes the data type consistency inspection template, the length consistency inspection template, the value range consistency inspection template and the enumeration value consistency inspection template. Preferably, the relevant field attribute consistency inspection template is selected according to the metadata information of the target field.

[0071] If the maximum value of the third similarity is greater than the similarity threshold, it indicates that the target field is only similar to the historical field corresponding to the maximum value of the third similarity in the field name, and whether the historical field follows a data standard is considered. If yes, the third knowledge fragment obtained includes the data standard followed by the historical field and the data quality checking template related to the data standard. Illustratively, the field annotation of the historical field is an ID card, which follows the ID card data standard, and the data quality checking template related to the data standard is the ID card specification checking template.

[0072] Further, if the maximum value of the third similarity is greater than the similarity threshold, but the historical field corresponding to the maximum value of the third similarity does not follow any data standard, or the maximum value of the third similarity is less than or equal to the similarity threshold, in both cases the knowledge fragment obtained from the government affair knowledge graph is empty, then according to the metadata information of the target field, a data quality checking rule is generated using the corresponding data quality checking template, without performing step S3.

[0073] Illustratively, the metadata information of the target field includes "whether to allow null", which is not allowed to be null, then the table name and field name of the target field are replaced in the non-null checking template to generate a non-null checking rule; "unique constraint" requires uniqueness, then the table name and field name of the target field are replaced in the uniqueness checking template to generate a uniqueness checking rule.

[0074] S3, when the knowledge fragment is not empty, a dynamic prompt word is constructed according to the metadata information of the target table and the target field and the knowledge fragment, which is input into a large language model to generate a data quality checking rule of the target field.

[0075] It should be noted that efficient prompt words are beneficial to improve the generation ability of the large language model. By setting the role through the prompt word, focusing on the knowledge and skills in a specific field, more professional and accurate content can be generated; by specifying the task target of the role, the large language model can quickly and accurately perform the task, improving the interaction efficiency.

[0076] The embodiment constructs a dynamic prompt word according to the metadata information of the target table and the target field and the knowledge fragment, which includes:

[0077] ① Set the role as a data quality checking rule generation expert; illustratively, describe in the prompt word: you are now a data quality checking SQL generation expert.

[0078] If there is a historical data quality checking rule in the knowledge fragment, the task target set for the role is to generate a data quality checking rule for the current field in the table to be checked by referring to the historical field and the historical data quality checking rule; if there is a data standard in the knowledge fragment, the task target set for the role is to generate a checking rule for whether the current field in the table to be checked is consistent with the data standard by referring to the data standard and the data quality checking template associated with the data standard; otherwise, the task target set for the role is to generate a checking rule for the consistency of the current field in the table to be checked with the historical field attribute by referring to the field attribute consistency checking template.

[0079] The table to be checked and the current field in the prompt word are set according to the metadata information of the target table and the target field; and the reference information required by the task target in the prompt word is set according to the knowledge fragment.

[0080] Specifically, after the description of the role and the task target, the table to be checked, the current field and the reference information corresponding to the task target are spliced with a line break: the target table name is taken and placed after "table to be checked:", the metadata information of the target field is taken and spliced in order after "current field:" with a preset first connector; if the information referred to by the set task target is the historical field and the historical data quality checking rule, the metadata information of the historical field is taken from the knowledge fragment and spliced in order after "historical field:", and the historical data quality checking rule is taken and spliced in order after "historical data quality checking rule:", and finally the complete prompt word information is obtained. Exemplarily, the connector is set as "|".

[0081] Preferably, a SQL generation rule is set in the prompt word for constraining the reasoning direction of the large language model.

[0082] The large language model of the present embodiment is supervised fine-tuning training based on the base large model, in order to ensure that the generated data quality checking SQL can be correctly executed, the data quality checking rule output by the large language model is subjected to syntax error detection, and the syntax detection result is introduced into the loss function. Therefore, the loss function of the large language model of the present embodiment includes: SQL generation loss L SQL and SQL syntax loss L Syntax Two parts, wherein the SQL generation loss is obtained by calculating the difference between the generated data quality checking rule SQL and the real data quality checking rule SQL, and the SQL syntax loss is obtained by calculating the syntax error score, and the formula is as follows:

[0083] L total = L SQL + λ·L Syntax ,

[0084] Wherein, λ represents an influence coefficient, which is a hyperparameter, used to control the influence degree of the syntax error loss on the total loss. The larger λ is, the more the model will pay more attention to the correction of syntax errors.

[0085] Further, the SQL generation loss L SQL The cross-entropy loss function is used, and the formula is as follows:

[0086]

[0087] Wherein, N represents the total number of all parts in the SQL, y i represents the true label of the i-th part in the SQL. If the part is correct, y i is 1, otherwise 0; represents the probability of the i-th part predicted by the model.

[0088] The SQL syntax loss L Syntax is calculated according to the number and weight of each type of syntax error in the syntax detection result, and the formula is as follows:

[0089]

[0090] Wherein, M represents the total number of syntax error types, W j represents the weight of the j-th syntax error, and Num j represents the number of the j-th syntax error. In this embodiment, the output data quality inspection rule SQL is constructed into an abstract syntax tree, and the abstract syntax data is analyzed to obtain the syntax detection result. The syntax error types include: missing keywords, mismatched parentheses, incorrect clause order, table name error, and column name error.

[0091] In the training process, the random gradient descent method is used to update the model parameters to minimize the loss function value. After multiple rounds of training, the performance of the model on the government data quality inspection task is gradually improved, and the trained large language model is obtained.

[0092] In implementation, the target table to be subjected to data quality inspection is selected through step S1, the knowledge fragments of each target field are obtained one by one through step S2, and the role, task target, target table, target field, and reference information are dynamically spliced to generate a text form of prompt word, which is input into the trained large language model through step S3, and the data quality inspection rule of the target field is output.

[0093] It should be noted that the present embodiment does not limit the execution mode of step S2 and step S3. In step S2, the knowledge fragment of the current target field can be extracted, and then step S3 is executed to obtain the data quality checking rule of the current target field, and then step S2 is returned to extract the next target field. Alternatively, in step S2, the knowledge fragment of each target field can be obtained in turn, and then step S3 is executed in turn to obtain the data quality checking rule of each target field.

[0094] Compared with the prior art, the method for generating a government data quality checking rule based on a large language model provided by the present embodiment first extracts a feature vector of a field from different dimensions and levels based on metadata information, and then adaptively retrieves relevant knowledge fragments in combination with a government knowledge graph, thereby achieving automatic adjustment of a retrieval strategy and flexible selection of knowledge fragments. Finally, a large language model is used to quickly generate a data quality checking rule, forming a complete and automated data quality guarantee system, which can comprehensively and deeply detect and improve data quality, effectively solve problems such as data consistency, accuracy and completeness, and ensure data reliability. According to the metadata information of the field, dynamic adjustment of the weights of different feature vectors is realized, the semantic, structural and enumerated value features of the field are weighted and fused, the ability of deep understanding and accurate analysis of the field is improved, and the accuracy of the generated sentence is ensured to closely match the actual structure of the database. The generation ability of the large language model is improved according to the metadata information and the adaptively retrieved knowledge fragments, and the syntax detection result of the data quality checking rule is introduced into the loss function to guide the model to continuously optimize the recognition and correction ability of syntax errors in the learning process, thereby improving the robustness and adaptability of the model when facing various complex syntax errors.

[0095] Embodiment 2

[0096] Another embodiment of the present application discloses a system for generating a government data quality checking rule based on a large language model, thereby realizing the method for generating a government data quality checking rule based on a large language model in embodiment 1. The specific implementation mode of each module is referred to the corresponding description in embodiment 1. As shown in Figure 2 The system comprises:

[0097] A metadata extraction module 101 is configured to obtain metadata information of a target table to be checked and each target field in the target table.

[0098] The knowledge fragment retrieval module 102 is configured to sequentially take out each target field, acquire a field semantic vector, a joint semantic vector, a field structure vector and / or an enumeration value vector according to metadata information of the target table and the target field, and further retrieve a knowledge fragment of a historical field with the largest similarity from the government affair knowledge graph; the knowledge fragment includes metadata information of the historical field, a data quality checking rule associated with the historical field, a data standard and a data quality checking template;

[0099] The checking rule generation module 103 is configured to, when the knowledge fragment is not empty, construct a dynamic prompt word according to the metadata information of the target table and the target field and the knowledge fragment, and input the dynamic prompt word into a large language model to generate a data quality checking rule of the target field.

[0100] Since the system embodiment and the foregoing method of generating a government affair data quality checking rule based on a large language model are related, they can be mutually referred to, and thus repeated description is not given here. Since the system embodiment and the method embodiment have the same principles, the system embodiment also has the corresponding technical effects of the method embodiment.

[0101] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory or a random access memory.

[0102] The above is only a preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for generating government data quality verification rules based on a large language model, characterized in that: The following steps are involved: Obtain metadata information of the target table to be verified and each target field therein; Extract each target field in turn, obtain the field semantic vector, joint semantic vector, field structure vector, and / or enumeration value vector based on the metadata information of the target table and target field, and then retrieve the knowledge fragment of the historical field with the greatest similarity from the government knowledge graph; the knowledge fragment includes: metadata information of the historical field, data quality verification rules associated with the historical field, data standards, and data quality verification templates; When the knowledge fragment is not empty, a dynamic prompt word is constructed based on the metadata information of the target table and target field as well as the knowledge fragment, and then passed to the large language model to generate the data quality check rules for the target field; The field semantic vector is obtained by using the embedding model to obtain the embedding vector of the target field annotation; the joint semantic vector is obtained by using the embedding model to obtain the embedding vector of the target table annotation and the target field annotation; the field structure vector is obtained by using the embedding model to obtain the embedding vector of multiple metadata of the target field; the enumeration value vector is obtained by using the embedding model to obtain the embedding vector of the enumeration value list if the target field is an enumeration type; Each historical field in the government affairs knowledge graph has multiple indexes, wherein the first index is a weighted fusion vector calculated based on the joint semantic vector, field structure vector and / or enumeration value vector of the historical field and their respective weights; the second index is the joint semantic vector of the historical field; and the third index is the field semantic vector of the historical field. The step of retrieving the knowledge fragment of the historical field with the greatest similarity from the government affairs knowledge graph includes: calculating a weighted fusion vector of the target field based on the joint semantic vector, field structure vector and / or enumeration value vector of the target field, and their respective weights; The first similarity, second similarity and third similarity between the weighted fusion vector, joint semantic vector and field semantic vector of the target field and the first index, second index and third index of each historical field in the government knowledge graph are calculated in sequence. As long as the maximum value of the calculated similarity is greater than the similarity threshold, the corresponding knowledge fragment is obtained and the search is exited, including: if the maximum value of the first similarity is greater than the similarity threshold, the corresponding knowledge fragment is obtained, including: metadata information of the historical field corresponding to the first similarity maximum value, and its associated historical data quality verification rules; if the maximum value of the second similarity is greater than the similarity threshold, the corresponding knowledge fragment is obtained, including: metadata information of the historical field corresponding to the second similarity maximum value, and the field attribute consistency verification template in the data quality verification template; if the maximum value of the third similarity is greater than the similarity threshold, the corresponding knowledge fragment is obtained, including: data standards followed by the historical field corresponding to the third similarity maximum value, and data quality verification templates related to the data standards.

2. The method for generating government data quality check rules based on a large language model according to claim 1 is characterized in that: The metadata information of the target table and each target field therein includes: target table name, target table annotation, target field name, target field annotation, basic attributes and constraints of each target field; the basic attributes include: data type, length, precision and enumeration value list; the constraints include: whether it is a primary key, whether it is a foreign key, whether it is allowed to be empty, unique constraint and value range constraint.

3. The method for generating government data quality check rules based on a large language model according to claim 1 is characterized in that: The weight of the field structure vector and the initial weight of the enumeration value vector are dynamically adjusted according to the metadata information of the field. The formula is as follows: ， in, and Represent the adjusted weight and initial weight of the field structure vector respectively, and Represent the adjusted weight and initial weight of the enumeration value vector respectively, Indicates the enumeration coefficient; represents the weight growth coefficient, Indicates the number of non-zero values ​​in the field metadata.

4. The method for generating government data quality check rules based on a large language model according to claim 1 is characterized in that: The step of constructing dynamic prompt words based on metadata information of the target table and target field and knowledge fragments includes: Set the role as data quality check rule generation expert; If the knowledge fragment contains historical data quality verification rules, the task objective set for the role is to generate data quality verification rules for the current field in the table to be verified, referring to the historical fields and historical data quality verification rules. If the knowledge fragment contains data standards, the task objective set for the role is to generate verification rules for whether the current field in the table to be verified is consistent with the data standards, referring to the data standards and their related data quality verification templates. Otherwise, the task objective set for the role is to generate verification rules for whether the current field in the table to be verified is consistent with the historical field attributes, referring to the field attribute consistency verification template. The to-be-checked table and current field in the prompt word are set according to the metadata information of the target table and target field; and the reference information required for the task target in the prompt word is set according to the knowledge fragment.

5. The method for generating government data quality check rules based on a large language model according to claim 1 is characterized in that: The large language model is obtained by supervised fine-tuning training based on the base large model; the loss function of the large language model includes SQL generation loss and SQL syntax loss; the SQL generation loss is calculated using the cross-entropy loss function, and the SQL syntax loss is calculated by performing syntax error detection on the data quality check rules output by the large language model, based on the number and weight of each type of syntax error in the syntax detection results.

6. A system for generating government data quality verification rules based on a large language model, characterized by: include: The metadata extraction module is used to obtain metadata information of the target table to be verified and each target field therein; The knowledge fragment retrieval module is used to sequentially extract each target field, obtain the field semantic vector, joint semantic vector, field structure vector and / or enumeration value vector based on the metadata information of the target table and target field, and then retrieve the knowledge fragment of the historical field with the greatest similarity from the government knowledge graph; the knowledge fragment includes: metadata information of the historical field, data quality verification rules, data standards and data quality verification templates associated with the historical field; The verification rule generation module is used to construct dynamic prompt words based on the metadata information of the target table and target field and the knowledge fragment when the knowledge fragment is not empty, and then pass them into the large language model to generate data quality verification rules for the target field; In the knowledge fragment retrieval module, the field semantic vector is obtained by using the embedding model to obtain the embedding vector of the target field annotation; the joint semantic vector is obtained by using the embedding model to obtain the embedding vector of the target table annotation and the target field annotation; the field structure vector is obtained by using the embedding model to obtain the embedding vector of multiple metadata of the target field; the enumeration value vector is obtained by using the embedding model to obtain the embedding vector of the enumeration value list if the target field is an enumeration type; Each historical field in the government affairs knowledge graph has multiple indexes, wherein the first index is a weighted fusion vector calculated based on the joint semantic vector, field structure vector and / or enumeration value vector of the historical field and their respective weights; the second index is the joint semantic vector of the historical field; and the third index is the field semantic vector of the historical field. The step of retrieving the knowledge fragment of the historical field with the greatest similarity from the government affairs knowledge graph includes: calculating a weighted fusion vector of the target field based on the joint semantic vector, field structure vector and / or enumeration value vector of the target field, and their respective weights; The first similarity, second similarity and third similarity between the weighted fusion vector, joint semantic vector and field semantic vector of the target field and the first index, second index and third index of each historical field in the government knowledge graph are calculated in sequence. As long as the maximum value of the calculated similarity is greater than the similarity threshold, the corresponding knowledge fragment is obtained and the search is exited, including: if the maximum value of the first similarity is greater than the similarity threshold, the corresponding knowledge fragment is obtained, including: metadata information of the historical field corresponding to the first similarity maximum value, and its associated historical data quality verification rules; if the maximum value of the second similarity is greater than the similarity threshold, the corresponding knowledge fragment is obtained, including: metadata information of the historical field corresponding to the second similarity maximum value, and the field attribute consistency verification template in the data quality verification template; if the maximum value of the third similarity is greater than the similarity threshold, the corresponding knowledge fragment is obtained, including: data standards followed by the historical field corresponding to the third similarity maximum value, and data quality verification templates related to the data standards.

Images