Water conservancy data quality inspection rule automatic generation method and system based on large language model

Abstract

The application provides a water conservancy data quality inspection rule automatic generation method and system based on a large language model, which comprises the following steps: using a large language model to analyze documents of the water conservancy industry and extracting table structure information; automatically generating data quality inspection rules for the table structure information based on the extracted table structure information, wherein the data quality inspection rules cover multiple data quality dimensions; and converting the data quality inspection rules into standardized SQL statements. The application not only has a significant improvement in the degree of automation and generation efficiency, but also effectively solves the problem of disconnection between industry knowledge and technical implementation in traditional methods through a dynamic optimization mechanism, and meets the requirements for data quality inspection rule accuracy, flexibility and dynamic adaptability in complex water conservancy data governance scenarios.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of water conservancy data processing, and particularly relates to a water conservancy data quality inspection rule automatic generation method and system based on a large language model. BACKGROUND

[0002] In modern data governance, data quality management plays a key role in ensuring the reliability of business decisions and data accuracy. However, the design and generation of water conservancy data quality inspection rules still face many challenges, mainly in the following aspects:

[0003] 1. Limitations of manual design methods

[0004] Traditional methods rely on close cooperation between industry experts and programmers. Industry experts are usually familiar with business logic, but lack database technology background, making it difficult to convert business data logic into specific and executable quality inspection rules. Programmers, on the other hand, are proficient in database technology, but often lack understanding of specific business domain (such as water conservancy industry) business requirements. Due to the large difference in knowledge background between the two parties, cross-professional communication is frequent, and information transmission is prone to deviation, resulting in missing or inaccurate quality inspection rules designed, and also making the overall rule design process inefficient.

[0005] 2. Deficiencies of tool-assisted methods

[0006] Some existing tools assist in generating data quality inspection rules through templating. The basic idea is to extract field information from database table design documents and rely on pre-set templates to generate corresponding rules. Although this method improves efficiency to some extent, the template is fixed and the rule is rigid, making it difficult to adapt to complex business scenarios and dynamically changing table structures. The pre-set template cannot fully reflect the specific needs of various industries and scenarios, resulting in obvious limitations in flexibility and applicability of the generated rules.

[0007] 3. Disconnection between industry knowledge and technical implementation

[0008] In the current design process, there is a serious knowledge gap between industry experts and programmers. Industry experts have difficulty translating business requirements into technical rules, while programmers have difficulty accurately capturing subtle differences in business logic during technical implementation. This disconnection makes the final generated data quality inspection rules often unable to fully cover business requirements, thereby affecting the overall effectiveness of data governance.

[0009] In summary, the existing technology has problems such as high manual involvement, low efficiency, insufficient flexibility, and disconnection between industry knowledge and technical implementation in the design of water conservancy data quality inspection rules. These problems seriously hinder the improvement of water conservancy data quality management level and the realization of large-scale data governance needs. SUMMARY

[0010] The present application aims to solve the above-mentioned problems in the background art, and provides a water conservancy data quality inspection rule automatic generation method and system based on a large language model, which not only has a significant improvement in automation degree and generation efficiency, but also effectively solves the problem of disconnection between industry knowledge and technical implementation in traditional methods through a dynamic optimization mechanism, meeting the requirements for data quality inspection rule accuracy, flexibility and dynamic adaptability in complex water conservancy data governance scenarios.

[0011] The technical solution adopted by the present application is: a water conservancy data quality inspection rule automatic generation method based on a large language model, comprising the following steps:

[0012] Using a large language model to analyze documents of the water conservancy industry and extract table structure information;

[0013] Based on the extracted table structure information, automatically generating data quality inspection rules for the table structure information, the data quality inspection rules covering multiple data quality dimensions;

[0014] Converting the data quality inspection rules into standardized SQL statements.

[0015] In the above technical solution, the document is a Word document of the water conservancy industry, the table structure information includes table name, field identifier and field description, and the analysis process standardizes the table structure information into JSON format.

[0016] In the above technical solution, the analysis process further extracts chapter numbers and code table information of fields in the document.

[0017] In the above technical solution, the automatically generated data quality inspection rules cover at least the data quality dimensions of uniqueness, integrity, specification, validity and accuracy.

[0018] In the above technical solution, for time series data, the automatically generated data quality inspection rules include dynamic change inspection rules, which detect abnormal changes by analyzing the data change rate within a predetermined time window and the fluctuation range of the maximum and minimum values of the data.

[0019] In the above technical solution, for non-time series data, the automatically generated data quality inspection rules include threshold inspection rules, which automatically set reasonable value ranges in combination with the characteristics of water conservancy industry data.

[0020] In the above technical solution, the method further includes providing an interactive rule optimization function to enable users to adjust the logic of the data quality inspection rules according to actual needs.

[0021] In the technical solution, the method further comprises verifying the generated data quality inspection rule by using a SQL verification module to detect accuracy and coverage of the rule.

[0022] In the technical solution, the method further comprises storing the generated data quality inspection rule into a database and outputting a list of the data quality inspection rule in JSON format or table form.

[0023] The application provides a water conservancy data quality inspection rule automatic generation system based on a large language model, comprising:

[0024] A document parsing module is configured to parse water conservancy industry documents by using a large language model, extract table structure information, and generate corresponding JSON structured data.

[0025] A rule generation module is configured to generate data quality inspection rules based on the JSON structured data, wherein the rules cover multiple data quality dimensions, and convert the generated rules into standardized SQL statements.

[0026] A rule optimization module is configured to provide an interactive interface to enable a user to optimize and adjust the data quality inspection rules.

[0027] A rule verification module is configured to verify accuracy and coverage of the data quality inspection rules by using SQL.

[0028] A storage and output module is configured to store the data quality inspection rules into a database and output in JSON format or table form.

[0029] The application has the following advantages: the application uses a large language model to automatically parse table structure information from water conservancy industry documents and automatically generate multi-dimensional data quality inspection rules and standardized SQL statements. This not only greatly improves rule design efficiency and reduces manual intervention, but also effectively eliminates the knowledge gap between industry experts and database technicians, ensuring accuracy and consistency of the rule generation process.

[0030] Further, the application uses Word documents of the water conservancy industry as data sources and standardizes the extracted table structure information (including table name, field identifier, and field description) into JSON format, which ensures the uniformity and structure of information extraction. This provides a reliable and standardized data basis for subsequent automatic generation of data quality rules, reducing the error rate in the data preprocessing process.

[0031] Further, the application further extracts chapter number and field code table information based on standardized JSON data, enabling the generated quality inspection rules to fully consider the detailed information described in the document. This enables the rules to reflect a more comprehensive business background in the document, helping to generate more refined and adaptive inspection rules for specific business scenarios.

[0032] Further, the automatically generated data quality inspection rules of the application cover multiple data quality dimensions such as uniqueness, integrity, standardization, validity and accuracy, ensuring comprehensive monitoring and management of data quality from multiple angles. This not only enables the discovery of data anomalies, but also improves the comprehensiveness and reliability of data governance, providing solid data support for business decision-making.

[0033] Further, for time series data such as water level and flow, the generated dynamic change inspection rules can detect abnormal fluctuations in real time by analyzing the data change rate and the fluctuation of maximum and minimum values within a predetermined time window. This helps to achieve real-time monitoring and early warning, ensuring that data remains within a reasonable range during dynamic changes, thereby improving the real-time and accuracy of data monitoring.

[0034] Further, the threshold inspection rules generated by the application for non-time series data automatically set a reasonable value range based on the characteristics of the water conservancy industry, effectively filtering and excluding abnormal data. This not only adapts to the data characteristics in different business scenarios, but also ensures that the generated rules are highly matched with the actual business needs of water conservancy data, improving the scientificity and applicability of data quality management.

[0035] Further, the application provides an interactive rule optimization function, allowing users to provide feedback and adjust the automatically generated data quality inspection rules according to actual needs. This dynamic optimization mechanism compensates for the shortcomings between manual design and automatic generation, reducing the risk of errors caused by understanding bias, and enabling the rules to be continuously improved to meet the changing needs of complex water conservancy data governance scenarios.

[0036] Further, the application uses a SQL verification module to automatically verify the generated SQL rules, effectively detecting the accuracy and coverage of the generated rules. This verification mechanism ensures the correctness of the automatically generated rules in terms of syntax and logic, reducing the risk of data quality problems caused by rule design errors, and improving the robustness and reliability of the overall data governance system.

[0037] Further, the application stores the generated quality inspection rules in a structured manner in a database and supports output in JSON format or table form, facilitating review, management and subsequent maintenance. This not only standardizes rule management, but also facilitates cross-departmental communication and tracking of data quality inspection results, greatly improving the management efficiency of large-scale data governance. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments to facilitate a clear understanding of the present invention, but these descriptions do not constitute a limitation on the present invention.

[0040] Example 1

[0041] like Figure 1 As shown, this invention provides an automatic generation method for water conservancy data quality inspection rules based on a large language model. The method includes the following steps:

[0042] a) Extract table structure information from Word documents in the water conservancy industry. The table structure information includes table name, field identifier, field description, chapter number, and field code table information. Convert the information into a standardized JSON format.

[0043] b) Generate a predefined format prompt based on the table structure information in the standardized JSON format. The predefined format prompt contains all the instructions and parameters required to generate data quality inspection rules and incorporates expert knowledge in the field of water conservancy.

[0044] c) Submit the generated predefined format prompts to the large language model to generate initial data quality check rules. The rules cover check types such as uniqueness, completeness, standardization, validity and accuracy. Among them, dynamic change check rules are generated for time series data (such as water level and flow rate), including 48-hour data change rate analysis and maximum and minimum value fluctuation detection. Threshold check rules are generated for non-time series data to automatically set reasonable value ranges. The generated rules are directly converted into standardized SQL statements.

[0045] d) Use the SQL validation module to automatically validate the generated standardized SQL statements to ensure that the SQL statements meet business requirements in terms of syntax and logic;

[0046] e) Provide an interactive interface that allows users to review and provide feedback on initially generated rules that have failed verification or have passed verification but have additional setting requirements, and adjust the predefined format prompts based on user feedback;

[0047] f) Repeat step (ce) to form a cycle of generation-feedback-optimization-regeneration until the generated data quality check rules meet the predetermined business requirements;

[0048] g) Store the final generated data quality inspection rules, including field information, rule descriptions, and corresponding SQL statements, in the database, and support output in JSON or tabular format.

[0049] Specifically, step a) further includes:

[0050] The Word document (containing table structure design and field descriptions) was converted to Markdown format, and the Markdown content was segmented based on heading level symbols to facilitate the extraction of table structure information and field descriptions related to water conservancy data. The extracted data was then converted to JSON format for subsequent processing.

[0051] The standardized JSON format includes the following fields:

[0052] Table name, table identifier, chapter number;

[0053] The information for each field in the table includes the field name, field identifier, data type and length, whether null values ​​are allowed, unit of measurement, primary key number, and field description (where the field description includes code table information).

[0054] Below is a sample Prompt for converting document slices to table-based JSON:

[0055] Please strictly adhere to the following JSON format when returning results, and do not include any other text.

[0056] If this is not a table structure design document, return:

[0057] {{

[0058] "is_table_structure":false,

[0059] "message": "Non-table structure design document"

[0060] }}

[0061] If it is a table structure design document, return:

[0062] {{

[0063] "is_table_structure":true,

[0064] "tables":[

[0065] {{

[0066] "table_name":"table_name",

[0067] "table_identifier":"table identifier",

[0068] "chapter_ID":"chapter number",

[0069] "fields":[

[0070] {{

[0071] "field_name":"field name",

[0072] "field_identifier":"field identifier",

[0073] "field_type":"type and length",

[0074] "nullable":"Y / N",

[0075] "unit":"unit of measurement",

[0076] "key":"primary key number",

[0077] "field_description":"field description (including code table information)"

[0078] }}

[0080] }}

[0082] }}

[0083] Notes:

[0084] 1. When extracting field descriptions, if the field involves a code table, convert the code table to Chinese description and extract the code table information completely.

[0085] 2. The chapter number should be taken from the chapter number before the table name, such as "6.1.1 Precipitation Monitoring Table", then the chapter number is "6.1.1 section".

[0086] 3. Features of table structure design document:

[0087] - Contains table name and table identifier

[0088] - Contains field definition table

[0089] - Contains field description section

[0090] - Contains code table information and threshold description

[0091] ​​Specifically, in step b) of generating a predefined format prompt, the predefined format prompt explicitly requires the large language model to generate data quality inspection rules based on the characteristics of water conservancy industry data, and requires that the rules cover five types during generation: uniqueness, completeness, standardization, validity, and accuracy. The data types of the quality inspection rules are established as needed, and a large amount of expert knowledge is set, including time-series rules responding to non-time-series rules. The rule content includes field names, inspection types, rule descriptions, and SQL implementations.

[0092] Specifically, in step c):

[0093] For time-series data, the generated data quality inspection rules include dynamic change inspection rules, which detect anomalies by analyzing the data change rate and fluctuations between the maximum and minimum values ​​over 48 hours.

[0094] For non-time-series data, the generated data quality inspection rules include threshold inspection rules. These threshold inspection rules automatically set a reasonable value range based on water conservancy characteristics (such as water level and flow rate). The threshold is adjusted according to the unit of measurement of the field in the JSON.

[0095] Below is an example of a prompt message for quality checks when converting table JSON information to JSON (the prompt messages for different types of water conservancy data need to be refined and revised):

[0096] Please generate one or more data quality check rules for the fields in the table structure (no check rules are required for primary key fields), and return the results in the following format:

[0097] json

[0098] {

[0099] "rules":[

[0100] {{

[0101] "field_name":"Field name",

[0102] "field_identifier":"Field identifier",

[0103] "checks":[

[0104] {{

[0105] "check_subject":"uniqueness",

[0106] "description":"Checks if a field is unique."

[0107] "sql":"SELECT...FROM{table_identifier}WHERE{field_identifier}=?"

[0108] }},

[0109] {{

[0110] "check_subject":"completeness",

[0111] "description":"Check if the field is empty."

[0112] "sql":"SELECT...FROM{table_identifier}WHERE{field_identifier}ISNULL"

[0113] }},

[0114] {{

[0115] "check_subject":"standardization",

[0116] "description":"Check if the field format meets the requirements."

[0117] "sql":"SELECT...FROM{table_identifier}WHERE{field_identifier}NOTLIKE?"

[0118] }} ]

[0120] }} ]

[0122] }

[0123] Require:

[0124] 1. Check Types: Field check types include one of the following five: uniqueness, completeness, standardization, validity, and accuracy. Each check can have multiple rules.

[0125] 2. Timing rules:

[0126] For numerical monitoring indicators containing "water level", "flow", "precipitation" and other fields in the field name, time series rules must be included (check_subject is "validity").

[0127] Time series rule 1: Analyze the latest 48-hour records and changes, and calculate the data change rate before and after 24 hours.

[0128] Time series rule 2: Check the difference between the maximum and minimum values within 48 hours, and consider it abnormal if it exceeds the preset common sense.

[0129] 3. Non-time series rules: If the field description contains threshold values, code tables or specific rules, they must be checked as constraints and implemented in SQL.

[0130] Code table judgment 1: If the field description 'description' contains code constraint description, it must be considered as a rule constraint, and the description of the code reference content should be retained in 'description' and implemented in SQL statements. (For example, confidence description: When there are multiple sources of the same type of monitoring site, the credibility of the site data. 1 represents the most credible site, 0 represents the over-protected site, and 2 represents others. The description should be retained in full).

[0131] Code table judgment 2: Only records that are not empty need to be detected, and optimization quality inspection rules such as SQL should be paid attention to, such as checking the value of the flow measurement method field to ensure it is one of the values specified in the code table.

[0132] Threshold judgment 1: For "validity check" of monitoring data, as a water conservancy expert, try to set specific threshold values for numerical monitoring data items based on the geographical, climatic and hydrological characteristics of XX province (such as water level cannot exceed 2000 meters above sea level in XX province, and cannot be less than -30 meters; flow cannot be greater than 80000 cubic meters per second, and cannot be less than -80000 cubic meters per second, where negative numbers represent reverse flow; the maximum rainwater collection area of a hydrological station is not more than 360000 square kilometers; the value range of cross-section water area is limited to 0 to 25000 square meters.), to exclude abnormal value data. However, pay attention to the measurement unit of the field in the JSON, and set the value range to be consistent with the measurement unit "unit".

[0133] Threshold judgment 2: If the field description 'description' contains threshold constraints, it must be considered as a rule constraint, and the specific threshold constraint content should be described and implemented in SQL statements.

[0134] Do not set null value judgment easily: {nullable} is not N, do not set null value judgment rules arbitrarily (such as monitoring method and monitoring indicators), unless several fields must exist at the same time, and association null value check rules can be set.

[0135] Water experts experience that need to set the quality inspection rules.

[0136] The collection, creation time field such as "TM" and "create_time" can not exceed the current time.

[0137] SQL requirements: SQL statements must refer to field identifiers (field_identifier) and table identifiers (table_identifier), and should use English lowercase.

[0138] Specifically, in step d), the initial generated data quality inspection rules are stored in the database according to table name, field name, check type, etc. The storage format includes table name, table identifier, field name, field identifier, quality inspection type, quality inspection Chinese description and SQL statement.

[0139] The following is an example code for storing rules to the database.

[0140] def store_rules_to_db (rules, connection) :

[0141] cursor = connection.cursor()

[0142] for rule in rules:

[0143] cursor.execute("""

[0144] INSERT INTO data_quality_rules

[0145] (field_name, check_subject, description, sql)

[0146] VALUES (%s, %s, %s, %s);

[0147] "", (rule['field_name'], rule['check_subject'], rule['description'], rule['sql']))

[0148] connection.commit()

[0149] Then the accuracy of the SQL verification rule is automatically invoked to ensure that the logic meets the business requirements. The system automatically executes the corresponding SQL statement to calculate the proportion of errors under a specific data quality check rule. For the current data quality type or indicator being checked, the error rate is calculated for each specific data quality dimension (e.g. uniqueness, completeness, specification, validity or accuracy) set by the rule. For example, for completeness checking, the system generates a SQL statement to count the proportion of missing values to obtain the data error rate under the rule.

[0150] Specifically, the step e) of providing an interactive interface enables the user to provide feedback on the initially generated data quality check rule based on the error rate calculated in step d), and to modify the generated prompt in the predefined format accordingly, thereby achieving dynamic optimization of the prompt in the predefined format and achieving the generated rule to the predetermined accuracy and business requirements through multiple iterations.

[0151] Specifically, in step g), the generated data quality check rule is structured and sorted according to information such as field name, check type, rule description and SQL statement before storage, and is stored in a database. At the same time, it supports outputting the complete rule list in JSON format or table form such as Excel, so that water experts or other users can review the automatically generated rule list, facilitating review and management.

[0152] Embodiment 2

[0153] The present application provides a water conservancy data quality inspection rule automatic generation system based on a large language model, which comprises:

[0154] A document parsing module for converting a Word document of the water conservancy industry into Markdown format and extracting table structure information to generate standardized JSON format data;

[0155] A Prompt generation module for generating a Prompt containing water expert knowledge based on the JSON format data;

[0156] A large language model module for receiving the generated Prompt and automatically generating data quality check rules, including corresponding SQL statements;

[0157] An interactive interface module for enabling user feedback on the generated rules and dynamically optimizing the Prompt, thereby forming a cycle of generation-feedback-optimization-regeneration;

[0158] An SQL verification module for automatically verifying the correctness of the generated SQL statements; and

[0159] A storage and output module is configured to store the generated data quality inspection rule in a database in a structured manner and support output in JSON or table form.

[0160] Specifically, the document parsing module includes:

[0161] A word2md document conversion module is configured to convert the content of paragraphs and tables in a Word document into Markdown format and support automatic parsing of title levels.

[0162] An md_splitter content segmentation module is configured to segment Markdown content according to title level symbols (such as #) and extract relevant table structure information.

[0163] An md2json JSON generation module is configured to parse Markdown content through a large language model, automatically extract table name, field identifier, field type, whether it can be empty, and other information, and generate JSON format.

[0164] Specifically, the Prompt generation module includes an input and preprocessing module, an expert knowledge integration and template design module, and a Prompt generation module.

[0165] The input and preprocessing module receives standardized JSON format data generated by the document parsing module, which includes table name, field identifier, field description, chapter number, and field code table information extracted from water conservancy industry documents.

[0166] The preprocessing process of the input and preprocessing module includes preprocessing the JSON data, organizing and summarizing the basic information of each field, and providing structured basis for generating Prompt.

[0167] The expert knowledge integration and template design module includes expert knowledge integration and template design.

[0168] The expert knowledge integration is to combine the knowledge provided by water conservancy industry experts to clearly define the requirements of each data quality inspection type, including:

[0169] Inspection types: uniqueness, integrity, specification, validity, and accuracy.

[0170] Time series rules: For time series data such as water level, flow, and precipitation, dynamic change inspection rules are generated, such as data change rate within 48 hours and abnormal fluctuation detection.

[0171] Non-time series rules: For other data, threshold inspection rules are generated, with reasonable ranges automatically set based on water conservancy characteristics, and thresholds dynamically adjusted based on the measurement units of fields in JSON.

[0172] Template design is to design a set of pre-defined format of prompt templates, which explicitly require large language models to generate data quality check rules based on the above expert knowledge. The template content includes:

[0173] Task description: The generated rules cover field name, check type, rule description and SQL implementation.

[0174] Data background: Contains table structure data, field specific information and water conservancy business scenario background.

[0175] Rule logic requirements: Detailed description of time series rules (such as 48-hour change rate, abnormal fluctuation) and non-time series rules (such as threshold detection) generation conditions.

[0176] Prompt generation module is used to automatically generate Prompt: according to the template and pre-processed JSON data, automatically assemble the Prompt text containing water conservancy expert knowledge, and provide clear instructions for subsequent large language model rule generation.

[0177] Example description: For example, the Prompt will clearly indicate "Please generate data quality check rules covering uniqueness, integrity, specification, validity and accuracy, among which 48-hour change rate and abnormal fluctuation detection rules are generated for time series data; Threshold check rules are generated for non-time series data, and the threshold is adjusted according to the measurement unit of the data".

[0178] Specifically, the large language model module includes receiving and analyzing the Prompt module, rule generation processing module and output and feedback module

[0179] Among them, the receiving and analyzing Prompt module: receives the pre-defined format of prompt output by the Prompt generation module, which contains detailed task requirements, data background and rule generation logic. Use large language model to deeply analyze Prompt, understand water conservancy business background, each check type requirement and the details of time series and non-time series rule generation.

[0180] Rule generation processing module automatically generates data quality check rules based on the analysis results, including:

[0181] Check type rule: Generate check rules covering uniqueness, integrity, specification, validity and accuracy.

[0182] Time series rule generation: For fields containing water level, flow, precipitation, etc., generate dynamic change rules to detect anomalies by analyzing 48-hour data change rate and maximum and minimum value fluctuation.

[0183] Non-temporal rule generation: For other data fields, generate threshold check rules, automatically set reasonable value ranges that meet water conservancy characteristics and adjust according to measurement units.

[0184] Rule formatting: The generated rules are arranged in a standardized format, usually structured JSON data, with corresponding SQL statements, ensuring that the generated results can be directly used for data quality inspection execution.

[0185] The output and feedback module outputs the generated rules (including field name, check type, rule description, and SQL implementation) to the downstream module for subsequent verification, storage, and optimization. After user feedback, the large language model module can generate revised rules based on the adjusted Prompt, implementing a cycle of generation-feedback-optimization-regeneration, ensuring that the rules are constantly improved to meet actual business needs.

[0186] Specifically, the interactive interface module aims to provide an interactive platform for users to review automatically generated water conservancy data quality inspection rules and provide feedback on rule logic. After user feedback, the system dynamically adjusts the prompt (Prompt) for generating rules based on feedback, implementing a cycle of generation-feedback-optimization-regeneration.

[0187] The implementation scheme of the interactive interface module of this embodiment is as follows:

[0188] Users can view the generated rules and corresponding SQL statements through a graphical interface or web interface. The interface supports inputting feedback, such as adjusting suggestions for rule descriptions, check logic, timing rules, or threshold settings.

[0189] After receiving user feedback, the system automatically adjusts the original pre-defined format prompt, incorporating user-proposed modifications, and updates the expert knowledge base related to water conservancy data characteristics.

[0190] The modified Prompt will be submitted to the large language model again to generate a new rule version. This process continues until the generated rules fully meet business needs.

[0191] The interactive interface module significantly reduces errors caused by cross-professional communication, shortens the rule optimization cycle, improves the accuracy and applicability of generated rules, and ensures that the generated rules truly reflect the business logic of the water conservancy industry. It forms a dynamic and adaptive rule generation closed loop, continuously improving the quality of quality inspection rules.

[0192] Specifically, the main task of the SQL verification module is to automatically call and execute the SQL statements generated by the large language model to verify the correctness and logical consistency of the SQL statements in the actual data environment.

[0193] The implementation scheme of the SQL verification module in this embodiment is as follows:

[0194] The system automatically extracts the SQL statements contained in the generated rules and executes them in the test database environment. By executing the SQL statements, the system calculates the data error rate under the quality inspection rule (for example, for integrity checking, the proportion of missing records is counted) to determine whether the rule logic meets the expectations. The execution results are compared with the expected rule effects. If it is found that the SQL statements have syntax errors or logical vulnerabilities, error reports or prompt information are automatically generated for reference by users and developers to further optimize or regenerate the rules.

[0195] The SQL verification module ensures that the automatically generated rules are correct and executable in terms of technical implementation, reduces the risk of data quality inspection failure caused by SQL errors, and improves the robustness and reliability of the overall data governance system.

[0196] Specifically, the storage and output module is responsible for storing the generated water conservancy data quality inspection rules in a structured format into the database, and supports rule list export in multiple formats (such as JSON or table), facilitating subsequent management, review and use.

[0197] The implementation scheme of the storage and output module in this embodiment is as follows:

[0198] The storage and output module includes a rule storage module and a rule output module:

[0199] The rule storage module stores the generated rules into the database according to information such as table name, field name, inspection type, rule description and SQL statement.

[0200] The rule output module provides the function of exporting complete rule lists in JSON format or Excel (.xlsx) format. The exported content includes field name, rule type, rule description and SQL statement, facilitating water conservancy experts or other users to review and share the automatically generated rules.

[0201] The storage and output module realizes long-term structured storage and efficient management of rules, facilitates subsequent data quality monitoring work, supports multiple output formats to meet the rule review and application needs in different use scenarios, improves cross-department collaboration efficiency, and ensures the standardization and convenient sharing of data quality inspection rules.

[0202] Embodiment 3

[0203] The application provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned technical scheme of the water conservancy data quality inspection rule automatic generation method based on a large language model.

[0204] Embodiment 4

[0205] The application provides an electronic device, comprising a memory and a processor, which are connected in communication with each other, the memory stores computer instructions, and the processor executes the computer instructions to implement the method for automatically generating water conservancy data quality inspection rules based on a large language model.

[0206] The application introduces a large model technology to build an intelligent system that can understand the business logic of industry experts and handle technical details of database table structures, thereby achieving automatic generation of data quality inspection rules. The specific goals are as follows:

[0207] (1) Improve efficiency and reduce manual intervention

[0208] The application automatically parses industry design documents (such as WORD format) through a large model, automatically extracts table structure and field information, combines pre-set water conservancy expert knowledge information, realizes the generation of quality inspection rules from design documents, and stores the quality inspection rules in the database according to the set format. Through intelligent means, the efficiency of generating water conservancy data quality inspection rules is greatly improved, solving the problem of manual complexity and low efficiency in the prior art.

[0209] (2) Improve flexibility and adaptability

[0210] The application can automatically generate corresponding quality inspection rules according to water conservancy business requirements and table structures through the powerful understanding ability of a large language model, and can quickly adjust the rules when business requirements change to meet the needs of dynamic data governance. The system can automatically adapt to changes in various database types and table structures, improving the adaptability and flexibility of rule generation.

[0211] (3) Bridge the gap between industry knowledge and technical implementation

[0212] One of the purposes of the application is to bridge the gap between industry knowledge and database technology implementation through the introduction of a large language model technology. The large model can automatically parse the provided water conservancy business knowledge and database design documents, deeply understand the business logic and data constraints, and automatically generate quality inspection rules that meet technical standards based on these understandings. This process eliminates misunderstandings and omissions in manual conversion, ensuring that technical personnel can accurately generate high-quality data quality inspection rules based on business requirements, thereby realizing seamless integration of industry knowledge and technical implementation.

[0213] The application is based on a large language model, which automatically extracts table structure information from Word documents in the water conservancy industry, including table name, field identifier, field description, etc., and generates standardized JSON format data, supporting chapter number extraction and field code table information parsing.

[0214] Based on the parsed table structure JSON data, the application automatically generates data quality inspection rules, covering five types of uniqueness, integrity, standardization, validity and accuracy. For time series data (such as water level and flow), dynamic change inspection rules are generated, including 48-hour data change rate analysis and maximum and minimum value fluctuation range detection. Non-time series rules such as threshold checking automatically set reasonable ranges based on water conservancy characteristics to ensure the accuracy and industry applicability of the generated rules. The generated rules are directly converted into standardized SQL statements and consistent with the rule description, meeting the data quality management requirements.

[0215] The application provides interactive rule optimization function, which supports users to dynamically adjust the generated quality inspection rule logic according to actual needs, ensuring that the rules are highly matched with business scenarios. At the same time, combined with the SQL verification module, the accuracy and coverage of the rules are automatically detected to ensure that the generated quality inspection rules are logically reasonable and feasible, and meet the high standard requirements of data governance.

[0216] The application realizes the structured storage of generated rules, including field information, rule description and SQL statement, supports rule list to be standardized output in JSON or table form, and facilitates review and management.

[0217] Through modular design, the application builds a full-process automation system from document parsing to rule generation, verification, storage and output, significantly improving the efficiency and adaptability of rule generation.

[0218] Compared with the prior art, the application significantly improves the efficiency and accuracy of water conservancy data quality inspection rule generation. The prior art mainly relies on manual cooperation of industry experts and programmers, which is tedious and prone to errors. The application introduces a large language model, combined with part of the industry knowledge provided by experts, to automatically complete table structure parsing and rule generation. The large model can be expanded based on expert knowledge to generate more comprehensive and accurate quality inspection rules, thereby greatly reducing the depth and frequency of manual intervention and greatly improving efficiency.

[0219] Compared with the traditional template method, the application has higher flexibility in generating dynamic time series rules and non-time series rules, and can quickly expand to generate scientific and reasonable quality inspection rules combined with a small amount of industry characteristic knowledge. The system not only quickly responds to changes in business needs, but also significantly reduces the difficulty of cross-professional communication, meeting the efficiency and dynamics requirements in complex water conservancy data governance scenarios.

[0220] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, the methods can be tangibly embodied in a machine-readable storage medium having stored thereon instructions that can be used to program a computer to perform any of the operations described herein. The software implementation can be for example, in the form of a computer program product. The software implementation can be implemented in a centralized fashion in one computer system or network, or be distributed over a network such computer systems, some of which can implement the

[0221] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or blocks can be implemented by computer program instructions. Figure 1 means for carrying out functions specified in the flowchart block or blocks.

[0222] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or blocks can be implemented by computer program instructions. Figure 1 means for carrying out functions specified in the flowchart block or blocks.

[0223] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or blocks can be implemented by computer program instructions. Figure 1 means for carrying out functions specified in the flowchart block or blocks.

[0224] The embodiments of the present application described above are illustrative, and not restrictive, of the present application. Various modifications can be made to the embodiments of the present application by persons of ordinary skill in the art without departing from the spirit and scope of the present application as defined by the following claims. Therefore, the scope of the present application should be determined by the following claims.

[0225] That which is not described in detail in the specification is considered to be of prior art to those skilled in the art.

Claims

1. A large language model-based water conservancy data quality inspection rule automatic generation method, characterized by: The method comprises the following steps: parsing a Word document in the water conservancy industry using a large language model to extract table structure information, the table structure information including table name, field identifier, and field description; the parsing process includes: converting the paragraphs and table contents in the Word document into Markdown format, supporting automatic parsing of title levels; dividing the Markdown content according to the title level symbols to extract relevant table structure information; parsing the Markdown content through a large language model to automatically extract table name, field identifier, field type, and whether the field can be empty information, and generate JSON format; and the parsing process further extracts chapter numbers and field code table information in the document; based on the extracted JSON format table structure information containing chapter numbers and field code table information, automatically generating JSON format data quality check rules for the table structure information, the data quality check rules covering multiple data quality dimensions, and for time series data, the automatically generated data quality check rules include dynamic change check rules, which detect abnormal changes by analyzing the data change rate within a predetermined time window and the fluctuation range of the maximum and minimum values of the data; convert the data quality check rules into standardized SQL statements.

2. The method of claim 1, wherein: The data quality dimensions covered by the automatically generated data quality check rules include at least uniqueness, integrity, specification, validity, and accuracy.

3. The method of claim 1, wherein: For non-time series data, the automatically generated data quality check rules include threshold check rules that automatically set reasonable value ranges in combination with the characteristics of water conservancy industry data.

4. The method of claim 1, wherein: The method further includes providing an interactive rule optimization function to enable users to adjust the logic of the data quality check rules according to actual needs.

5. The method of claim 1, wherein: The method further includes using a SQL verification module to verify the generated data quality check rules to detect the accuracy and coverage of the rules.

6. The method of claim 1, wherein: The method further includes storing the generated data quality check rules in a database and outputting a list of the data quality check rules in JSON format or table form.

7. A large language model-based water conservancy data quality inspection rule automatic generation system, characterized in that, For implementing the method of any one of claims 1-6, comprising: a document parsing module for parsing water conservancy industry documents using a large language model to extract table structure information and generate corresponding JSON structured data; a rule generation module for generating data quality check rules based on the JSON structured data, the rules covering multiple data quality dimensions, and converting the generated rules into standardized SQL statements; a rule optimization module for providing an interactive interface for users to optimize and adjust the data quality check rules; a rule verification module for verifying the accuracy and coverage of the data quality check rules using SQL; and a storage and output module for storing the data quality check rules in a database and outputting them in JSON format or table form.

Images