Intelligent data comparison rule generation system and method based on natural language

Abstract

The invention discloses an intelligent data comparison rule generation system and method based on a natural language, and relates to the technical field of data comparison. In order to solve the problems of high technical threshold and low efficiency in a traditional data comparison rule generation method, the adopted scheme comprises a comparison resource definition module used for defining a comparison resource data structure; the comparison task writing module is used for determining the source, field, range and resource relationship of data to be compared, storing related information into a knowledge base, and storing metadata obtained by natural splitting and involved in task writing into a vector library; the comparison rule generation module provides a visual interface and is used for converting the natural language description input by the user through the visual interface into an executable comparison rule; and the comparison task execution module is used for executing the comparison task based on the comparison rule, and storing or visually displaying the comparison result after obtaining the comparison result. The method is applied to scenes of data comparison, quality detection, consistency verification and the like.

Description

Technical Field

[0001] This invention relates to the field of data comparison technology, specifically to an intelligent data comparison rule generation system and method based on natural language. Background Technology

[0002] In recent years, with the development of Natural Language Processing (NLP) technology, especially the emergence of large models (such as GPT and BERT), it has become possible to describe tasks using natural language and automatically generate comparison rules. However, traditional data comparison methods usually rely on manually writing or configuring complex comparison rules, which has the following problems: users need to have high coding skills and database knowledge, making it difficult to get started quickly; manually configuring comparison rules is time-consuming and error-prone, especially when dealing with multi-table comparisons or complex conditions; it is difficult to dynamically adapt to different comparison tasks and scenarios, and cannot meet diverse business needs; accurately understanding natural language descriptions and generating efficient comparison rules remains a technical challenge. Summary of the Invention

[0003] This invention addresses the problems of high technical barriers, low efficiency, and poor flexibility in traditional data comparison rule generation methods by providing an intelligent data comparison rule generation system and method based on natural language. It automatically generates comparison rules by describing the comparison task in natural language, simplifying the configuration and execution of data comparison tasks.

[0004] In a first aspect, the present invention provides an intelligent data comparison rule generation system based on natural language, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0005] An intelligent data comparison rule generation system based on natural language, the structure of which includes:

[0006] The comparison resource definition module is used to define the comparison resource data structure, which serves as the basis for subsequent comparison rule generation.

[0007] The comparison task drafting module is used to clarify the source, fields, scope and relationship between comparison resources of the data to be compared, store relevant information in the knowledge base, and store the metadata obtained by natural splitting involved in the process of drafting the comparison task in the vector library so that the metadata can be called in the future to support the generation of comparison rules or the execution of tasks through vector retrieval.

[0008] The comparison rule generation module provides a visual interface for converting natural language descriptions input by users through the visual interface into executable comparison rules;

[0009] The comparison task execution module executes comparison tasks based on executable comparison rules, and then stores or visualizes the comparison results.

[0010] Optionally, the resource data structure defined by the resource definition module can be compared, including table name, field name, field type, primary key, and foreign key information.

[0011] Further optionally, the comparison rule generation module involved specifically includes:

[0012] The user input unit provides a visual interface for users to input natural language descriptions according to their needs.

[0013] The Natural Language Parsing Unit (NLP) is used to extract key information from user input using the BERT model. The task is defined as a sequence labeling task. First, the natural language description is segmented using a word segmenter. Then, each word or sub-word after segmentation is labeled to generate a label sequence, which is then converted into an input format that is compatible with the BERT model.

[0014] The rule generation unit is used to form executable comparison rules based on the extracted key information.

[0015] Preferably, the natural language parsing unit uses the BERT model to extract key information from user input, including table name, query field name, related field name, and comparison conditions;

[0016] Based on the extracted key information, the rule generation unit generates structured comparison rules that include task description, data source, fields, comparison conditions, and inter-table relationships.

[0017] Optionally, the comparison task execution module involved may specifically include:

[0018] The rule submission unit is used to submit executable comparison rules to the unified task scheduling unit.

[0019] The unified task scheduling unit is used to configure the task execution method (timed execution or immediate execution) according to the comparison rules. It is also used to match and compare related data based on the data source, fields, comparison conditions and inter-table relationships in the comparison rules, then obtain the comparison results and store the results in the specified location according to the configuration. It also supports displaying the results in the form of charts or tables through visualization tools.

[0020] Secondly, the present invention provides an intelligent data comparison rule generation method based on natural language, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0021] A method for generating intelligent data comparison rules based on natural language includes the following steps:

[0022] S1. Define the comparison resource data structure as the basis for subsequent comparison rule generation;

[0023] S2. Clarify the source, fields, scope, and relationships between comparison resources of the data to be compared, store the relevant information in the knowledge base, and store the metadata obtained by natural splitting involved in the process of drafting the comparison task in the vector library so that these metadata can be called in the future through vector retrieval to support the generation of comparison rules or the execution of tasks.

[0024] S3. Provides a visual interface to convert the natural language descriptions entered by users through the visual interface into executable comparison rules;

[0025] S4. Execute the comparison task based on the executable comparison rules, and store or visualize the comparison results.

[0026] Optionally, step S1 is executed, and the defined resource data structure includes table name, field name, field type, primary key and foreign key information.

[0027] Alternatively, step S3 may specifically include the following operations:

[0028] Users can input natural language descriptions through a visual interface according to their needs;

[0029] Using the BERT model, the task of extracting key information from user input is defined as a sequence labeling task. First, the natural language description is segmented using a word segmenter, and then each word or sub-word after segmentation is labeled to generate a label sequence, which is then converted into an input format that is compatible with the BERT model.

[0030] Based on the extracted key information, executable comparison rules are formed.

[0031] Preferably, the key information extracted from user input includes table name, query field name, related field name, and comparison conditions;

[0032] Based on the extracted key information, structured comparison rules are formed, which include task description, data source, fields, comparison conditions, and relationships between tables.

[0033] Alternatively, step S4 may specifically include the following operations:

[0034] The executable comparison rules are submitted to the unified task scheduling system, which then configures the task execution method based on the comparison rules, including scheduled execution or immediate execution.

[0035] The unified task scheduling system matches and compares related data based on the data source, fields, comparison conditions, and inter-table relationships in the comparison rules;

[0036] The unified task scheduling system acquires the comparison results and stores them in a specified location according to the configuration. It also supports displaying the results in the form of charts or tables through visualization tools.

[0037] The present invention provides an intelligent data comparison rule generation system and method based on natural language, which has the following advantages compared with the prior art:

[0038] 1. This invention utilizes natural language processing technology to achieve the automatic generation and execution of data comparison rules. It has advantages such as intelligence, efficiency, flexibility, and scalability, and can be widely applied to scenarios such as data comparison, data quality detection, and data consistency verification, significantly improving the efficiency and accuracy of data processing.

[0039] 2. This invention uses natural language processing technology to automatically convert user-input natural language descriptions into data comparison rules and generate executable comparison logic, simplifying the configuration and execution of data comparison tasks and solving the problems of high technical threshold, low efficiency, and poor flexibility in traditional data comparison rule generation methods. Attached Figure Description

[0040] Appendix Figure 1 This is a module connection block diagram of Embodiment 1 of the present invention;

[0041] Appendix Figure 2 This is a flowchart of the method in Embodiment 2 of the present invention. Detailed Implementation

[0042] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0043] Example 1:

[0044] Combined with appendix Figure 1 This embodiment proposes an intelligent data comparison rule generation system based on natural language, the structure of which includes:

[0045] The comparison resource definition module is used to define the comparison resource data structure, including table name, field name, field type, primary key and foreign key information, which serve as the basis for subsequent comparison rule generation;

[0046] The comparison task drafting module is used to clarify the source, fields, scope and relationship between comparison resources of the data to be compared, store relevant information in the knowledge base, and store the metadata obtained by natural splitting involved in the process of drafting the comparison task in the vector library so that the metadata can be called in the future to support the generation of comparison rules or the execution of tasks through vector retrieval.

[0047] The comparison rule generation module provides a visual interface for converting natural language descriptions input by users through the visual interface into executable comparison rules;

[0048] The comparison task execution module executes comparison tasks based on executable comparison rules, and then stores or visualizes the comparison results.

[0049] In this embodiment, the comparison rule generation module specifically includes:

[0050] The user input unit provides a visual interface for users to input natural language descriptions according to their needs.

[0051] The Natural Language Parsing Unit (NLP) is used to extract key information from user input using the BERT model. This task is defined as a sequence labeling task. First, the natural language description is segmented using a word segmenter. Then, each word or sub-word after segmentation is labeled to generate a label sequence, which is then converted into an input format adapted to the BERT model. The key information extracted includes table names, query field names, related field names, and comparison conditions.

[0052] The rule generation unit is used to generate executable comparison rules based on the extracted key information, including task description, data source, fields, comparison conditions, and inter-table relationships.

[0053] In this embodiment, the comparison task execution module specifically includes:

[0054] The rule submission unit is used to submit executable comparison rules to the unified task scheduling unit.

[0055] The unified task scheduling unit is used to configure the task execution method (timed execution or immediate execution) according to the comparison rules. It is also used to match and compare related data based on the data source, fields, comparison conditions and inter-table relationships in the comparison rules, then obtain the comparison results and store the results in the specified location according to the configuration. It also supports displaying the results in the form of charts or tables through visualization tools.

[0056] Example 2:

[0057] Combined with appendix Figure 2 This embodiment proposes an intelligent data comparison rule generation method based on natural language, which includes the following steps:

[0058] S1. Define the comparison resource data structure, including table name, field name, field type, primary key and foreign key information, as the basis for subsequent comparison rule generation.

[0059] S2. Clarify the source, fields, scope, and relationships between comparison resources of the data to be compared, store the relevant information in the knowledge base, and store the metadata obtained by natural splitting involved in the process of drafting the comparison task in the vector library so that these metadata can be called in the future to support the generation of comparison rules or the execution of tasks through vector retrieval.

[0060] S3. Provide a visual interface to convert the natural language descriptions entered by the user into executable comparison rules, including the following operations:

[0061] Users can input natural language descriptions through a visual interface according to their needs;

[0062] Using the BERT model, the task of extracting key information from user input is defined as a sequence labeling task. First, the natural language description is segmented using a word segmenter. Then, each word or sub-word after segmentation is labeled to generate a label sequence, which is converted into an input format adapted to the BERT model. The key information extracted includes table name, query field name, related field name, and comparison conditions.

[0063] Based on the extracted key information, executable comparison rules are formed, which include task description, data source, fields, comparison conditions, and relationships between tables.

[0064] S4. Execute the comparison task based on executable comparison rules, and store or visualize the comparison results. This includes the following operations:

[0065] The executable comparison rules are submitted to the unified task scheduling system, which then configures the task execution method based on the comparison rules, including scheduled execution or immediate execution.

[0066] The unified task scheduling system matches and compares related data based on the data source, fields, comparison conditions, and inter-table relationships in the comparison rules;

[0067] The unified task scheduling system acquires the comparison results and stores them in a specified location according to the configuration. It also supports displaying the results in the form of charts or tables through visualization tools.

[0068] When implementing Implementation Example 1 or Implementation Example 2, it is assumed that the knowledge base stores a bidding information table and a blacklist information table. Wherein:

[0069] The bidding information table contains 15 fields: “BIAODUANGUID” (unique identifier for the bid section, varchar type, primary key), “DANWEIGUID” (supplier identifier, varchar type), “SUPPLIERCODE” (supplier code, varchar type), “DANWEINAME” (supplier name, varchar type), “BMR_DATE” (bid registration time, date type), “UPR_DATE” (bid document upload time, date type), “LIANXIREN” (bidding contact person, varchar type), “TOUBIAOPRICE” (initial bid price, decimal type), “LAST_TOTALPRICE” (final bid amount, decimal type), “UNITORGNUM” (unified social credit code, varchar type), “ZHUCEADDRESS” (registered address, varchar type), “LIXIRENTEL” (contact number, varchar type), “ISZB” (whether it is a successful bidder, varchar type), “SHR_DATE” (announcement time of the transaction result, varchar type), and “ZHONGBIAOPRICEDW” (unit of the winning bid amount, varchar type).

[0070] The blacklist information table contains 5 fields: "DANWEIGUID" (supplier identifier, varchar type, primary key), "DANWEINAME" (supplier name, varchar type), "CHUFATYPE" (penalty type, varchar type), "BEGIN_TIME" (start time of being added to the blacklist, varchar type), and "END_TIME" (end time of being added to the blacklist, date type).

[0071] The relationship between the bidding information table and the blacklist information table is as follows: the "DANWEIGUID" field in the bidding information table is equal to the "DANWEIGUID" field in the blacklist information table, that is, the two tables are linked through "DANWEIGUID".

[0072] Assume the user input is: From the bidding information table and the blacklist information table, filter out suppliers with penalty type 1 based on the condition that the supplier identifier is the same, and extract fields such as supplier code, supplier name, bidding contact person, bid document upload time, bid registration time, initial bid price, and final bid amount.

[0073] The key information extracted by the natural language parsing unit of the comparison rule generation module includes table names (tender information table and blacklist information table), query field names ("SUPPLIERCODE", "DANWEINAME", "LIANXIREN", "UPR_DATE", "BMR_DATE", "TOUBIAOPRICE", "LAST_TOTALPRICE"), associated field names ("DANWEIGUID"), and comparison conditions ("CHUFATYPE").

[0074] The rule generation unit of the comparison rule generation module generates executable comparison rules that include task description, data source, fields, comparison conditions, and inter-table relationships. The specific code is as follows:

[0075] {

[0076] "description":"Based on the bidding information table and the blacklist information table, suppliers with penalty type 1 are filtered out according to their unit ID."

[0077] "sources":[

[0078] {"name":"tbinfo","type":"Table"},

[0079] {"name":"hmdxx","type":"Table"}

[0080] ],

[0081] "fields":[

[0082] {"name":"DANWEIGUID","type":"VARCHAR"},

[0083] {"name":"SUPPLIERCODE","type":"VARCHAR"},

[0084] {"name":"DANWEINAME","type":"VARCHAR"},

[0085] {"name":"LIANXIREN","type":"VARCHAR"},

[0086] {"name":"BMR_DATE","type":"DATE"},

[0087] {"name":"UPR_DATE","type":"DATE"},

[0088] {"name":"TOUBIAOPRICE","type":"VARCHAR"},

[0089] {"name":"LAST_TOTALPRICE","type":"VARCHAR"},

[0090] {"name":"CHUFATYPE","type":"VARCHAR"}

[0091] ],

[0092] "conditions":[

[0093] {"field":"DANWEIGUID","operator":"=","value":"Same"},

[0094] {"field":"CHUFATYPE","operator":"=","value":"1"}

[0095] ],

[0096] "relationships":[

[0097] {"source1":"tbinfo","source2":"hmdxx","relation":"DANWEIGUID is the same"} ]

[0099] }

[0100] Subsequently, the executable comparison rules are submitted to the unified task scheduling system. The unified task scheduling system configures the task execution method according to the comparison rules, including scheduled execution or immediate execution. The unified task scheduling system matches and compares the related data based on the data source, fields, comparison conditions and inter-table relationships in the comparison rules. The unified task scheduling system obtains the comparison results and stores the results in the specified location according to the configuration. It also supports displaying the results in the form of charts or tables through visualization tools.

[0101] In summary, the intelligent data comparison rule generation system and method based on natural language of the present invention realizes the automatic generation and execution of data comparison rules through natural language processing technology. It has advantages such as intelligence, efficiency, flexibility and scalability, and can be widely used in scenarios such as data comparison, data quality detection and data consistency verification, significantly improving the efficiency and accuracy of data processing.

[0102] The above specific examples illustrate the principles and implementation methods of the present invention in detail. These embodiments are merely for the purpose of helping to understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made to the present invention by those skilled in the art without departing from the principles of the present invention should fall within the patent protection scope of the present invention.

Claims

1. A natural language-based intelligent data comparison rule generation system, characterized in that, Its structure includes: The comparison resource definition module is used to define the comparison resource data structure, which serves as the basis for subsequent comparison rule generation. The comparison task drafting module is used to clarify the source, fields, scope and relationship between comparison resources of the data to be compared, store relevant information in the knowledge base, and store the metadata obtained by natural splitting involved in the process of drafting the comparison task in the vector library so that the metadata can be called in the future to support the generation of comparison rules or the execution of tasks through vector retrieval. The comparison rule generation module provides a visual interface for converting natural language descriptions input by users through the visual interface into executable comparison rules; The comparison task execution module executes comparison tasks based on executable comparison rules, and then stores or visualizes the comparison results.

2. The intelligent data comparison rule generation system based on natural language according to claim 1, characterized in that, By comparing the resource data structure defined in the resource definition module, we can identify the table name, field name, field type, primary key, and foreign key information.

3. The intelligent data comparison rule generation system based on natural language according to claim 2, characterized in that, The comparison rule generation module specifically includes: The user input unit provides a visual interface for users to input natural language descriptions according to their needs. The Natural Language Parsing Unit (NLP) is used to extract key information from user input using the BERT model. The task is defined as a sequence labeling task. First, the natural language description is segmented using a word segmenter. Then, each word or sub-word after segmentation is labeled to generate a label sequence, which is then converted into an input format that is compatible with the BERT model. The rule generation unit is used to form executable comparison rules based on the extracted key information.

4. The intelligent data comparison rule generation system based on natural language according to claim 3, characterized in that, The natural language parsing unit uses the BERT model to extract key information from user input, including table name, query field name, related field name, and comparison conditions; The rule generation unit generates structured comparison rules based on the extracted key information, which include task description, data source, fields, comparison conditions, and inter-table relationships.

5. The intelligent data comparison rule generation system based on natural language according to claim 4, characterized in that, The comparison task execution module specifically includes: The rule submission unit is used to submit executable comparison rules to the unified task scheduling unit. The unified task scheduling unit is used to configure the task execution method according to the comparison rules. It is also used to match and compare related data based on the data source, fields, comparison conditions and inter-table relationships in the comparison rules, then obtain the comparison results and store the results in the specified location according to the configuration. It also supports displaying the results in the form of charts or tables through visualization tools.

6. A method for generating intelligent data comparison rules based on natural language, characterized in that, Includes the following steps: S1. Define the comparison resource data structure as the basis for subsequent comparison rule generation; S2. Clarify the source, fields, scope, and relationships between comparison resources of the data to be compared, store the relevant information in the knowledge base, and store the metadata obtained by natural splitting involved in the process of drafting the comparison task in the vector library so that these metadata can be called in the future through vector retrieval to support the generation of comparison rules or the execution of tasks. S3. Provides a visual interface to convert the natural language descriptions entered by users through the visual interface into executable comparison rules; S4. Execute the comparison task based on the executable comparison rules, and store or visualize the comparison results.

7. The intelligent data comparison rule generation method based on natural language according to claim 6, characterized in that, Execute step S1, which defines the resource data structure including table name, field name, field type, primary key, and foreign key information.

8. The intelligent data comparison rule generation method based on natural language according to claim 7, characterized in that, Step S3 specifically includes the following operations: Users can input natural language descriptions through a visual interface according to their needs; Using the BERT model, the task of extracting key information from user input is defined as a sequence labeling task. First, the natural language description is segmented using a word segmenter, and then each word or sub-word after segmentation is labeled to generate a label sequence, which is then converted into an input format that is compatible with the BERT model. Based on the extracted key information, executable comparison rules are formed.

9. The intelligent data comparison rule generation method based on natural language according to claim 8, characterized in that, Key information extracted from user input includes table name, query field name, related field name, and comparison conditions; Based on the extracted key information, structured comparison rules are formed, which include task description, data source, fields, comparison conditions, and relationships between tables.

10. The intelligent data comparison rule generation method based on natural language according to claim 9, characterized in that, Step S4 specifically includes the following operations: The executable comparison rules are submitted to the unified task scheduling system, which then configures the task execution method based on the comparison rules, including scheduled execution or immediate execution. The unified task scheduling system matches and compares related data based on the data source, fields, comparison conditions, and inter-table relationships in the comparison rules; The unified task scheduling system acquires the comparison results and stores them in a specified location according to the configuration. It also supports displaying the results in the form of charts or tables through visualization tools.

Images