Search intention recognition method based on NL2SQL model

Abstract

The invention discloses a search intention recognition method based on an NL2SQL model, relates to the technical field of data retrieval, and solves the problems that in the prior art, SQL query writing in the data analysis process needs a large amount of manual participation, time is consumed, and the requirements for the technical level and experience of operators are high. According to the method, the original data is converted into the data items by constructing the business knowledge base and the business dictionary base, and the SQL statements are constructed based on the data items. The method has the advantages that natural language query can be understood and converted, so that non-technical users can easily perform database query. Natural language query is converted into SQL statements by training a pre-training language model, so that the threshold of using a database by a user is greatly reduced. According to the method, the retrieval accuracy is improved, the user experience is enhanced, and database query becomes more visual and convenient. The method brings innovation and progress to the field of information retrieval.

Description

Technical Field

[0001] The present invention relates to the field of data retrieval technology, and in particular to a search intent recognition method based on an NL2SQL model. Background Art

[0002] Database is one of the cornerstones of modern information technology. Almost all information is stored in the form of database. However, the use of database requires professional technical knowledge, which is difficult for business personnel with non-technical background to understand and master. On the other hand, with the explosive growth of information volume, users' demand for data query is becoming more and more urgent. Therefore, how to enable business personnel to use database easily has become an urgent problem to be solved.

[0003] In the traditional data analysis process, if business personnel need to query data or perform analysis through SQL, they usually have to first clarify the specific requirements of the data they need to query, including the fields to be queried, the conditions to be set, and the possible aggregation calculation methods; then they manually write SQL, that is, data engineers or data analysts write corresponding SQL statements based on these requirements. This requires them not only to understand business needs, but also to ensure the accuracy and performance of SQL. This process takes a lot of time and energy, and has high requirements on the technical knowledge and experience of the operators. Otherwise, the written SQL statements may be inappropriate and affect subsequent data queries.

[0004] In view of this, a search intent identification method based on the NL2SQL model is needed. Summary of the invention

[0005] In view of the problem that a large amount of manual participation is required to write SQL queries in the data analysis process in the prior art, which is time-consuming and has high requirements on the technical level and experience of the operators, resulting in low automation and low efficiency, the present invention provides a search intent recognition method based on the NL2SQL model, which converts the original data into data items by constructing a business knowledge base and a business dictionary, and constructs SQL statements based on these data items. The advantage of this method is that it can understand and convert natural language queries, so that non-technical users can also easily perform database queries. It converts natural language queries into SQL statements by training a pre-trained language model, greatly reducing the threshold for users to use the database. This method not only improves the accuracy of retrieval, but also enhances the user experience, making database queries more intuitive and convenient. In addition, the NL2SQL model can handle complex query requirements, including multi-table queries and conditional screening, which are difficult to achieve with traditional SQL retrieval methods. In this way, the NL2SQL model not only makes up for the shortcomings of traditional SQL retrieval, but also brings innovation and progress to the field of information retrieval. The specific technical scheme is as follows:

[0006] A search intent identification method based on the NL2SQL model includes the following steps:

[0007] S1: Construct a business knowledge base and a business dictionary base, wherein the business knowledge base includes business processes, business rules and business concepts, and the business dictionary base includes professional terms and synonyms and abbreviations of professional terms.

[0008] Among them, business processes, business rules and business concepts, as well as professional terms and their synonyms and abbreviations are provided by business personnel or industry experts. They are reference tools for expert opinions and can be set up according to specific scenarios.

[0009] S2: Define a data storage structure in the database, and after collecting data, store the data in the corresponding data storage structure in the database according to the contents of the constructed business knowledge base and business dictionary library to form data items.

[0010] Step S1 and step S2 are the data preparation stage, and the data storage structure includes entity, relationship, time, result, id, type, task, previous task node of the current task, next task of the current task, etc. In the collected data, a column of data stored according to the data storage structure is a data item.

[0011] The business knowledge base and business dictionary library are tools used to help store data according to the data storage structure.

[0012] In the business knowledge base, when the business process identifies that a certain task is contained in the collected data, it can obtain the previous task node or the next task node of the task. For another example, business rules can be defined in the data stored in the database according to the data storage structure, and there will be no data storage errors or data storage omissions. For example, business rules can be specified as the data storage must contain "results" or must contain "entities" or "relationships", but not necessarily "XXX". For another example, in the business concept, the corresponding explanation of a certain task or a certain result can be obtained, which can appear in the remarks column in the business storage structure, or only appear in a certain data item in the form of annotations.

[0013] The business dictionary library is used to expand the data items stored according to the data storage structure. For example, each data structure in the data item can contain the professional terms (or professional descriptions) and their synonyms, abbreviations or expansions in the data structure. While expanding the specific data descriptions in a certain type of data structure, the business dictionary library also lays the foundation for subsequent data queries and understanding of natural sentences, improves the accuracy of data queries and understanding of natural sentences, and prevents certain data from being queried or certain natural sentence descriptions from being recognized.

[0014] S3: construct SQL statements based on data items;

[0015] Specifically, this step is to first construct a template of an SQL statement based on the data storage structure, then fill in the corresponding data storage structure in the data item based on the SQL statement template, and then form an SQL statement corresponding to the data item.

[0016] S4: constructing a data set for training a pre-trained language model based on the collected raw data and the SQL statements constructed based on the corresponding data items;

[0017] S5: Train a pre-trained language model based on the dataset, such as BERT or GPT, to obtain the final NL2SQL model.

[0018] S6: Input natural language into the trained NL2SQL model to obtain the SQL query statement generated by the NL2SQL model.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] The present invention converts raw data into data items by constructing a business knowledge base and a business dictionary, and constructs SQL statements based on these data items. The advantage of this method is that it can understand and convert natural language queries, so that non-technical users can also easily perform database queries. It converts natural language queries into SQL statements by training a pre-trained language model, greatly reducing the threshold for users to use the database. This method not only improves the accuracy of retrieval, but also enhances the user experience, making database queries more intuitive and convenient. In addition, the NL2SQL model can handle complex query requirements, including multi-table queries and conditional screening, which are difficult to achieve with traditional SQL retrieval methods. In this way, the NL2SQL model not only makes up for the shortcomings of traditional SQL retrieval, but also brings innovation and progress to the field of information retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0022] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0025] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0026] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0027] In one embodiment of the present invention, a search intent identification method based on the NL2SQL model is provided. Figure 1 As shown, the following steps are included:

[0028] S1: Construct a business knowledge base and a business dictionary base, wherein the business knowledge base includes business processes, business rules and business concepts, and the business dictionary base includes professional terms and synonyms and abbreviations of professional terms.

[0029] Among them, business processes, business rules and business concepts, as well as professional terms and their synonyms and abbreviations are provided by business personnel or industry experts. They are reference tools for expert opinions and can be set up according to specific scenarios.

[0030] S2: Define a data storage structure in the database, and after collecting data, store the data in the corresponding data storage structure in the database according to the contents of the constructed business knowledge base and business dictionary library to form data items.

[0031] Step S1 and step S2 are the data preparation stage. The data storage structure includes entity, relationship, time, result, id, type, task, previous task node of the current task, next task of the current task, etc. In the collected data, a column of data stored in the data storage structure is a data item. (That is, if the data storage structure is arranged in a table, a row of the table is a data item)

[0032] The business knowledge base and business dictionary library are tools used to help store data according to the data storage structure.

[0033] In the business knowledge base, when the business process identifies that a certain task is contained in the collected data, it can obtain the previous task node or the next task node of the task. For another example, business rules can be defined in the data stored in the database according to the data storage structure, and there will be no data storage errors or data storage omissions. For example, business rules can be specified as the data storage must contain "results" or must contain "entities" or "relationships", but not necessarily "XXX". For another example, in the business concept, the corresponding explanation of a certain task or a certain result can be obtained, which can appear in the remarks column in the business storage structure, or only appear in a certain data item in the form of annotations.

[0034] The business dictionary library is used to expand the data items stored according to the data storage structure. For example, each data structure in the data item can contain the professional terms (or professional descriptions) and their synonyms, abbreviations or expansions in the data structure. While expanding the specific data descriptions in a certain type of data structure, the business dictionary library also lays the foundation for subsequent data queries and understanding of natural sentences, improves the accuracy of data queries and understanding of natural sentences, and prevents certain data from being queried or certain natural sentence descriptions from being recognized.

[0035] S3: construct SQL statements based on data items;

[0036] Specifically, this step is to first construct a template of an SQL statement based on the data storage structure, then fill in the corresponding data storage structure in the data item based on the SQL statement template, and then form an SQL statement corresponding to the data item.

[0037] S4: constructing a data set for training a pre-trained language model based on the collected raw data and the SQL statements constructed based on the corresponding data items;

[0038] S5: Train a pre-trained language model based on the dataset, such as BERT or GPT, to obtain the final NL2SQL model.

[0039] S6: Input natural language into the trained NL2SQL model to obtain the SQL query statement generated by the NL2SQL model.

[0040] The following is a specific example to illustrate the specific steps from collecting training data to finally training the model.

[0041] Step 1: Data Collection

[0042] Collect historical maintenance records, which are stored in the grid's maintenance database. The data model of this database should have defined the table structure of maintenance records, such as a maintenance_records table that may contain fields such as equipment_id, maintenance_date, maintenance_result, and maintenance_staff.

[0043] Step 2: Data Preprocessing

[0044] When cleaning data, you need to ensure that each record conforms to the database's data model requirements. For example, if the data model requires that the maintenance_date field must be in a specific date format, then the date data needs to be formatted accordingly.

[0045] Step 3: Application of business knowledge base and business dictionary

[0046] Use rules in your business knowledge base to ensure that the data you collect complies with your business logic. For example, if your business rule states that every maintenance record must have a valid `maintenance_result`, you need to ensure that every record contains this information.

[0047] Use business dictionaries to standardize professional terms and ensure that the terms in natural language queries are consistent with the field names in the database. For example, ensure that "transformer" is uniformly represented as `equipment_id` in the database, and "maintenance result" corresponds to the `maintenance_result` field.

[0048] Step 4: Data Annotation

[0049] When converting historical maintenance records into natural language queries and corresponding SQL query statements, the data model of the business database directly affects the generation of SQL statements. For example, if `equipment_id` is the unique identifier of the transformer in the data model, then this field needs to be used in the SQL query statement to specify the query condition.

[0050] During the data annotation process, you need to determine which fields are necessary based on the data model and their correct order in the SQL query statement. For example, if foreign key relationships are defined in the data model, you need to ensure that these relationships are handled correctly when generating SQL query statements.

[0051] Step 5: Generate training data

[0052] The generated training data will directly reflect the data model of the business database. Each natural language query and the corresponding SQL query statement are generated based on the table structure and field relationship in the data model. In this way, when the NL2SQL model is trained, it learns how to map the keywords in the natural language query to the specific fields in the data model and generate the correct SQL query statement.

[0053] In the "query" process, the data model of the business database is the foundation. The NL2SQL model needs to understand how the keywords in the natural language query are mapped to the fields in the data model. For example, when the user asks "query the last maintenance result of transformer T101", the model needs to know that "transformer T101" corresponds to the `equipment_id` field, and "last maintenance result" corresponds to the `maintenance_result` field. In this way, the model can generate the correct SQL query statement: "SELECTmaintenance_result FROM maintenance_records WHERE equipment_id='T101'ORDERBY maintenance_date DESC LIMIT 1;"

[0054] Through this process, the data model of the business database not only plays a normative role in the data collection and preprocessing stage, but also directly affects the generation of SQL query statements in the data annotation and training data generation stage. This ensures that the generated training data is consistent with the actual database structure and business logic, so that the NL2SQL model can accurately understand and execute natural language queries.

[0055] To ensure that each sample in the dataset contains the necessary information, such as query intent, entity recognition, relationship recognition, etc., we need to follow the following steps:

[0056] Define query intent: Query intent refers to the goal that the user wants to achieve through natural language query. For example, a user may want to know the last maintenance result of a transformer, which is a query intent.

[0057] Entity recognition: Entity recognition refers to identifying specific entities from natural language queries, such as equipment number, date, maintenance personnel, etc. For example, in the query "Query the last maintenance result of transformer T101", transformer T101 is an entity that needs to be identified.

[0058] Relationship identification: Relationship identification refers to identifying the relationships between entities. These relationships are usually reflected in SQL queries as join conditions or query conditions between tables. For example, the relationship between transformer T101 and maintenance date.

[0059] Labeling query intent: In the dataset, each sample needs to be clearly labeled with the query intent. This can be achieved through manual labeling or using a pre-trained intent recognition model.

[0060] Label entities: Use named entity recognition (NER) technology to label entities in natural language queries. These entities need to be associated with tables and fields in the database.

[0061] Label relationships: Label the relationships between entities. These relationships will help the model understand how to construct SQL query statements. For example, label the relationship between transformer T101 and maintenance date.

[0062] Build SQL template: Build the corresponding SQL query template based on the marked query intent, entity and relationship. For example, for the query intent "query the last maintenance result", entity "transformer T101", and relationship "last maintenance", the constructed SQL template may be: "SELECT maintenance_result FROM maintenance_records WHERE equipment_id='T101'ORDER BY maintenance_date DESC LIMIT 1;".

[0063] Dataset construction: Integrate the above annotation information into the dataset to ensure that each sample contains query intent, entities, relations, and corresponding SQL templates.

[0064] The following takes power grid maintenance as an example to illustrate the overall implementation process of this method:

[0065] Assume we have the following historical maintenance records:

[0066] Transformer number: T101

[0067] Maintenance date: 2023-10-05

[0068] Maintenance result: normal

[0069] Maintenance personnel: Zhang San

[0070] We can convert this record into the following training sample:

[0071] Query intention: Query the last maintenance results

[0072] Entity recognition: Transformer T101

[0073] Relationship identification: The maintenance date of transformer T101 is 2023-10-05

[0074] SQL template: "SELECT maintenance_result FROM maintenance_records WHEREequipment_id='T101'ORDER BY maintenance_date DESC LIMIT 1;"

[0075] In this way, we ensure that each sample in the dataset contains the necessary information that will be used to train the NL2SQL model to understand and execute natural language queries. In practice, these steps can be automated through a machine learning workflow that includes stages such as data preprocessing, feature engineering, model training, and evaluation.

[0076] In summary, the present invention converts raw data into data items by constructing a business knowledge base and a business dictionary, and constructs SQL statements based on these data items. The advantage of this method is that it can understand and convert natural language queries, so that non-technical users can also easily perform database queries. It converts natural language queries into SQL statements by training a pre-trained language model, greatly reducing the threshold for users to use the database. This method not only improves the accuracy of retrieval, but also enhances the user experience, making database queries more intuitive and convenient. In addition, the NL2SQL model can handle complex query requirements, including multi-table queries and conditional screening, which are difficult to achieve with traditional SQL retrieval methods. In this way, the NL2SQL model not only makes up for the shortcomings of traditional SQL retrieval, but also brings innovation and progress to the field of information retrieval.

[0077] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0078] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0079] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0080] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A search intent identification method based on the NL2SQL model, characterized in that: The following steps are involved: Build business knowledge base and business dictionary; Define a data storage structure in the database, and after collecting the original data, store the data in the corresponding data storage structure in the database according to the contents of the constructed business knowledge base and business dictionary library to form data items; Construct SQL statements based on data items; Constructing a data set for training a pre-trained language model based on the collected raw data and the SQL statements constructed based on the corresponding data items; The NL2SQL model is obtained by training the pre-trained language model through the dataset.

2. According to the method for identifying search intent based on the NL2SQL model in claim 1, it is characterized in that: The business knowledge base includes business processes, business rules and business concepts, and the business dictionary includes professional terms and synonyms and abbreviations of professional terms.

3. According to the search intention identification method based on the NL2SQL model of claim 1, it is characterized in that: The data storage structure includes entity, relationship, time, result, id, type, task, previous task node of the current task, and next task of the current task.

4. The search intention identification method based on the NL2SQL model according to claim 1, characterized in that: Constructing SQL statements based on data items is specifically as follows: first constructing a template for SQL statements based on the data storage structure, then filling in the corresponding data storage structure in the data item based on the SQL statement template, and finally forming the SQL statement corresponding to the data item.

5. The search intention identification method based on the NL2SQL model according to claim 1, characterized in that: The pre-trained language model uses BERT or GPT.

6. The search intention identification method based on the NL2SQL model according to claim 1, characterized in that: The original data is derived from information related to power grid business.

7. The search intention identification method based on the NL2SQL model according to claim 6 is characterized in that: The relevant materials shall be presented in the form of all documents, reports and manuals.

8. The search intention identification method based on the NL2SQL model according to claim 6 is characterized in that: Information related to power grid business includes power grid operation procedures, maintenance manuals for power equipment, and analysis reports on the power market.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the search intent identification method based on the NL2SQL model described in any one of claims 1 to 8.

10. A processor, characterized in that: The processor is used to run a program, wherein the program executes the search intent identification method based on the NL2SQL model described in any one of claims 1 to 8 when running.

Images