Intelligent conversational method and server based on tabular data retrieval

Abstract

The application relates to an intelligent conversation method and a server based on table data retrieval, wherein the method comprises the following steps: based on input corpus, calling an entity extraction and syntax analysis method, obtaining entity extraction and syntax analysis results, and according to table entity word information, performing feature splicing and feature coding, calling a query language conversion model based on the coded features, and generating an SQL statement; based on the SQL statement, submitting a database execution engine, obtaining an execution result, and returning the execution result through an interface as a reply to the input corpus. Thus, the problems of high complexity of the coding layer, poor accuracy of the algorithm and low reasoning efficiency in the related art are solved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to an intelligent conversation method and server based on tabular data retrieval. Background Technology

[0002] In intelligent conversational systems, tabular data retrieval addresses Natural Language (NL) questions by providing answers based on the content of a data table using NL technology. This offers a relatively ideal solution without involving the construction of complex systems.

[0003] Currently, based on text-based knowledge and a limited amount of tabular data, intelligent conversational technologies are widely used in the intelligent assistant services of many car brands. However, for question-answering scenarios involving massive amounts of real-time connected vehicle data, conversational technologies based on tabular data retrieval are relatively scarce, and there are no effective implementation examples yet. Furthermore, intelligent conversational technologies based on text-based knowledge and a limited amount of tabular data struggle to adapt to complex, multi-field, and multi-valued table scenarios, making it difficult to handle diverse user questions and achieve accurate responses. Additionally, the training phase places high demands on memory and computing power, which urgently needs to be addressed. Summary of the Invention

[0004] This application provides an intelligent conversation method and server based on tabular data retrieval, which solves the problems of high complexity of the coding layer in related technologies, resulting in poor algorithm accuracy and low inference efficiency.

[0005] The first aspect of this application provides an intelligent conversation method based on tabular data retrieval, comprising the following steps: receiving input corpus, invoking a syntax correction method to obtain corrected input corpus; based on the corrected input corpus, querying a preset high-frequency question SQL database, if a match is found, returning the retrieved SQL result; if no match is found, invoking an intent classification method based on the corrected input corpus, returning an unanswerable state if no match is found; if a search intent is found, invoking entity extraction and syntactic analysis methods based on the input corpus to obtain entity extraction and syntactic analysis results; based on the entity extraction results, searching a tabular entity vocabulary to obtain query results for tabular entities, and based on the tabular entity query results, invoking a table retrieval method to obtain target tables and filtering corresponding tabular entity word information; based on the entity extraction and syntactic analysis results of the input corpus and the tabular entity word information, performing feature concatenation and feature encoding, and based on the encoded features, invoking a query language conversion model to generate an SQL statement; based on the SQL statement, submitting it to a database execution engine, obtaining the execution result, and returning it through an interface as a response to the input corpus.

[0006] Based on the aforementioned technical means, this embodiment of the application, to ensure accurate correspondence between SQL statements and natural language, inputs the input corpus, the syntactic analysis results, and the interaction features between the target table and the entity words into a query language conversion model to construct SQL statements. The encoding layer of the query language conversion model incorporates the interaction features between the target table and entity words, as well as the syntactic information of the input natural language. Using the matched target table as input significantly reduces the feature network size and the complexity of the encoding layer, making it adaptable to scenarios with complex tables and multiple fields and values. Using the interaction features between field values ​​and user questions as input features for statement analysis and assembly ensures accurate correspondence between the corpus content and table fields when only field values ​​are mentioned in the input. Extracting the syntactic analysis results from the input corpus for SQL statement assembly ensures accurate correspondence between the SQL statements and natural language during the assembly process.

[0007] Optionally, the step of calling a query language conversion model based on the encoded features to generate an SQL statement includes: obtaining the target column of the query, the aggregation operation method of the target column, the filter condition target column, the comparison logic and comparison value corresponding to the filter condition target column, the classification prediction result of the logic between the filter conditions, and combining the preset SQL syntax specification and the relationship between tables to generate the SQL statement.

[0008] Optionally, the step of receiving the input corpus and invoking a grammar correction method to obtain the corrected input corpus includes: segmenting the input corpus into words and using a window shifting method to obtain word combinations with contextual features; scoring the word combinations using a pre-trained linguistic statistics model and identifying the sites with the largest deviations from the mean scores as potential error sites; obtaining a candidate character set for the characters corresponding to the potential error sites using a preset homophone and similar-looking character dictionary and a preset common confusion dictionary; and scoring the sentences in which the candidate characters replace the characters at the error sites again to obtain the optimal result, which is then used as the corrected result.

[0009] Optionally, the step of invoking the intent classification method based on the corrected input corpus includes: using a pre-trained text classification model to predict whether the input corpus belongs to a preset retrieval intent; if it belongs to the preset retrieval intent, then extracting entity words based on the input corpus; if it does not belong to the preset retrieval intent, then providing a non-retrieval intent prompt and performing the operation that the table data retrieval service cannot respond.

[0010] Optionally, the step of extracting entity words from the input corpus includes: extracting entity words from the input corpus as potential entity words; performing word segmentation and part-of-speech extraction on the input corpus as language segmentation; and using the potential entity words and the language segmentation as the entity words.

[0011] Optionally, the step of extracting entity words from the input corpus as potential entity words includes: performing representation normalization processing on the input corpus to obtain normalized sentences; performing time and numerical extraction on the normalized sentences according to regular expressions; calling a pre-trained NER model to extract generic names from the normalized sentences; and using the extracted entity words and their corresponding entity word types as the potential entity words.

[0012] Optionally, the table entity recall and target table recall method includes: matching the table entity mapping table according to the entity words to obtain the recall result of the table entity; and querying the table relationship graph data based on the table entity recall result, and combining the path search algorithm to obtain the query result of the shortest path.

[0013] Optionally, the language conversion model includes: the main body of the model adopts the ERINE pre-trained model framework; the encoding layer uses the character features of the input corpus, the character features of the table column names, the pairing features of the corpus entities and the table entities, and the syntactic analysis features, and the encoded results are used as input; the inference layer uses the target column of the query, the aggregation operation method of the target column of the query, the target column of the filtering conditions, the comparison logic and comparison value corresponding to the target column of the filtering conditions, and the logic between the filtering conditions as the prediction target.

[0014] Optionally, the training method of the language conversion model includes: using the difference between the task prediction result and the actual result of the inference layer as the loss value, and using the minimum sum of loss values ​​as the iteration target for joint training.

[0015] Optionally, before searching the table entity lexicon, the method further includes: determining the data retrieval table; extracting enumerated fields from the data retrieval table and extracting field information to establish a mapping relationship table containing field values, field names, and table names; extracting field name and table name entity word records containing enumerated and numeric fields from the data retrieval table and adding them to the mapping relationship table; calling the FastText word vector model to vectorize the field values; and using the mapping relationship table as the table entity lexicon.

[0016] The second aspect of this application provides an intelligent conversation server based on tabular data retrieval. A preprocessing module receives input corpus, calls a syntax correction method, and obtains corrected input corpus. An analysis module queries a preset high-frequency question SQL database based on the corrected input corpus. If a match is found, the retrieved SQL result is returned; if not, an intent classification method is called based on the corrected input corpus. If no match is found, an "unable to answer" status is returned; if a search intent is found, entity extraction and syntactic analysis methods are called based on the input corpus to obtain entity extraction and syntactic analysis results. A filtering module... The first module is used to search the table entity vocabulary based on the entity extraction results, obtain the query results for table entities, and call the table retrieval method based on the table entity query results to obtain the target table and filter the corresponding table entity word information; the second module is used to perform feature concatenation and feature encoding based on the entity extraction and syntactic analysis results of the input corpus and the table entity word information, and call the query language conversion model based on the encoded features to generate an SQL statement; the third module is used to submit the SQL statement to the database execution engine, obtain the execution result, and return it through the interface as a response to the input corpus.

[0017] Optionally, the generation module is further used to obtain the target column of the query, the aggregation operation method of the target column, the filter condition target column, the comparison logic and comparison value corresponding to the filter condition target column, the classification prediction result of the logic between the filter conditions, and generate the SQL statement by combining the preset SQL syntax specification and the relationship between the tables.

[0018] Optionally, the preprocessing module is used to: segment the input corpus into words, and combine the window translation method to obtain word segments with contextual features; score the word segments using a pre-trained linguistic statistics model, and identify the sites with the largest deviations from the mean scores as potential error sites; obtain a candidate character set for the characters corresponding to the potential error sites using a preset homophone and similar-looking character dictionary and a preset common confusion dictionary; and score the sentences in which the candidate characters replace the characters at the error sites again to obtain the optimal result, which is used as the corrected result.

[0019] Optionally, the analysis module is used to: predict whether the input corpus belongs to a preset retrieval intent using a pre-trained text classification model; if it belongs to the preset retrieval intent, extract entity words based on the input corpus; if it does not belong to the preset retrieval intent, provide a non-retrieval intent prompt and perform the operation that the table data retrieval service cannot respond.

[0020] Optionally, the analysis module is further configured to: extract entity words from the input corpus as potential entity words; perform word segmentation and part-of-speech extraction on the input corpus as language segmentation; and use the potential entity words and the language segmentation as the entity words.

[0021] Optionally, the analysis module is further configured to: perform representation normalization processing on the input corpus to obtain normalized sentences; extract time and numerical values ​​from the normalized sentences according to regular expressions; call a pre-trained NER model to extract generic names from the normalized sentences; and use the extracted entity words and their corresponding entity word types as the potential entity words.

[0022] Optionally, the table entity recall and target table recall method includes: matching the table entity mapping table according to the entity words to obtain the recall result of the table entity; and querying the table relationship graph data based on the table entity recall result, and combining the path search algorithm to obtain the query result of the shortest path.

[0023] Optionally, the language conversion model includes: the main body of the model adopts the ERINE pre-trained model framework; the encoding layer uses the character features of the input corpus, the character features of the table column names, the pairing features of the corpus entities and the table entities, and the syntactic analysis features, and the encoded results are used as input; the inference layer uses the target column of the query, the aggregation operation method of the target column of the query, the target column of the filtering conditions, the comparison logic and comparison value corresponding to the target column of the filtering conditions, and the logic between the filtering conditions as the prediction target.

[0024] Optionally, the training method of the language conversion model includes: using the difference between the task prediction result and the actual result of the inference layer as the loss value, and using the minimum sum of loss values ​​as the iteration target for joint training.

[0025] Optionally, it further includes: a construction module for determining a data retrieval table before searching the table entity lexicon; extracting enumerated fields from the data retrieval table and extracting field information to establish a mapping relationship table containing field values, field names, and table names; extracting field name and table name entity word records containing enumerated and numeric fields from the data retrieval table and adding them to the mapping relationship table; calling the fasttext word vector model to vectorize the field values; and using the mapping relationship table as the table entity lexicon.

[0026] Therefore, this application has at least the following beneficial effects:

[0027] This application embodiment constructs an entity lexicon for the data retrieval table, fully extracts the prior information of the table, enhances the table information, and improves the accuracy of the semantic expression of the table information. After receiving the input corpus, the matching comparison between the input corpus and the data retrieval table switches between the comparison of entity words in the input corpus and the table entity lexicon, providing a solution for input natural language and multi-table association queries, shortening the length of the comparison statement, and enabling fast matching. To ensure that the SQL statement corresponds accurately to the natural language, the input corpus, syntactic analysis results, and interaction features between the target table and entity words are input into the query language conversion model to construct the SQL statement. The interaction features between the target table and entity words and the syntactic information of the input natural language are introduced into the encoding layer of the query language conversion model. Using the target table obtained through matching as input, the feature network can be significantly reduced, lowering the complexity of the encoding layer and making it suitable for scenarios with complex tables and multiple fields and values. By using the interaction features between field values ​​and user questions as input features for sentence analysis and assembly, accurate correspondence between the corpus content and table fields can be ensured when the input only mentions field values. Extracting the syntactic analysis results from the input corpus for SQL statement assembly ensures accurate correspondence between SQL statements and natural language during the assembly process. Therefore, this application can adapt to diverse user questions, achieve accurate correspondence between SQL statements and natural language, while maintaining a small feature network, low encoding complexity, reduced implementation difficulty, and improved computational efficiency.

[0028] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0029] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0030] Figure 1 This is a flowchart illustrating an intelligent conversation method based on tabular data retrieval provided according to an embodiment of this application;

[0031] Figure 2 This is a schematic diagram of a vehicle-to-everything (V2X) intelligent conversation system according to an embodiment of this application;

[0032] Figure 3 This is a schematic diagram illustrating the overall implementation process of a smart session according to an embodiment of this application;

[0033] Figure 4 This is a schematic diagram of an intelligent session server based on tabular data retrieval according to an embodiment of this application. Detailed Implementation

[0034] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0035] The following describes an intelligent conversation method and server based on tabular data retrieval according to embodiments of this application, with reference to the accompanying drawings. To address the issues mentioned in the background technology center regarding the difficulty in adapting to complex tables with multiple fields and values, the inability to meet diverse user needs, the inability to achieve accurate matching, and the high memory and computing power requirements during the training phase, this application provides an intelligent conversation method based on table data retrieval. This method constructs an entity lexicon for the data retrieval table, fully extracts prior information from the table, enhances table information, and improves the accuracy of semantic expression of table information. Upon receiving the input corpus, the matching comparison between the input corpus and the data retrieval table switches between comparing the entity words in the input corpus with the table entity lexicon, providing a solution for input natural language and multi-table association queries, shortening the length of the comparison statement, and enabling rapid matching. To ensure accurate matching between SQL (Structured Query Language) statements and natural language, the input corpus, syntactic analysis results, and interaction features between the target table and entity words are input into the query language conversion model to construct the SQL statement. The encoding layer of the query language conversion model incorporates the interaction features between the target table and entity words, as well as the syntactic information of the input natural language. Using the target table obtained through matching as input, the feature network can be significantly reduced, lowering the complexity of the encoding layer and making it suitable for scenarios with complex tables and multiple fields and values. By using the interaction features between field values ​​and user questions as input features for sentence analysis and assembly, accurate correspondence between the corpus content and table fields can be ensured when the input only mentions field values. Extracting the syntactic analysis results from the input corpus for SQL sentence assembly ensures accurate correspondence between SQL statements and natural language during sentence assembly. Therefore, this method can adapt to diverse user questions, achieve accurate correspondence between SQL statements and natural language, and simultaneously reduces the feature network size and encoding complexity, lowering the implementation difficulty and improving computational efficiency.

[0036] Specifically, Figure 1 This is a flowchart illustrating an intelligent conversation method based on tabular data retrieval provided in an embodiment of this application.

[0037] like Figure 1 As shown, this intelligent conversation method based on tabular data retrieval mainly includes the following steps:

[0038] Step S101: Receive the input corpus, call the syntax correction method, and obtain the corrected input corpus.

[0039] In this embodiment, receiving input corpus and invoking a grammar correction method to obtain corrected input corpus includes: segmenting the input corpus into words and using a window translation method to obtain word combinations with contextual features; scoring the word combinations using a pre-trained linguistic statistics model and identifying the sites with the largest deviations from the mean scores as potential error sites; obtaining a candidate character set for the characters corresponding to the potential error sites using a preset homophone and similar-looking character dictionary and a preset common confusion dictionary; and scoring the sentences in which candidate characters replace characters at error sites again to obtain the optimal result, which is then used as the corrected result.

[0040] Understandably, this involves using a pre-trained language model to perform grammatical correction and adjustment on the input corpus. Specifically:

[0041] Problems with user voice or text input may include grammatical errors, affecting the accuracy of downstream semantic understanding. In this embodiment, a pre-trained Kenlm model can be used to score the list processed by ngrams (a language model commonly used in large-vocabulary continuous speech recognition) after word segmentation. Based on a threshold of deviation from the mean, error sites are identified. Combined with a pre-built dictionary of homophones, similar-looking characters, and common confusion words, the characters and words at the error sites are replaced to generate a candidate set. If the Kenlm score of the candidate set is higher than the original sentence by 5%, the original sentence is replaced as the corrected output; otherwise, the original sentence is retained.

[0042] It should be noted that the embodiments of this application provide a method for natural language semantic understanding, which improves the system's tolerance for grammatical errors in input questions.

[0043] Step S102: Based on the corrected input corpus, query the preset high-frequency question SQL database. If a match is found, return the SQL result. If no match is found, call the intent classification method based on the corrected input corpus. If no match is found, return the status that no answer is possible. If the retrieval intent is found, call the entity extraction and syntactic analysis methods based on the input corpus to obtain the entity extraction and syntactic analysis results.

[0044] The pre-defined high-frequency problem SQL database can be created according to actual conditions without specific limitations.

[0045] It is understood that the embodiments of this application can receive input corpus, perform entity word extraction and dependency parsing on the input corpus respectively, and obtain entity word extraction and syntactic analysis results. Before performing entity word extraction on the input corpus, the following steps can be performed first: identify whether the input corpus belongs to the search intent; if it belongs to the search intent, perform the step of entity word extraction on the input corpus; if it does not belong to the search intent, prompt that it is not a search intent, and the table data search service cannot respond.

[0046] In some embodiments, this application can collect historical input data from the user to establish a high-frequency question SQL database. By including frequently asked natural language questions from the user and the relationship between natural language and data query language, a high-frequency FAQ (Frequently Asked Questions) database can be established to quickly and accurately respond to frequently asked key data query questions from the user.

[0047] High-frequency natural language questions from the user side can be collected from historical consultation logs and data retrieval requirements, while high-frequency data query languages ​​can be collected from database query logs. For example, SQL queries with a frequency of three or more can be extracted and translated into natural language. The correspondence between natural language and data query languages ​​can be established through manual translation and annotation. Furthermore, the natural language questions can be generalized using a sentence vector model to vectorize the generalized sentences, storing them as sentence vectors in the Faiss sentence vector library to achieve efficient retrieval and matching of input questions.

[0048] Therefore, in this embodiment of the application, after receiving the input corpus and before extracting entity words from the input corpus, the following steps can be performed first:

[0049] Step 1: Use the high-frequency problem SQL library to perform similarity matching on natural language and generate similarity scores;

[0050] Step 2: Extract SQL statements with similarity reaching the matching threshold from the high-frequency question SQL database, use them as SQL statements, and execute them to generate response information.

[0051] Step 3: If no SQL query with similarity reaching the matching threshold is found, proceed with the entity word extraction step based on the input corpus.

[0052] After establishing a high-frequency FAQ question database, to improve efficiency, before performing corpus analysis, the database is first matched with the high-frequency FAQ question database after receiving user input corpus. Based on the sentence vector results of the user input question, the cosine similarity can be used to search for the most similar target in the Faiss sentence vector database. If the TOP1 result reaches the threshold, the sentences are considered to be nearly equivalent, and the SQL of the recalled sentence is directly used as the SQL generation result of the input statement. This skips the step of semantic analysis of the input corpus and can directly return the SQL statement.

[0053] It should be noted that the system determines whether the user's input question is a search intent. If it is not a search intent, the system will return that it cannot answer. Furthermore, the response results of FAQ and knowledge graph can be given priority, without any restrictions here.

[0054] In some embodiments, this application can use a text binary classification model trained by TextCNN to determine the retrieval intent, and also provides a natural language semantic understanding method to improve the system's ability to filter unanswerable question types; at the same time, it also provides a method for generating and using a high-frequency FAQ question library to ensure the timeliness and accuracy of responses to high-frequency questions.

[0055] In this embodiment of the application, based on the corrected input corpus, the intent classification method is invoked, including: for the input corpus, using a pre-trained text classification model to predict whether it belongs to a preset retrieval intent; if it belongs to the preset retrieval intent, entity words are extracted based on the input corpus; if it does not belong to the preset retrieval intent, a non-retrieval intent prompt is given, and the operation that the table data retrieval service cannot respond is executed.

[0056] Specifically, the input corpus refers to the questions received from the user side, in order to... Figure 2 Taking the vehicle-to-everything (V2X) intelligent dialogue system as an example, after the microphone device of the mobile phone or vehicle terminal collects the voice signal input, it transmits the voice binary stream information to the intelligent dialogue system through the voice assistant. The system first triggers the ASR (speech-to-text service) service to complete the speech-to-text conversion. The generated text is used as the input corpus, and the text information flows to the dialogue management service module. This module is responsible for managing the calling strategy of the question-and-answer interface. In this embodiment, the question-and-answer interface is the table dialogue service, that is, the method shown in this embodiment. In the table dialogue service, the conversion from natural language to query language is completed. The generated SQL query language completes the query of the answer in the execution engine, and then the query result is returned to the dialogue management service module in text form through the table question-and-answer service for response.

[0057] Specifically, the response requires semantic understanding of the input corpus in natural language to clarify the object of the question it seeks to understand. Semantic understanding mainly includes two parts: first, entity word extraction based on the input corpus; and second, dependency parsing of the input corpus to obtain syntactic analysis results.

[0058] Entity extraction mainly involves extracting latent entity words from the input natural language to match them with the entity words in the table, thereby obtaining word pairs between the input segment and the entity words in the table.

[0059] Dependency parsing primarily targets the word segmentation results and part-of-speech information obtained from the input corpus to predict the relationships between words, thus supplementing the features of word relationships. Specifically, the dependency parsing tool of HanLP (a big data integrated platform) can be used, but this embodiment does not limit it. A type of dependency relationship and its popular description are shown in the table below.

[0060] Assuming the input corpus is "Check the best-selling car models in March 2022", a targeted dependency relationship is shown in Table 1 below:

[0061] Table 1

[0062]

[0063]

[0064] Step S103: Based on the entity extraction results, search the table entity thesaurus to obtain the query results of the table entities, and based on the table entity query results, call the table recall method to obtain the target table and filter the corresponding table entity word information.

[0065] It is understood that, in the embodiments of this application, entity words can be matched against the entity words in the table entity lexicon to obtain the table matching words, and the target table corresponding to the table matching words can be extracted.

[0066] In this embodiment of the application, the table entity recall and target table recall method includes: matching the table entity mapping table according to the entity words to obtain the recall result of the table entity; and querying the table relationship graph data based on the recall result of the table entity, and combining the path search algorithm to obtain the query result of the shortest path.

[0067] Specifically, the table entity lexicon contains several entity words and related word information (including but not limited to entity word type and other entity word information). The extracted entity words are matched one by one with each entity word in the table entity lexicon to determine whether there is a matching corpus object in the data retrieval table. To improve comparison efficiency and accuracy, this embodiment extracts entity words from the input corpus and constructs a table entity lexicon for the data retrieval table. During the matching comparison between the input corpus and the data retrieval table, the entity words in the input corpus are compared with the table entity lexicon, eliminating useless non-entity word information and shortening the length of the compared sentences, thereby improving comparison efficiency. Furthermore, retaining key entity word information that reflects the meaning of the sentence during comparison ensures matching accuracy.

[0068] It should be noted that the process of matching and comparing table entity words with the entity lexicon can refer to the matching and comparison methods of words or sentences in related technologies. This embodiment does not limit this. For example, the extracted entity words and their corresponding entity word categories can be vectorized using FastText to recall table entity words with a cosine similarity > 0.9, and the field names corresponding to the table entity words can be obtained as table matching words. If the extracted entity words include time types, all time-related fields in the table entity lexicon can be recalled. If there are numeric types, the numeric values ​​are within the range and the units are consistent, and these are used as matching objects. The table attribute word field names, word segments, table entity words, and table field names are obtained and combined into a set of word pairs. This embodiment only uses the above implementation method as an example for introduction. Other methods can refer to the introduction of this embodiment and will not be elaborated here.

[0069] After obtaining the table matching terms, the table with the strongest correlation to each table matching term is extracted from the data retrieval tables and used as the target table. This embodiment does not limit the method for determining the target table; for example, it can be determined based on the number of table matching terms contained in each table, or based on the number of related terms in the table and the strength of the correlation.

[0070] To deepen the understanding of the target table determination method, a target table determination method based on graph data is proposed here, as follows:

[0071] In some embodiments, before receiving the input corpus, the relationships between data objects (such as tables and fields) in the data retrieval table can be further extracted and stored in graph data; then, extracting the target table corresponding to the table matching words can specifically be done by: extracting the shortest path in the path set corresponding to the table matching words according to the graph data, and determining the target table corresponding to the shortest path.

[0072] In this process, the table schema (database organization and structure) is constructed in the form of graph data. Based on the relationships between tables and the inclusion relationships between tables and fields provided by the data dictionary, triplet data is generated and stored in the graph database. This is used for filtering target tables and establishing relationships during cross-table queries. Correspondingly, after linking the matched table entity words to the graph data, a path search algorithm can be used to obtain the path set of the subgraph. In this embodiment, the shortest path, which hits all paired word segments and contains the fewest tables, is preferred. The table names included in the shortest path are all target tables, and the field names mapped to the entity words in the paired fields are the target fields.

[0073] In the above method for determining the target table, based on the matching information of the recalled entity words, the matched table entity words are linked to the entities in the graph data to obtain the path set of the subgraph. Then, the target table with the strongest correlation is determined according to the number and length of the paths, which can significantly improve the extraction speed.

[0074] Step S104: Based on the entity extraction and syntactic analysis results of the input corpus and the entity word information in the table, perform feature concatenation and feature encoding. Based on the encoded features, call the query language conversion model to generate SQL statements.

[0075] It is understood that this embodiment calls the query language transformation model to assemble SQL statements. The query language transformation model refers to the model used to convert natural language into query language (SQL). The query language transformation model can be built based on the seq2sql framework (a model that uses reinforcement learning to generate SQL from natural language), and there are no limitations on this.

[0076] In this embodiment of the application, based on the encoded features, a query language conversion model is invoked to generate an SQL statement, including: obtaining the target column of the query, the aggregation operation method of the target column of the query, the target column of the filtering conditions, the comparison logic and comparison value corresponding to the target column of the filtering conditions, the classification prediction result of the logic between the filtering conditions, and combining the preset SQL syntax specification and the relationship between tables to generate the SQL statement.

[0077] In this embodiment, the language conversion model includes: the main body of the model adopts the ERINE pre-trained model framework; the encoding layer uses the character features of the input corpus, the character features of the table column names, the pairing features of the corpus entities and the table entities, and the syntactic analysis features, and the encoded results are used as input; the inference layer uses the target column of the query, the aggregation operation method of the target column of the query, the target column of the filtering conditions, the comparison logic and comparison value corresponding to the target column of the filtering conditions, and the logic between the filtering conditions as the prediction target.

[0078] The training method for the language conversion model includes: using the difference between the task prediction result and the actual result of the inference layer as the loss value, and using the minimum sum of loss values ​​as the iteration target for joint training.

[0079] Specifically, to ensure accurate correspondence between the SQL statement and the input query, this embodiment inputs the input corpus, syntactic analysis results, and interaction features between the target table and entity words into the query language conversion model. The encoding layer of the query language conversion model incorporates the interaction features between the target table and entity words, as well as the syntactic information of the input natural language. The SQL response is constructed based on the linguistic features of the input corpus. By using only the matched target table, rather than the values ​​of each field in the associated table as input, the feature network is significantly reduced, making it suitable for scenarios with complex tables and multiple fields and values. This improves the effectiveness and accuracy of the analysis, while also increasing the efficiency of the training phase and prediction settlement, thus enhancing the user experience. Furthermore, this embodiment uses the interaction features between field values ​​and user questions as input features for statement analysis and assembly, solving the problem of accurate correspondence between questions and table fields when only field values ​​are mentioned in the user question. Moreover, this embodiment further extracts the syntactic analysis results of the input corpus for SQL statement assembly, ensuring logical coherence and grammatical soundness in each word segmentation assembly.

[0080] In this embodiment, sentences are assembled based on the input corpus, syntactic analysis results, target table, and interaction features between entity words, ensuring the comprehensiveness of feature information and improving prediction accuracy; at the same time, the overall amount of information is limited, effectively controlling prediction and training time.

[0081] Step S105: Based on the SQL statement, submit it to the database execution engine. After obtaining the execution result, return it through the interface as a response to the input corpus.

[0082] Understandably, the SQL query is submitted to the database for execution. If the execution is successful, the result is returned, and the generated information serves as the response. After the database execution engine completes the query for the answer, the generated SQL query can be returned to the dialogue management service module in text form via the table-based question-and-answer service. The query result text is then converted into speech via the TTS (text-to-speech) service. The speech information is returned to the terminal's voice assistant in binary stream form, and the terminal's speaker is used to broadcast the speech, thus providing the response.

[0083] If the execution fails, the form question and answer service can indicate that it cannot provide an accurate answer. In this embodiment, the handling method in this case is not limited. The system can call traditional intelligent answering methods (such as knowledge graphs) to query for answers, or query according to a pre-configured high-frequency question and answer database, etc. The specific settings can be made according to actual usage needs, which will not be elaborated here.

[0084] The above embodiments do not limit the specific extraction method for entity words based on the input corpus. To ensure comprehensive entity word extraction, some embodiments perform entity word extraction based on the input corpus, including: extracting entity words from the input corpus as potential entity words; performing word segmentation and part-of-speech extraction on the input corpus as language segmentation; and combining potential entity words and language segmentation as entity words. Specifically, extracting entity words from the input corpus as potential entity words includes: performing representation normalization on the input corpus to obtain normalized sentences; extracting time and numerical values ​​from the normalized sentences using regular expressions; calling a pre-trained NER model to extract generic names from the normalized sentences; and using the extracted entity words and their corresponding entity word types as potential entity words.

[0085] Specifically, entity word extraction based on the input corpus can be performed in the following sub-steps:

[0086] S21. Extract entity words from the input corpus as potential entity words;

[0087] The model is directly invoked to extract entity words of a custom type from the input corpus, and the extracted entity words and entity word information (such as entity word type) are used as potential entity words.

[0088] Entity word extraction directly from the input corpus can be configured according to the type of input corpus, the types of entities it contains, and the main object of analysis; no limitations are imposed here. To ensure the comprehensiveness of direct extraction, in some embodiments, direct entity word extraction from the input corpus can be performed according to the following steps:

[0089] S211. Perform representation normalization processing on the input corpus to obtain normalized sentences;

[0090] S212. Extract time and numerical values ​​from the normalized statement according to the regular expression, call the pre-trained NER model to extract the common name from the normalized statement, and use the extracted entity words and their corresponding entity word types as potential entity words.

[0091] For the input natural language question, Chinese characters are converted to numbers, and the unit representation is normalized. Then, regular expressions are used to extract patterns of time and numerical units. The entity extraction model used in this case is mainly used to extract names of people, places, and organizations from the input question. The model can be BiLSTM+CRF, trained on the People's Daily labeled dataset, but is not limited to this.

[0092] S22. Perform word segmentation and part-of-speech extraction on the input corpus to obtain language word segmentation;

[0093] After adding entities containing field values, field names, and table names to the custom dictionary, the user-input corpus is segmented and tagged with parts of speech. The resulting segments are then used as language segments.

[0094] It should be noted that this embodiment does not limit the word segmentation tool. For example, it can be done with the help of the Hanlp word segmentation tool. For details, please refer to the relevant technologies, which will not be elaborated here.

[0095] S23. Treat potential entity words and language segmentation as entity words.

[0096] In this embodiment, entity word extraction is divided into two parts: direct extraction based on corpus and word segmentation extraction. In addition to direct entity word extraction, word segmentation is performed on the corpus to supplement the vocabulary, which can ensure the comprehensiveness of information extraction.

[0097] In some embodiments, the process of calling the query language transformation model to assemble sentences based on the input corpus, syntactic analysis results, and interaction features between the target table and entity words in the above embodiments can be performed according to the following sub-steps:

[0098] S51. Encode the input corpus, syntactic analysis results, target table and entity words and then vectorize them as input features;

[0099] In the encoding layer, the user input question, table information, entity extraction results, and dependency parsing results are encoded and vectorized, with features separated by [SEP].

[0100] The encoding of the user input question contains a sequence of characters W = [w1, ..., w...]. i ,…,w n ], w i This represents the i-th character in a natural language question.

[0101] The table information includes column names, represented as C = [c1, ..., c2]. i c n ], c i This represents the column name of the i-th column in the associated table. Each column name consists of one or more characters, and the cell value corresponding to each column name is represented as c. i =[c i_1 c i_i c i_n ], c i_i This is represented by the encoding of the i-th character in the i-th list name;

[0102] The entity extraction results include entity words from the question, matched table entity words, and the corresponding field names and type names for the table entity words. WC = [[qw1, tw1, c1, t1], ..., [qw i,tw1,c i , t i ],...,[qw n ,tw n c n , t n ]], where qw i =[q1,..q n ], tw i =[t1,..t] n ], C i =[c1,..c m ]. qw i and q i S34 and S35 are the recall word pairs, the input word in the i-th group, and the i-th character of that word; tw i and t i These are word pairs, consisting of the table entity word in the i-th group and its i-th character; C i and c i These are word pairs, the table field name in the i-th group, and its i-th character; t i The types of word pairs include three types: VALUE, COLUMN, and TABLE.

[0103] Dependency parsing results are used to extract relation pairs between entity words and other words, S = [[w i w j ,s_type],...], where w i For the i-th entity word, w j For a certain and w i For words with dependency relationships, s_type represents the dependency relationship type. The overall input sequence is represented as X = "[XLS],W,[SEP],C,[SEP],WC,[SEP],S,[SEP]"; where [XLS] represents the start marker, W represents the natural language question, [SEP] represents the separator between segments, C represents the column name of the associated table, WC represents the word pairs of entity words, table entity words, and field names in the natural language question, and S represents the word pairs and dependency relationships formed by entity words and context words with dependency relationships in the natural language question.

[0104] S52. Perform multi-task joint prediction on the input features to obtain the prediction results of each sub-task; wherein, multi-task joint prediction includes: SELECT target column prediction, target column aggregation method prediction, WHERE condition target column prediction, WHERE condition logical operation prediction, WHERE condition numerical prediction, and WHERE condition comparison logic prediction.

[0105] In this embodiment, the prediction task is divided into 6 sub-tasks: sc (SELECT target column), sca (aggregation method of target column), wc (target column of WHERE condition), wco (logical operation between WHERE join conditions), wv (value of WHERE condition), and wo (comparison logic of WHERE condition).

[0106] The `sc` (SELECT target column) subtask performs a binary classification prediction on all fields of the target table to determine whether they belong to the `SELECT target column`. The `sca` (target column aggregation method) subtask performs a clustering operation on the target column predicted by `sc`, using the cluster type ("", 'MAX', 'MIN', 'COUNT', 'SUM', 'AVG', 'ORDER'). The system performs eight binary classification predictions (BY', HAVING'), where the first six operations take the category with the highest predicted value; wc(target column of WHERE condition) performs binary classification on the field of the word pair of type 'value' in the entity extraction result to determine whether it is the WHERE target column; wv(value of WHERE condition) extracts the question entity words from the word pair of type 'value' in the entity extraction result based on the prediction result of wc; wo(comparison logic of WHERE condition) performs a seven-class classification prediction by comparing the prediction results of the target column of type 'order' in sc and the target column of wc with the comparison type ('>', '<', '>=', '<=', '!=', 'asc', 'desc'); wco(logical operation between WHERE join conditions) performs a three-class classification prediction by judging the join logic between the target columns predicted by wc (", 'AND', 'OR').

[0107] During training, the prediction results of the six sub-tasks are compared with the target value to calculate the loss value, and the loss values ​​are summed to obtain the total loss value for the joint training of the six sub-tasks.

[0108] S53. Based on the prediction results of each subtask, assemble the clauses according to the SQL syntax concatenation rules to obtain the SQL statement.

[0109] In the actual prediction process, based on the prediction results of the 6 sub-tasks, clauses are assembled using SQL syntax to obtain SQL statements.

[0110] It should be noted that if the target columns in SELECT and WHERE belong to different tables, the tables can be joined based on the table schema information, introducing the JOIN syntax structure.

[0111] This embodiment provides an SQL concatenation method based on six-task training and prediction, which improves the accuracy and inference efficiency of the algorithm.

[0112] In some embodiments, before searching the table entity lexicon, the method further includes: determining the data retrieval table; extracting enumerated fields from the data retrieval table and extracting field information to establish a mapping table containing field values, field names, and table names; extracting field name and table name entity word records containing enumerated and numeric fields from the data retrieval table and adding them to the mapping table; calling the FastText word vector model to vectorize the field values; and using the mapping table as the table entity lexicon.

[0113] Specifically, the data retrieval table stores natural language information from the intelligent conversation system using a structured data storage and presentation method. In the table-based question-and-answer service, it performs the conversion from natural language to query language, facilitating the rapid generation of conversational responses. The table header consists of a series of field names, and the table data comprises the field values. Tables can be linked through one or more shared fields. For details on the specific format and storage method of the data retrieval table based on the intelligent conversation system, please refer to the relevant technical descriptions; they will not be elaborated upon here.

[0114] To fully extract prior information from the table and ensure rapid matching and comparison of subsequent input values, this embodiment records entity words and related information retrieved from the table using statistical data, constructing a table entity library to enhance the table information. Different tables contain different types of entity words, and the construction of the table entity library for each entity word type can be configured according to the application scenario and usage needs, without limitation here.

[0115] In some embodiments, retrieving entity words from a statistical data table can specifically involve extracting field values ​​from enumeration-type fields in the table, establishing a mapping table containing field values, field names, and table names, with the field type labeled as "value". Entity word records for field names and table names, containing both enumeration and numeric fields, are added, with types "column" and "table" respectively. Then, for the field value columns, a pre-trained word vector model using FastText (a word vector and text classification tool) is used for vectorization, and the vector representations of entity words are stored using Faiss (Facebook AI Similarity Search, a massive data repository). The FastText word vector model can be trained based on encyclopedia corpora.

[0116] The above construction method establishes an entity-level mapping relationship between field values ​​(VALUE), field names (COLUMN), and table names (TABLE) by summarizing the enumerated types of field values, field names, and table names. This relationship is used to associate with word segmentation in natural language and can adapt to most entity word types. In this embodiment, only the above construction method is used as an example for introduction. Other construction methods can refer to the above introduction and will not be elaborated here.

[0117] In some embodiments, table names and field names in the database are typically in English. To facilitate applications in Chinese question-and-answer scenarios and further extract prior information from the tables, information enhancement can be achieved by adding (Chinese) comments. Specifically, during database construction, a data dictionary (which defines and describes data items, data structures, data flows, data storage, and processing logic) is usually established, and table and field names are commented. Adding comments to table information can be done by supplementing the data dictionary's comment information, further refining it by using Chinese words and phrases to more fully explain the field and table names, their meanings, and uses. These comments are then manually verified to improve their quality. Manual verification primarily involves repairing and filling in incomplete, incorrectly described, and missing information, requiring the descriptions to be consistent with the actual information within the fields and to be as concise as possible.

[0118] Furthermore, in some embodiments, additional annotation information can be added to numerical fields to achieve information enhancement. The additional annotation information for numerical fields may include supplementary descriptions such as extracted unit information, maximum value information, and minimum value information, which serve as enhancement information for the field and generate a table attribute thesaurus for subsequent information association and matching needs.

[0119] It should be noted that, apart from statistical entity words, this embodiment only introduces information enhancement methods in two ways: improving Chinese annotations and supplementing numerical fields with annotation information. Other methods that can achieve data enhancement of data retrieval tables are also applicable to this application and will not be elaborated here.

[0120] To enhance understanding of the above method embodiments, this embodiment describes an overall process implementation that includes the main methods described above, such as... Figure 3 The diagram shown is a schematic representation of the overall method implementation process provided in this embodiment.

[0121] (1) Receive user input data;

[0122] (2) Perform grammatical error correction on the input corpus;

[0123] (3) Call the pre-established high-frequency FAQ SQL library to search for similar questions in the input corpus after grammar correction, and identify whether there are questions in the high-frequency FAQ SQL library with similarity higher than the threshold; if there are, directly extract the SQL response corresponding to the similar question from the high-frequency FAQ SQL library to generate the system response; if there are no similar questions, then execute step (4).

[0124] (4) Recognize the retrieval intent of the corrected input corpus; if it is not a retrieval intent, the system response can be generated based on the knowledge graph; if it is a retrieval intent, then proceed to step (5).

[0125] (5) Extract entity words and segment words based on the table entity lexicon and the table attribute lexicon, and treat the generated words as entity words;

[0126] The table entity thesaurus and the table attribute thesaurus are generated by performing data augmentation processing on the data retrieval tables.

[0127] (6) Perform syntactic analysis based on entity words and the input corpus after grammatical correction to obtain syntactic analysis results;

[0128] The syntactic analysis results, entity words, and the grammatically corrected input corpus are used as input features;

[0129] (7) Match the target table corresponding to the entity words based on the table schema data;

[0130] (8) Generate SQL query language based on the target table and input features.

[0131] It should be noted that this embodiment only introduces one process including the above method. The execution of other steps in different order can be referred to the description in this embodiment. The specific principles and implementation methods of each step can be referred to the description in the above method embodiment, and will not be repeated here.

[0132] Next, with reference to the accompanying drawings, a smart conversational device based on tabular data retrieval according to an embodiment of this application is described.

[0133] Figure 4 This is a block diagram of an intelligent session server based on tabular data retrieval according to an embodiment of this application.

[0134] like Figure 4 As shown, the intelligent session server 10 based on tabular data retrieval includes: a preprocessing module 100, an analysis module 200, a filtering module 300, a generation module 400, and a feedback module 500.

[0135] The preprocessing module 100 receives the input corpus, calls a syntax correction method, and obtains the corrected input corpus. The analysis module 200 queries a preset high-frequency question SQL database based on the corrected input corpus. If a match is found, the SQL result is returned; if not, the intent classification method is called based on the corrected input corpus. If no match is found, an "unable to answer" status is returned; if the search intent is found, entity extraction and syntactic analysis methods are called based on the input corpus to obtain the entity extraction and syntactic analysis results. The filtering module 300 queries the database based on the entity extraction results. The system retrieves table entity words from the input corpus, obtains query results for table entities, and, based on these results, calls a table retrieval method to obtain the target table and filters the corresponding table entity word information. The generation module 400 performs feature concatenation and encoding based on the entity extraction and syntactic analysis results from the input corpus, along with the table entity word information. Based on the encoded features, it calls a query language conversion model to generate an SQL statement. The feedback module 500 submits the SQL statement to the database execution engine, obtains the execution result, and returns it via an interface as a response to the input corpus.

[0136] Optionally, the generation module 400 is further used to obtain the target column of the query, the aggregation operation method of the target column, the filter condition target column, the comparison logic and comparison value corresponding to the filter condition target column, the classification prediction result of the logic between the filter conditions, and generate an SQL statement by combining the preset SQL syntax specification and the relationship between tables.

[0137] Optionally, the preprocessing module 100 is used to: segment the input corpus into words, and combine the window translation method to obtain word combinations with contextual features; score the word combinations using a pre-trained linguistic statistics model, and take the site with the largest deviation from the mean score as a potential error site; for the characters corresponding to the potential error sites, obtain a candidate character set using a preset homophone and similar-looking character dictionary and a preset common confusion dictionary; score the sentences in which the candidate characters replace the characters at the error sites again, and obtain the optimal result as the result after error correction.

[0138] Optionally, the analysis module 200 is used to: predict whether the input corpus belongs to a preset retrieval intent using a pre-trained text classification model; if it belongs to the preset retrieval intent, extract entity words based on the input corpus; if it does not belong to the preset retrieval intent, provide a non-retrieval intent prompt and perform the operation that the table data retrieval service cannot respond.

[0139] Optionally, the analysis module 200 is further configured to: extract entity words from the input corpus as potential entity words; perform word segmentation and part-of-speech extraction on the input corpus as language word segmentation; and combine potential entity words and language word segmentation as entity words.

[0140] Optionally, the analysis module 200 is further configured to: perform representation normalization processing on the input corpus to obtain normalized sentences; extract time and numerical values ​​from the normalized sentences according to regular expressions; call the pre-trained NER model to extract common names from the normalized sentences; and use the extracted entity words and their corresponding entity word types as potential entity words.

[0141] Optionally, the table entity recall and target table recall methods include: matching the table entity mapping table according to the entity words to obtain the recall results of the table entities; and querying the table relationship graph data based on the table entity recall results, and combining the path search algorithm to obtain the query results of the shortest path.

[0142] Optionally, the language conversion model includes: the main body of the model adopts the ERINE pre-trained model framework; the encoding layer uses the character features of the input corpus, the character features of the table column names, the pairing features of the corpus entities and the table entities, and the syntactic analysis features, and the encoded results are used as input; the inference layer uses the target column of the query, the aggregation operation method of the target column of the query, the target column of the filtering conditions, the comparison logic and comparison value corresponding to the target column of the filtering conditions, and the logic between the filtering conditions as the prediction target.

[0143] Optionally, the training method for the language conversion model includes: using the difference between the task prediction result and the actual result of the inference layer as the loss value, and using the minimum sum of loss values ​​as the iteration target for joint training.

[0144] Optionally, it also includes: a construction module for determining the data retrieval table before searching the table entity lexicon; extracting enumerated fields from the data retrieval table and extracting field information to establish a mapping relationship table containing field values, field names, and table names; extracting field name and table name entity word records containing enumerated and numeric fields from the data retrieval table and adding them to the mapping relationship table; calling the fasttext word vector model to vectorize the field values; and using the mapping relationship table as the table entity lexicon.

[0145] It should be noted that the foregoing explanation of the intelligent conversation method embodiment based on tabular data retrieval also applies to the intelligent conversation device based on tabular data retrieval in this embodiment, and will not be repeated here.

[0146] The intelligent conversational device based on table data retrieval proposed in this application constructs an entity lexicon for the data retrieval table, fully extracts the prior information of the table, enhances the table information, and improves the accuracy of the semantic expression of the table information. After receiving the input corpus, the matching comparison between the input corpus and the data retrieval table switches between the comparison of entity words in the input corpus and the table entity lexicon, providing a solution for input natural language and multi-table association queries, shortening the length of the comparison statement, and enabling fast matching. To ensure that the SQL statement corresponds accurately to the natural language, the input corpus, syntactic analysis results, and interaction features between the target table and entity words are input into the query language conversion model to construct the SQL statement. The interaction features between the target table and entity words and the syntactic information of the input natural language are introduced into the encoding layer of the query language conversion model. Using the target table obtained through matching as input, the feature network can be significantly reduced, lowering the complexity of the encoding layer and making it suitable for scenarios with complex tables and multiple fields and values. By using the interaction features between field values ​​and user questions as input features for sentence analysis and assembly, accurate correspondence between the corpus content and table fields can be ensured when the input only mentions field values. Extracting the syntactic analysis results from the input corpus for SQL statement assembly ensures accurate correspondence between SQL statements and natural language during the assembly process. Therefore, this device can adapt to diverse user questions, achieve accurate correspondence between SQL statements and natural language, while maintaining a small feature network, low encoding complexity, reduced implementation difficulty, and improved computational efficiency.

[0147] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0148] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0149] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0150] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0151] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0152] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0153] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0154] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A smart conversation method based on tabular data retrieval, characterized in that, Includes the following steps: Receive the input corpus, call the syntax correction method, and obtain the corrected input corpus; Based on the corrected input corpus, a preset high-frequency question SQL database is queried. If a match is found, the SQL result is returned. If no match is found, an intent classification method is called based on the corrected input corpus. If no match is found, an unanswerable state is returned. If a search intent is found, entity extraction and syntactic analysis methods are called based on the corrected input corpus to obtain entity extraction and syntactic analysis results. Based on the entity extraction results, search the table entity thesaurus to obtain the query results of the table entities, and based on the table entity query results, call the table recall method to obtain the target table and filter the corresponding table entity word information; Based on the entity extraction and syntactic analysis results of the error-corrected input corpus, and the entity word information in the table, feature concatenation and feature encoding are performed. Based on the encoded features, the query language conversion model is called to generate SQL statements. Based on the SQL statement, the database execution engine is submitted, and after obtaining the execution result, it is returned through the interface as a response to the corrected input corpus. The methods for recalling table entities and target tables include: Based on the entity words, match the table entity mapping table to obtain the recall results of the table entities; Based on the recall results of table entities, query the table relationship graph data and combine it with the path search algorithm to obtain the query results of the shortest path; The process of generating SQL statements based on encoded features by calling a query language transformation model includes: Obtain the target column for the query, the aggregation operation method for the target column, the target column for filtering conditions, the comparison logic and comparison value corresponding to the target column for filtering conditions, the classification prediction result of the logic between filtering conditions, and generate the SQL statement by combining the preset SQL syntax specifications and the relationship between tables. The step of invoking the intent classification method based on the corrected input corpus includes: For the corrected input corpus, a pre-trained text classification model is used to predict whether it belongs to the preset retrieval intent; If it belongs to the preset search intent, then entity words are extracted based on the corrected input corpus; If the search does not fall within the preset search intent, a non-search intent prompt will be displayed, and the operation that the table data search service cannot respond to will be executed. The step of extracting entity words based on the corrected input corpus includes: Extract entity words from the corrected input corpus as potential entity words; The corrected input corpus is segmented and part-of-speech extracted to serve as language segmentation. The potential entity words and the language segmentation are used as the entity words; The query language conversion model includes: The main body of the model adopts the ERNIE pre-trained model framework; The encoding layer uses the character features of the error-corrected input corpus, the character features of the table column names, the pairing features of corpus entities and table entities, and the syntactic analysis features, and encodes the results as input. The inference layer takes the target column of the query, the aggregation operation method of the target column, the filter condition target column, the comparison logic and comparison value corresponding to the filter condition target column, and the logic between the filter conditions as the prediction target. The training method for the query language conversion model includes: The difference between the task prediction results and the actual results of the inference layer is used as the loss value, and the minimum sum of the loss values ​​is used as the iteration objective for joint training. Before searching the table entity thesaurus, the following is also included: Define the data retrieval table; Extract the enumerated fields from the data retrieval table, extract the field information, and establish a mapping table containing field values, field names, and table names; Extract the field name and table name entity records containing enumeration and numeric fields from the data retrieval table and add them to the mapping relationship table; The field values ​​are vectorized using the FastText word vector model. The mapping table is used as the entity lexicon of the table.

2. The method according to claim 1, characterized in that, The process of receiving input corpus, invoking a syntax correction method, and obtaining corrected input corpus includes: The input corpus is segmented into words, and the window translation method is used to obtain word combinations with contextual features. For the word segmentation combination, a pre-trained linguistic statistics model is used for scoring, and the site with the largest deviation from the mean score is identified as a potential error site. For characters corresponding to potential error sites, a candidate character set is obtained by using preset homophones, similar-looking characters, and a preset common confusion dictionary; For statements where candidate characters replace characters at incorrect positions, a second evaluation is conducted to obtain the optimal result, which is then used as the corrected result.

3. The method according to claim 1, characterized in that, The step of extracting entity words from the corrected input corpus as potential entity words includes: The corrected input corpus is then subjected to representation normalization processing to obtain normalized sentences; The normalized statement is processed using regular expressions to extract time and numerical values. A pre-trained NER model is then called to extract generic names from the normalized statement. The extracted entity words and their corresponding entity word types are used as the potential entity words.

4. An intelligent session server based on tabular data retrieval, characterized in that, include: The preprocessing module is used to receive the input corpus, call the syntax correction method, and obtain the corrected input corpus. The analysis module is used to query a preset high-frequency question SQL database based on the corrected input corpus. If the query matches, the SQL result is returned. If the query does not match, the intent classification method is called based on the corrected input corpus. If the query does not match, the state of being unable to answer is returned. If the query intent is matched, entity extraction and syntactic analysis methods are called based on the corrected input corpus to obtain entity extraction and syntactic analysis results. The filtering module is used to search the table entity thesaurus based on the entity extraction results, obtain the query results of table entities, and call the table recall method based on the table entity query results to obtain the target table and filter the corresponding table entity word information. The generation module is used to perform feature concatenation and feature encoding based on the entity extraction and syntactic analysis results of the error-corrected input corpus and the entity word information in the table. Based on the encoded features, it calls the query language conversion model to generate SQL statements. The feedback module is used to submit the SQL statement to the database execution engine, obtain the execution result, and return it through the interface as a response to the corrected input corpus. The filtering module is also used to match the table entity mapping table according to the entity words to obtain the recall results of the table entities; based on the recall results of the table entities, query the table relationship graph data and combine it with the path search algorithm to obtain the query results of the shortest path; The generation module is further used to obtain the target column of the query, the aggregation operation method of the target column, the filter condition target column, the comparison logic and comparison value corresponding to the filter condition target column, the classification prediction result of the logic between the filter conditions, and generate the SQL statement by combining the preset SQL syntax specification and the relationship between tables. The analysis module is used to: predict whether the input corpus after error correction belongs to a preset retrieval intent using a pre-trained text classification model; If the search intent is corrected, entity words are extracted based on the corrected input corpus; if the search intent is not corrected, a non-search intent prompt is given, and the tabular data retrieval service cannot respond. The analysis module is further used to: extract entity words from the corrected input corpus as potential entity words; and perform word segmentation and part-of-speech extraction on the corrected input corpus as language word segmentation. The potential entity words and the language segmentation are used as the entity words; The query language conversion model includes: the main body of the model adopts the ERNIE pre-trained model framework; the encoding layer uses the character features of the error-corrected input corpus, the character features of the table column names, the pairing features of the corpus entities and the table entities, and the syntactic analysis features, and the encoded results are used as input; the inference layer uses the query target column, the aggregation operation method of the query target column, the filtering condition target column, the comparison logic and comparison value corresponding to the filtering condition target column, and the logic between the filtering conditions as the prediction target; The training method of the query language conversion model includes: using the difference between the task prediction result and the actual result of the inference layer as the loss value, and using the minimum sum of loss values ​​as the iteration target, to perform joint training; A construction module is used to determine the data retrieval table before searching the table entity lexicon; extract the enumerated fields from the data retrieval table and extract the field information to establish a mapping relationship table containing field values, field names, and table names; extract the field name and table name entity word records containing enumerated and numeric fields from the data retrieval table and add them to the mapping relationship table; call the FastText word vector model to vectorize the field values; and use the mapping relationship table as the table entity lexicon.

5. The server according to claim 4, characterized in that, The preprocessing module is used for: The input corpus is segmented into words, and the window translation method is used to obtain word combinations with contextual features. For the word segmentation combination, a pre-trained linguistic statistics model is used for scoring, and the site with the largest deviation from the mean score is identified as a potential error site. For characters corresponding to potential error sites, a candidate character set is obtained by using preset homophones, similar-looking characters, and a preset common confusion dictionary; For statements where candidate characters replace characters at incorrect positions, a second evaluation is conducted to obtain the optimal result, which is then used as the corrected result.

6. The server according to claim 4, characterized in that, The analysis module is further used for: The corrected input corpus is then subjected to representation normalization processing to obtain normalized sentences; The normalized statement is processed using regular expressions to extract time and numerical values. A pre-trained NER model is then called to extract generic names from the normalized statement. The extracted entity words and their corresponding entity word types are used as the potential entity words.

Images