A system and method for semantic analysis using primitive-level enumeration.

Abstract

The embodiments described herein provide a semantic analysis framework that can be called Uni-Parser. The Uni-Parser framework may be applied to question answering in both a knowledge base and a database. The three main stages of the Uni-Parser framework are enumeration, ranking, and generation. In the enumeration stage, primitives are enumerated based on matching the question to a data structure. After enumerating the primitives, the Uni-Parser framework may rank the primitives using a trained ranker model. Then, the top-ranked primitives may be used as input to a generator, which is a sequence-to-sequence model that creates a logical form.

Description

Technical Field

[0001] This disclosure claims priority to U.S. Non-Provisional Patent Application No. 18 / 059,691, filed Nov. 29, 2022, which claims priority to U.S. Provisional Patent Application No. 63 / 355,438, filed Jun. 24, 2022, the entireties of which are hereby expressly incorporated by reference herein.

[0002] Embodiments generally relate to natural language processing and machine learning systems, and more specifically, to systems and methods for semantic analysis using primitive level enumeration for question answering.

Background Art

[0003] In database operations such as retrieving results in response to a search query, often a specific form of command is required. To write commands in a specific logical form, a user needs to acquire a high-level database language. Machine learning systems are widely used to convert natural language questions into queries in a format used by a database or knowledge base. In this way, a user may enter a natural language query such as “what is the month having the highest revenue in the past five years?”. Analyzing natural language questions into an executable logical form is a useful and interpretable way to perform question answering on structured data. Existing approaches enumerate executable logical forms, and when dealing with complex questions having multi-hop / multi-table relationships, the number of enumerated logical forms increases exponentially. Further, existing approaches rely on different model structures for different data modalities (e.g., knowledge base and database) rather than a unified adaptable structure. Therefore, there is a need for improved systems and methods for semantic analysis.

Brief Description of the Drawings

[0004] [Figure 1] This is a simplified diagram illustrating semantic analysis frameworks in several embodiments.

[0005] [Figure 2] This is a simplified diagram illustrating an enumeration of primitives for a knowledge base in several embodiments.

[0006] [Figure 3] This is a simplified diagram illustrating an enumeration of primitives for a database in several embodiments.

[0007] [Figure 4] This is a simplified diagram illustrating primitive ranking for a knowledge base in several embodiments.

[0008] [Figure 5] This is a simplified diagram illustrating primitive ranking for a database in several embodiments.

[0009] [Figure 6] This is a simplified diagram illustrating the generation of a logical format for a knowledge base using several embodiments.

[0010] [Figure 7] This is a simplified diagram illustrating the generation of a logical format for a database using several embodiments.

[0011] [Figure 8] This is a simplified diagram illustrating a computing device that implements the semantic analysis framework shown in Figures 1 to 7, according to one embodiment described herein.

[0012] [Figure 9] Figures 1 to 7 show a simplified block diagram of a networked system suitable for implementing the semantic analysis framework and other embodiments described herein.

[0013] [Figure 10] An exemplary logical flow diagram illustrating a semantic analysis method based on the frameworks shown in FIGS. 1-9 according to some embodiments described herein.

[0014] [Figure 11] Provide a chart illustrating the exemplary performance of different embodiments described herein. [Figure 12] Provide a chart illustrating the exemplary performance of different embodiments described herein. [Figure 13] Provide a chart illustrating the exemplary performance of different embodiments described herein. [Figure 14] Provide a chart illustrating the exemplary performance of different embodiments described herein. [Figure 15] Provide a chart illustrating the exemplary performance of different embodiments described herein.

[0015] [Figure 16A] Provide an exemplary comparison between the output of the semantic analysis framework described in the embodiments herein and another semantic analyzer. [Figure 16B] Provide an exemplary comparison between the output of the semantic analysis framework described in the embodiments herein and another semantic analyzer.

[0016] The embodiments of the present disclosure and their advantages are best understood by referring to the following detailed description. It should be understood that like reference numerals are used to identify like elements illustrated in one or more of the drawings, and the drawings are shown for the purpose of exemplifying the embodiments of the disclosure and not for the purpose of limiting it.

DETAILED DESCRIPTION OF THE INVENTION

[0017] As used herein, the term “Network” may include any artificial intelligence network or system, neural network or system, and / or any hardware or software-based framework including any training or learning model implemented therein or with it.

[0018] As used herein, the term “module” may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0019] Parsing natural language questions to generate executable logical forms that can be performed on a database or knowledge base is a useful and interpretable way to perform question-answering on structured data. For example, a natural language question such as "How many students are enrolled in Statistics?" can be answered given a database containing student data. A semantic parser can generate a logical form that can be executed by the database to provide the answer. For example, SELECT count(*) FROM Courses JOIN Course_Attendance ON Courses.Courseld = Course_Attendance.CourseId WHERE Courses.CourseName = “Statistics”. Existing approaches for semantic parsing enumerate executable logical forms. For example, when data is structured as a knowledge base, entities may be interconnected by edges representing relationships. Existing approaches start with entities and enumerate logical forms that contain relationships of two “hops” within the structure. Given M connections from a starting entity to an entity one hop away, and N connections from each of these entities, M*N logical forms are obtained. In sufficiently complex data structures, this leads to inefficient resource usage and performance degradation. Furthermore, existing approaches do not provide a unified structure applicable to both knowledge bases and databases.

[0020] In light of the need for improved systems and methods for semantic analysis, embodiments described herein provide a semantic analysis framework (referred to as "Uni-Parser") that can be applied to question answering in both knowledge bases and databases. The Uni-Parser framework may, through enumeration, ranking, and generation, generate one or more logical forms that can be executed on data structures in response to natural language input. In the enumeration stage, primitives are enumerated based on matching natural language questions to data structures (e.g., either databases or knowledge bases). For example, primitives in a database may be column names of a particular table, or values ​​of a particular cell in a table (or operations that provide relevant cell values ​​as output). In a knowledge base, primitives may be represented by nodes in a graph. This is the opposite of a logical form consisting of primitives and corresponding operations on them, for example, in a database, it might be SELECT count(*) FROM Courses JOIN Course_Attendance ON Courses.CourseID = Course_Attendance.CourseID WHERE Courses.CourseName = “Statistics”. This logical form includes several primitives corresponding to a specific table column name such as “Courses.CourseID” and a cell value such as “Statistics”. Rather than enumerating all possible logical forms using all different variations and / or combinations of primitives, Uni-Parser enumerates the primitives themselves. In this way, instead of enumerating M*N logical forms (where M is the number of first-hop entities and N is the number of second-hop entities from a particular entity in a natural language question in the knowledge base), Uni-Parser may enumerate M+N primitives.

[0021] After enumerating the primitives, the Uni-Parser framework may rank the primitives using a trained ranker model. The top-ranked primitives may then be used as input to a generator, which is a trained sequence-to-sequence model. The sequence-to-sequence model may then produce an output sequence in logical form in response to the input sequence of the top-ranked primitives. Details of the enumeration, ranking, and generation stages are described in more detail below with respect to Figures 1 to 7.

[0022] The embodiments described herein offer several advantages. For example, the same semantic analysis framework may be used for question answering in both databases and knowledge bases. By enumerating primitives rather than logical forms, the search space is significantly reduced, and candidate generation and ranking become more efficient and scalable. Furthermore, the construction of logical forms from primitives and operations is deferred to the generation stage. This leads to a more generalized model that can operate on complex logical forms and generalize to questions involving invisible schemas.

[0023] Figure 1 is a simplified diagram illustrating several embodiments of a semantic analysis framework. Framework 100 includes an enumerator 104 operably connected to a ranker 106, which is operably connected to a generator 108. A natural language question 102 (e.g., "How many students are enrolled in Statistics?") may be the input to the enumerator 104. Given the question 102 and a known data structure (e.g., a knowledge base containing multiple entities interconnected by edges representing relationships between entities, or a database containing one or more tables, each table having its own columns and rows for storing values), the enumerator 104 may enumerate several primitives. For example, for a given entity shown in question 102, the first and second hop entities from a given entity in the knowledge base may be enumerated as primitives. The enumeration of primitives is described in more detail with respect to Figure 2 for knowledge bases and Figure 3 for databases.

[0024] The enumerated primitives may be passed to a ranker 106 that ranks the primitives based on question 102. Ranker 106 may be a trained model that has been trained on similar or identical data structures. Ranker 106 is described in more detail in Figure 4 for knowledge bases and in Figure 5 for databases.

[0025] The higher-ranked primitives may be passed to the generator 108. Based on questions 1-2 and the ranked primitives, the generator 108 may produce a logical form 110. The logical form 110 may be executed on a data structure to provide an answer to question 102. The generator 108 may be a trained model that has been trained on similar or identical data structures. The generator 108 is described in more detail in Figure 6 for knowledge bases and in Figure 7 for databases.

[0026] Figure 2 is a simplified diagram illustrating the enumeration of primitives for a knowledge base according to several embodiments. Based on a provided question 202, an enumerator (e.g., enumerator 104 in Figure 1) may enumerate primitives from the knowledge base 204. The enumerator may begin by discovering entities in question 202, for example by using a named entity recognition (NER) algorithm. For example, named entities such as "currency", "New York", "Central Park", and "construction" may be identified in question 202. In some embodiments, the NER algorithm identifies entities in question 202 based on belonging to a given class of entities (e.g., place, person, etc.).

[0027] Next, entities identified in question 202 may be matched (or fuzzy matched) with entities in knowledge base 204. For example, "Central Park" may be selected as the starting entity based on matching this entity with the phrase "central park" in question 202.

[0028] In some embodiments, only one entity is selected from the knowledge base 204, while in other embodiments, two or more entities may be selected. The matched entity(s) may be used as a starting point for enumeration. To mitigate the issue of entity disambiguation, a ranker model may be used to select entity candidates based on the similarity between the question and the one-hop in-and-out relationship of the entities.

[0029] After selecting one or more starting entities, the enumerator may enumerate all primitives within two hops of the starting entities in the knowledge base. Entities one hop away from “Central Park,” as shown and enumerated in Listing 206, as reflected in Knowledge Base 204, may include items such as “architecture.construction_cost central_park” and “architecture.landscape_project central_park.” Two hops away from “central park,” additional entities may be included, such as “travel.get_destination” and “money_unit.currency.” As shown in Listing 206, the enumeration is performed at the primitive level, i.e., the basic unit of the knowledge base, rather than as a query consisting of primitives and operations on primitives.

[0030] Figure 3 is a simplified diagram illustrating the enumeration of primitives for a database in several embodiments. Based on the provided question 302, an enumerator (e.g., enumerator 104 in Figure 1) may enumerate primitives from database 304. In a database, primitives for an input question may include column names or cell values. For example, in a database containing one or more data tables, all column names of all data tables may be enumerated first, and only related cell values ​​within one or more data tables may be enumerated next based on the input question. Since the number of column names is generally manageable even in very large databases, all column names of all tables in the database may be enumerated. For example, as shown in enumeration listing 310, column names (associated with those tables) are enumerated including Students.StudentId, Students.StudentName (both from table 306), and columns from table 308 that may be included are not shown, but column names from table 310 are also included, as illustrated by the exemplary Courses.CourseName entry.

[0031] Enumerating individual cells would be inefficient due to the sheer volume of cells. Instead of enumerating all cells, the enumerator may detect entities within question 302, for example, by using a named entity recognition algorithm, such as "students," "course," and "statistics." The entities in question 302 may then be matched (or fuzzy matched) with cell values ​​in database 304. For example, as illustrated, question 302 contains the word "statistics," which matches the cell value "Statistics" in the CourseName column of table 308 for all courses. Thus, this maintains the relationship between table 308 and the CourseName column: "Courses.CourseName <op>As listed in enumerated list 310, it is referred to as "Statistics". In some embodiments, if question 302 contains a number and string matching fails to find a match, the value is paired with all column names in the database.

[0032] Figure 4 is a simplified diagram illustrating primitive ranking for a knowledge base according to several embodiments. Enumeration lists 402 and 404 are lists of first-hop and second-hop primitives, as enumerated by an enumerator (e.g., enumerator 104), as described with reference to Figures 1 and 2. In some embodiments, the same ranker model 106 may be used for primitives of both classes (e.g., both first-hop and second-hop).

[0033] In some embodiments, as illustrated in Figure 4, the enumeration list provided by the enumerator may be a concatenation of the question, special tokens indicating the first / second hop, and the primitive itself (either by the enumerator, ranker, or another unit). For example, enumeration list 404 begins with the text of question 202 in Figure 2 and the special first hop token " <firsthop>This includes items with primitives such as "architecture.construction_cost central_park". Table 402 includes similar items with Question 202, the second hop token, and then primitives concatenated thereafter. These are exemplary concatenation patterns, and other concatenation patterns may be used. The enumerated list is provided to Ranker 106.

[0034] (As illustrated in Figure 1) Ranker 106 may be trained to eliminate irrelevant primitives by measuring the similarity between the question and the enumerated primitive candidates (for example, as shown in Lists 402 and 404). In some embodiments, Ranker 106 utilizes a cross-encoder architecture. Question X and category token p c Given primitives p having , ranker 106 may use a BERT-based encoder that takes a concatenation of their vector representations as input and outputs a logit representing the similarity between the primitives and the question, i.e.,

number

number

number

number

number

[0035] When training Ranker 106, negative sampling may be performed to pair negative primitive samples with positive samples. In some embodiments, random sampling is used to select negative samples. In other embodiments, the sampling policy samples hard negative candidates to train Ranker 106. In the case of a knowledge base, hard negative candidates at the second hop may be sampled from primitives connected to the ground truth first hop. Furthermore, a bootstrap-type negative sampling policy may be utilized, i.e., the model may be recursively trained using false positive candidates generated from the last training epoch.

[0036] During inference, ranker 106 may generate output values ​​associated with each of the enumerated primitives. Based on these values, the primitives may be ranked. In some embodiments, first-hop / second-hop primitives are ranked separately (or, in the case of a database, column name / cell value). This is illustrated for a knowledge base as ranked lists 406 and 408 in Figure 4. In some embodiments, the top k primitives of each category may be selected and passed to a generator (e.g., generator 108). For example, k may be set to 5. In a knowledge base, the top-ranked first-hop and second-hop primitives may be formed into a two-hop path by combining one first-hop primitive with each of the second-hop primitives. However, to provide generator 108 with valid primitive candidates, they may be further filtered to remove second-hop primitives that are not reachable from any of the first-hop primitives.

[0037] Figure 5 is a simplified diagram illustrating primitive ranking for a database according to several embodiments. Enumeration lists 502 and 504 are lists of column name and cell value primitives, as described with reference to Figures 1 and 3, and enumerated by an enumerator (e.g., enumerator 104).

[0038] In some embodiments, as illustrated in Figure 5, the enumeration list provided by the enumerator may be a concatenation of the question, a special token indicating whether the primitive is a column name or a cell value, and the primitive itself (either by the enumerator, ranker, or another unit). For example, enumeration list 502 begins with the text of question 302 in Figure 3 and the special cell value token "<tb_cl_vl> ", and the primitive "Courses.CourseName <op>It contains a single item with "Statistics". Table 504 contains similar items with Question 202, a column name token, followed by a primitive concatenated thereafter. These are exemplary concatenation patterns, and other concatenation patterns may be used. An enumerated list is provided to Ranker 106.

[0039] The training and use of Ranker 106 may be substantially the same as described with reference to the database in Figure 4. The primitives of the enumerated database are applied, not the primitives of the enumerated knowledge base. When selecting hard negatives in the database to train Ranker 106,<tb_cl> Regarding category primitives, the training method may treat those with the same table name as the ground truth but different column names as hard negatives.<tb_cl_vl> Regarding categories, the training method may treat candidates with the same table and column names as the ground truth but with different cell values ​​as hard negatives.

[0040] Ranked lists 506 and 508 similarly provide rankings of database primitives, as described with respect to ranked lists 406 and 408 in Figure 4 for the knowledge base.

[0041] Figure 6 is a simplified diagram illustrating the generation of logical forms for a knowledge base according to several embodiments. In the knowledge base, the generator 108 receives input 602, which are primitives (concatenated with questions and special tokens) ranked highly by ranker 106. In some embodiments, the primitive order may be randomized so that the generator 108 must learn to use semantic meaning instead of misusing positional information to select highly ranked primitives.

[0042] The generator 108 may learn to generate logical forms (queries composed of primitives that can be executed on corresponding data structures) by understanding the meaning of its elements (primitives and operations) and constructing them. The generator 108 may be trained to generate an output logical form optimized by mutual entropy loss for each token. In inference, beam search may be used to decode the top k target logical forms in an autoregressive manner.

[0043] An exemplary logical form output 604 is shown. As illustrated, output 604 is (JOIN money_unit.currency (JOIN architecture.construction_cost central_park)). Note that a logical form consists of primitives containing knowledge base entities and operations (e.g., JOIN) performed on those entities. This logical form is used as a query against the knowledge base to provide answers to questions (e.g., questions 102 or 202).

[0044] Figure 7 is a simplified diagram illustrating the generation of a logical format for a database in several embodiments. In general, training and using the generator 108 for the database is the same as described for the knowledge base in Figure 6.

[0045] In the database, generator 108 contains primitives ranked higher than ranker 106, but [Question; |table_name1| column_name1, column_name2 <op>It accepts a formatted input 702 as value ...|table_name|2 ...]. In the database, the formatting, which organizes column names and cell values ​​according to the table to which they belong, naturally changes their order from a ranked order.

[0046] As illustrated, the output may be generated as output 704, which is SELECT count(*) FROM Courses JOIN Course_Attendance ON Courses.CourseId = Course_Attendance.CourseId WHERE Courses.CourseName = "Statistics". Similar to the logical format shown in Figure 6, this logical format output 704 may be used as a query against the database to provide answers to questions (e.g., questions 102 or 302).

[0047] Figure 8 is a simplified diagram illustrating a computing device implementing the Uni-Parser framework described in Figures 1 to 7, according to one embodiment described herein. As shown in Figure 8, the computing device 800 includes a processor 810 coupled to memory 820. The operation of the computing device 800 is controlled by the processor 810. Although the computing device 800 is shown having only one processor 810, it is understood that the processor 810 may represent one or more central processing units, multicore processors, microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), etc., within the computing device 800. The computing device 800 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0048] Memory 820 may be used to store software executed by computing device 800 and / or one or more data structures used during the operation of computing device 800. Memory 820 may include one or more types of machine-readable media. Some common forms of machine-readable media may include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0049] The processor 810 and / or memory 820 may be located in any preferred physical arrangement. In some embodiments, the processor 810 and / or memory 820 may be implemented on the same board, the same package (e.g., system-in-package), the same chip (e.g., system-on-a-chip), etc. In some embodiments, the processor 810 and / or memory 820 may include distributed, virtualized, and / or containerized computing resources. Inconsistent with such embodiments, the processor 810 and / or memory 820 may be located in one or more data centers and / or cloud computing facilities.

[0050] In some examples, memory 820 may include non-temporary tangible machine-readable media containing executable code that, when operated by one or more processors (e.g., processor 810), can cause one or more processors to perform methods further described herein. For example, as illustrated, memory 820 includes instructions for a Uni-Parser module 830 which may be used to implement and / or emulate a system and model and / or to implement any of the methods further described herein. The Uni-Parser module 830 may receive input 840, such as input training data (e.g., questions and corresponding logical forms) via a data interface 815, and generate output 850 which may be in logical form based on a question to a given data structure.

[0051] The data interface 815 may include a communication interface and a user interface (such as a voice input interface or a graphical user interface). For example, the computing device 800 may receive input 840 (such as a training dataset) from a networked database via the communication interface. Alternatively, the computing device 800 may receive input 840 from a user via the user interface, such as a question and / or a data structure.

[0052] In some embodiments, the Uni-Parser module 830 is configured to generate a logical format for answering a question using a provided data structure. The Uni-Parser module 830 may further include an enumerator submodule 831 (e.g., similar to the enumerator 104 in Figure 1), a ranker submodule 832 (e.g., similar to the ranker 106 in Figure 1), and a generator submodule 833 (e.g., similar to the generator 108 in Figure 1). In one embodiment, the Uni-Parser module 830 and its submodules 831-833 may be implemented by hardware, software, and / or a combination thereof.

[0053] Some examples of computing devices, such as computing device 800, may include non-temporary tangible machine-readable media containing executable code that, when operated by one or more processors (e.g., processor 810), can cause one or more processors to execute the process of the Method. Some common forms of machine-readable media that may contain the process of the Method are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0054] Figure 9 is a simplified block diagram of a networked system suitable for implementing the Uni-Parser framework described in Figures 1 to 7 and other embodiments described herein. In one embodiment, block diagram 900 shows a system including a user device 910 that can be operated by a user 940, data vendor servers 945, 970 and 980, a server 930, and other forms of devices, servers, and / or software components that operate to perform various methodologies according to the embodiments described. Illustrative devices and servers may include devices, standalone and enterprise-class servers, which may be similar to the computing device 800 described in Figure 8, and run an OS such as MICROSOFT® OS, UNIX® OS, LINUX® OS, or other suitable device and / or server-based OS. The devices and / or servers illustrated in Figure 9 may be deployed in other ways, and the operations and / or services provided by such devices and / or servers may be combined or separated for a given embodiment, and may be performed by a larger or smaller number of devices and / or servers. One or more devices and / or servers may be operated and / or maintained by the same or different entities.

[0055] The user device 910, data vendor servers 945, 970, and 980, and server 930 can communicate with each other via network 960. The user device 910 may also be used by a user 940 (e.g., a driver, system administrator, etc.) to access various functions available to the user device 910, which may include processes and / or applications associated with server 930 to receive output data anomaly reports.

[0056] The user device 910, the data vendor server 945, and the server 930 may each include one or more processors, memory, and other suitable components for executing instructions such as program code and / or data stored in one or more computer-readable media to implement the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer-readable media, such as memory or data storage devices, inside and / or outside of various components of the system 900, and / or may be accessible via the network 960.

[0057] The user device 910 may be implemented as a communication device that can utilize appropriate hardware and software configured for wired and / or wireless communication with the data vendor server 945 and / or server 930. For example, in one embodiment, the user device 910 may be implemented as other types of computing devices capable of transmitting and / or receiving data, such as an autonomous vehicle, a personal computer (PC), a smartphone, a laptop / tablet computer, a wristwatch with appropriate computer hardware resources, eyeglasses with appropriate computer hardware (e.g., GOOGLE GLASS®), other types of wearable computing devices, an implantable communication device, and / or an iPad® from APPLE®. Although only one communication device is shown, multiple communication devices may function similarly.

[0058] The user device 910 in Figure 9 includes a user interface (UI) application 912 and / or other applications 916, which may correspond to applications having executable processes, procedures, and / or associated hardware. For example, the user device 910 may receive a message from the server 930 in logical form or a direct answer to a question and display that message via the UI application 912. In other embodiments, the user device 910 may include additional or different modules having dedicated hardware and / or software, as needed.

[0059] In various embodiments, the user device 910 may include other applications 916 where it is desired to provide functionality to the user device 910 in a particular embodiment. For example, other applications 916 may include security applications for implementing client-side security functions, programmatic client applications for interfacing with appropriate application programming interfaces (APIs) via the network 960, or other types of applications. Other applications 916 may also include communication applications such as email, text, voice, social networking, and IM applications that enable the user to send and receive emails, phone calls, texts, and other notifications via the network 960. For example, other applications 916 may be email or instant messaging applications that receive query result messages from the server 930. Other applications 916 may include device interfaces and other display modules capable of receiving input and / or output information. For example, other applications 916 may include processor-executable software programs for asset management, including a graphical user interface (GUI) configured to provide the user 940 with an interface for viewing answers to questions.

[0060] The user device 910 may further include a database 918 stored in the temporary and / or non-temporary memory of the user device 910, which may store various applications and data and be used during the execution of various modules of the user device 910. The database 918 may store a user profile associated with user 940, predictions previously viewed or saved by user 940, historical data received from server 930, and the like. In some embodiments, the database 918 may be local to the user device 910. However, in other embodiments, the database 918 may be external to the user device 910 and accessible by the user device 910, which includes a cloud storage system and / or database accessible via network 960.

[0061] The user device 910 includes at least one network interface component 917 configured to communicate with the data vendor server 945 and / or server 930. In various embodiments, the network interface component 917 may include various other types of wired and / or wireless network communication devices, including DSL (e.g., Digital Subscriber Line) modems, PSTN (Public Switched Telephone Network) modems, Ethernet devices, broadband devices, satellite devices, and / or microwave, radio frequency, infrared, Bluetooth, and short-range wireless communication devices.

[0062] The data vendor server 945 may provide the server 930 with a training dataset containing questions and their corresponding logical formats, corresponding to the server hosting the database 919. The database 919 may be implemented by one or more relational databases, distributed databases, cloud databases, etc.

[0063] The data vendor server 945 includes at least one network interface component 926 adapted to communicate with the user device 910 and / or the server 930. In various embodiments, the network interface component 926 may include various other types of wired and / or wireless network communication devices, including DSL (e.g., Digital Subscriber Line) modems, PSTN (Public Switched Telephone Network) modems, Ethernet devices, broadband devices, satellite devices, and / or microwave, radio frequency, infrared, Bluetooth, and short-range wireless communication devices. For example, in one implementation, the data vendor server 945 may transmit asset information from the database 919 to the server 930 via the network interface 926.

[0064] Server 930 may house the Uni-Parser module 830 and its submodules as shown in Figure 1. In some implementations, the Uni-Parser module 830 may receive data from the database 919 in the data vendor server 945 via the network 960 and generate logical forms and / or direct answers to questions. The generated logical forms and / or answers may be sent via the network 960 to the user device 910 for review by the user 940.

[0065] The database 932 may be stored in the temporary and / or non-temporary memory of the server 930. In one implementation, the database 932 may store data obtained from the data vendor server 945. In one implementation, the database 932 may store parameters of the Uni-Parser module 830. In one implementation, the database 932 may store previously generated logical forms and corresponding input feature vectors.

[0066] In some embodiments, the database 932 may be local to the server 930. However, in other embodiments, the database 932 may be external to the server 930 and accessible by the server 930, which includes a cloud storage system and / or database accessible via the network 960.

[0067] Server 930 includes at least one network interface component 933 adapted to communicate with user devices 910 and / or data vendor servers 945, 970, or 980 via network 960. In various embodiments, the network interface component 933 may include various other types of wired and / or wireless network communication devices, including DSL (e.g., Digital Subscriber Line) modems, PSTN (Public Switched Telephone Network) modems, Ethernet devices, broadband devices, satellite devices, and / or microwave, radio frequency (RF), and infrared (IR) communication devices.

[0068] Network 960 may be implemented as a single network or as a combination of multiple networks. For example, in various embodiments, network 960 may include the Internet or one or more intranets, terrestrial networks, wireless networks, and / or other suitable types of networks. Thus, network 960 may correspond to small-scale communication networks such as private or local area networks, or large-scale networks such as wide-area networks or the Internet, accessible by various components of system 900.

[0069] The knowledge base and / or database used for queries using the enumeration of primitives and the generated logical forms may be stored in one of the exemplified network devices, such as data vendor server 945, server 930, or user device 910. Such a knowledge base / database may or may not be associated with the databases described above (i.e., databases 918, 919, and 932), or may or may not be embedded in them.

[0070] Figure 10 is an exemplary logic flow diagram illustrating a semantic analysis method 1000 based on the Uni-Parse framework shown in Figures 1 to 9, according to some embodiments described herein. One or more of the processes of method 1000 may be implemented, at least in part, in the form of executable code stored in a non-temporary tangible machine-readable medium that causes one or more processors to execute one or more of the processes when executed by one or more processors. In some embodiments, method 1000 corresponds to the operation of a Uni-Parser module 830 (e.g., Figures 8 to 9) that performs semantic analysis.

[0071] As illustrated, Method 1000 includes several enumerated steps, but embodiments of Method 1000 may include additional steps before, after, and between the enumerated steps. In some embodiments, one or more of the enumerated steps may be omitted or performed in a different order.

[0072] In step 1001, the system receives a natural language question (e.g., question 102, 202, or 302) via a communication interface. For example, the question may be entered by a user into a user interface. The communication interface may be, for example, a computer network interface, and the natural language question is received by the server (e.g., server 930) from a user device (e.g., user device 910) via the network interface (e.g., network interface 933).

[0073] In step 1002, the system identifies a first entity from the natural language question via a named entity recognition (NER) procedure. For example, the first entity may be a node in a knowledge graph, such as "Central Park" as described in Figure 2. In a database, the first entity may be a cell value, such as "Statistics" as shown in Figure 3.

[0074] In step 1003, the system enumerates (for example, by enumerator 104) several primitives that represent entities or entity relationships in a database or knowledge base based on the first entity. For example, in a database, the primitives may include table column names and cell values ​​(including cell values ​​that match the first entity described in step 1002). For example, in a knowledge base, the primitives may include knowledge base entities and entity relationships (including knowledge base entities that match the first entity described in step 1002). In some embodiments, the knowledge base entity that matches the first entity is the starting entity, from which other primitives are enumerated, and the other primitives are one or two hops away from the starting entity.

[0075] In step 1004, the system ranks several primitives (e.g., by ranker 106) based on their relevance to the natural language question. In some embodiments, a separate ranked list is generated for each class of primitives (e.g., first hop and second hop, or column name and cell value). The primitives enumerated in step 1004 may be provided to the ranker in a form concatenated with special tokens that identify the natural language question and / or the class of primitives (e.g., first hop, second hop, column name, or cell value). The ranker may be a model (e.g., a neural network model) trained to minimize contrast loss. The contrast loss may be based on positive and negative primitive samples extracted from the ground truth logic form. The method for selecting negative primitive samples may be random, or they may be selected as “hard” samples that can improve training. For example, in a knowledge base, negative samples may be selected from primitives connected within the knowledge base to first-hop entities extracted from the ground truth logical form. In a database, negative samples may be selected by selecting cell values ​​in the same table and column as the cell values ​​extracted from the ground truth logical form.

[0076] In step 1005, the system selects a subset of higher-ranked primitives based on the ranking. This may be performed by ranker 106, generator 108, or another component of the system. The quantity of primitives selected may be based on a predetermined value. If the ranked list contains multiple ranked lists (e.g., a "first hop" list and a "second hop" list), each list may be ranked individually, and higher-ranked primitives may be selected from each list.

[0077] In step 1006, the system generates a logical form that can be executed on a database or knowledge base, based on a natural language question and a subset of several primitives (for example, by generator 108). The executable logical form may include at least one primitive from the subset. The logical form may be, for example, logical form 604 or 704.

[0078] In step 1007, the system transmits the logical format to a database system or knowledge base system via a communication interface. For example, the database system or knowledge base system may be a server such as the data vendor server 945 shown in Figure 9, and the logical format is transmitted to that server so that the server can execute the logical format on the database (or knowledge base).

[0079] In step 1008, the system receives query results via a communication interface, which are responses to natural language questions based on a logical format. The answers may be displayed to the user, for example, on the same user interface used to input the question.

[0080] Figures 11 to 15 provide charts illustrating exemplary performance of different embodiments described herein. Bert Ranking as described in Gu et al., Beyond iid: three levels of generalization for question answering on knowledge bases, In Proceedings of the Web Conference, pages 3477-3488, 2021; ReTrack as described in Chen et al., Retrack: a flexible and efficient framework for knowledge base question answering, in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 325-336, 2021; UnifiedSKG as described in Xie et al., Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models, arXiv: 2201.05966, 2022; Ye et al., Rng-kbqa: Generation augmented iterative ranking for knowledge base question answering. Examples include RNG-KBQA as described in arXiv:2109.08678, 2021, Topic Units as described in Lan et al., Knowledge base question answering with topic units, 2019, and Yih et al., Semantic parsing via staged query graph generation: Question answering with knowledge base, In Proceedings of the Joint Conference of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing of the AFNLP, 2015, STAGG, Lan et al.; Query graph generation for answering multi-hop complex questions from knowledge bases, Association for Computational Linguistics, 2020, QGG, Das et al.; Case-based reasoning for natural language queries over knowledge bases, arXiv: 2104.08762, 2021, CBR, Gu and Su; ArcaneQA, Gu and Su; Arcaneqa: Dynamic program induction and contextualized encoding for knowledge base question answering, arXiv: 2204.08109, 2022, ArcaneQA, Bogin et al.; Global reasoning over database structures for text-to-sql parsing, arXiv: Examples include Global-GNN (as described in 1908.11214, 2019), EditSQL (as described in Zhang et al., Editing-based sql query generation for cross-domain context-dependent questions, arXiv: 1909.00786, 2019), and Scholak et al.T5-Base as described in Picard: Parsing incrementally for constrained auto-regressive decoding from language models, arXiv: 2109.05093, 2021, Wang et al.; RAT-SQL as described in Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers, arXiv: 1911.04942, 2019, Lin et al.; BRIDGE as described in Bridging textual and tabular data for cross-domain text-to-sql semantic parsing, arXiv: 2012.12627, 2020, Scholak et al.; T5-3B as described in Picard: Parsing incrementally for constrained auto-regressive decoding from language models, arXiv: 2109.05093, 2021, Hwang et al.; A comprehensive exploration on wikisql SQLova, as described in "with table-aware word contextualization," arXiv: 1902.01069, 2019; X-SQL, as described in He et al.'s "X-sql: reinforce schema representation with context," arXiv: 1908.08113, 2019; IE-SQL, as described in Ma et al.'s "Mention extraction and linking for sql query generation," arXiv: 2012.10074, 2020; and Guo and Gao, "Content enhanced bert-based text-to-sql generation," arXiv: 1910.We compared it with NL2SQL (described in 07179, 2019), HydraNet (described in Lyu et al., Hybrid ranking network for text-to-sql, arXiv: 2008.04759, 2020), and TAPEX (described in Liu et al., Tapex: Table pre-training via learning a neural sql executor, arXiv: 2107.07653, 2021).

[0081] The datasets used for comparison include GRAILQA, as described in Gu et al., Beyond iid: three levels of generalization for question answering on knowledge bases, In Proceedings of the Web Conference, pages 3477-3488, 2021; Spider, as described in Yu et al., Spider: A large-scale human labeled dataset for complex and cross-domain semantic parsing and text-to-sql task, arXiv: 1809.08887, 2018; and the test and dev datasets of WikiSQL, as described in Zhong et al., Seq2sql: Generating structured queries from natural language using reinforcement learning, arXiv: 1709.00103, 2017.

[0082] In the experiment,<tb_cl_vl> To construct category primitives, related cell values ​​relevant to the question were found. Given a question and a database, string matching was computed between a phrase of any length in the question and a list of cell values ​​under each column of all tables. A fuzzy matching algorithm was used to match the question with possible cell values ​​mentioned in the database. Numerical values ​​in the question were also detected, and all column names were formed with those values ​​as primitives. Because column names in the WikiSQL dataset are ambiguous, such as "No", "Pick#", and "Rank", cell values ​​were used to supplement the meaning of the column names. Rows were identified using the cell values ​​to be matched, and column names were matched with the cell values ​​in the same row. The primitive blanker was started using BERT-base-uncased. 48 negative candidates were sampled for each primitive category. The ranker was trained over 10 epochs using a learning rate of 1e-5 and a batch size of 8. Bootstrapped hard negative sampling was performed every two epochs. Ground truth entity linking was used to enumerate the training candidates. The generator was trained on the Spider dataset using T5-base and T5-3B. The top 15<tb_cl> The category primitives are used, and the top 5<tb_cl_vl> The category primitives were returned by the ranker, and the T5-base model was fine-tuned over 200 epochs using a learning rate of 5e-5 and a batch size of 64. For the T5-3B model, it was run on 16 A100 GPUs for 100 epochs using a batch size of 1024. Also, T5-base and T5-large were used for the WikiSQL dataset, with the top 5 as input to the generator.<tb_cl> Category primitives and the top 3<tb_cl_vl> Category primitives were used. T5-base / large was fine-tuned over 20 epochs using a learning rate of 3e-5 and a batch size of 16.

[0083] Figures 11-12 illustrate perfect match (EM) and F1 scores for GRAILQA test / development splits, comparing an alternative method to an embodiment of the method described herein (Uni-Parse). The reported models are based on a BERT-based model for the ranker and a T5-based model for the generator. The best results in development are shown in bold, and test results that are better than development are underlined. As shown in the figures, the tested embodiment (Uni-Parser) generally outperformed conventional methods. Compared to methods that enumerate logical forms such as Bert Ranking and RNG-KBQA, the approach described herein achieves better performance in constructive inference ability and zero-shot settings. In particular, for F1 in the constructive inference ability setting, it showed a 3.1% improvement over the baseline. This matches the expectation that the generator learns the construction of primitives.

[0084] Figures 13 and 14 summarize the results for the Spider and WikiSQL datasets, respectively. For the Spider dataset, Uni-Parser achieves competitive performance over all baseline models. Compared to generative models that use the entire DB table schema as input, such as BRIDGE and UnifiedSKG, Uni-Parser achieves a 3% improvement, suggesting the advantages of this method. Compared to other T5-3B models, Uni-Parser achieves comparable performance with fewer training epochs. The upper block of Figure 14 shows a comparison between small pre-trained models, and the lower block shows a comparison between large pre-trained models.

[0085] Figure 15 illustrates the effect of the hard negative policy on ranking for WebQSP and WikiSQL datasets. "No CG" means that primitives are not distinguished by category. "No CG" shows lower performance than the setting that uses categories in the input. By comparing the settings with and without hard negative (second two columns from the right), we demonstrate that the proposed hard negative sampling helps rankers better determine positive primitives from negative ones.

[0086] Figure 16 provides an illustrative comparison of the output of the semantic analysis framework described in the embodiments herein with that of another semantic analyzer. The top-5 ranked logical forms using RNG-KBQA contain a lot of redundant information between them and none are equivalent to an ideal logical formula. In contrast, the top-5 ranked formulas of the Uni-Parser method find the correct first-hop and second-hop primitives and generate a correct logical form. Note that even if a suitable primitive is not ranked top-1, the generator has the ability to correctly select it.

[0087] This description and accompanying drawings illustrating aspects, embodiments, implementations, or applications of the invention should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational modifications may be made without departing from the spirit and scope of this description and the claims. In some examples, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure the embodiments of this disclosure. Similar figures in two or more figures represent the same or similar elements.

[0088] This description includes specific details that describe several embodiments that are not inconsistent with the Disclosure. Numerous details are included to provide a complete understanding of the embodiments. It will be apparent to those skilled in the art that some embodiments can be carried out without some or all of these specific details. The specific embodiments disclosed herein are illustrative but not limiting. Those skilled in the art will recognize other elements that are not specifically described herein but are within the scope and spirit of the Disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in relation to one embodiment may be incorporated into other embodiments unless they are not otherwise specifically described or if the one or more features render one embodiment nonfunctional.

[0089] While exemplary embodiments have been shown and described, a wide range of modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some examples, some features of the embodiments may be adopted without corresponding use of other features. Those skilled in the art will recognize many variations, substitutions, and modifications. Therefore, the scope of the invention should be limited only by the following claims, which should be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.< / op> < / op> < / firsthop> < / op>

Claims

1. A method for generating a logical format executable on a database or knowledge base in response to a natural language query, Receiving the aforementioned natural language question via the communication interface, Identifying a first entity from the natural language question via a named entity recognition procedure, The enumeration tool enumerates a plurality of primitives that represent entities or entity relationships in the database or knowledge base based on the first entity, The ranker module ranks the multiple primitives based on their respective relevance to the natural language questions, Based on the aforementioned ranking, a subset of the higher-ranked primitives is selected, The generator generates the logical form executable on the database or knowledge base based on the natural language question and the subset of the plurality of primitives, The logical format is transmitted to a database system or knowledge base system via the aforementioned communication interface. A method comprising receiving, via the communication interface, a query result in response to the natural language question based on the logical format.

2. The method according to claim 1, further comprising identifying the starting knowledge base entity based on matching the starting knowledge base entity with the first entity from the natural language question, wherein the starting knowledge base entity is used as a starting point for enumeration of the plurality of primitives.

3. The method according to claim 2, wherein the plurality of primitives include a knowledge base primitive located within a predetermined number of hops from the starting knowledge base entity.

4. The aforementioned plurality of primitives are, The column names of the tables in the aforementioned database, The method according to claim 1, comprising: the cell value in the database identified by matching the cell value with the first entity from the natural language question.

5. The method according to claim 3, further comprising providing the ranker module with the plurality of primitives, each of which has a natural language question and a special token indicating a first hop or a second hop concatenated to it by the enumeration.

6. The method according to claim 1, wherein the ranker module is trained to minimize contrast loss, and the contrast loss is based on positive primitive samples and negative primitive samples extracted from a ground truth logic form.

7. The method according to claim 6, wherein the negative primitive sample is selected from the primitives connected in the knowledge base to a first hop entity extracted from the ground truth logic form.

8. The method according to claim 6, wherein the negative primitive sample is selected by selecting cell values ​​in the same table and column as the cell values ​​extracted from the ground truth logical form.

9. A system for generating a logical format executable on a database or knowledge base in response to a natural language query, The database or knowledge base and a memory for storing multiple processor-executable instructions, A communication interface for receiving the aforementioned natural language question, The system includes one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory and perform an operation, wherein the operation is Identifying a first entity from the natural language question via a named entity recognition procedure, The enumeration tool enumerates a plurality of primitives that represent entities or entity relationships in the database or knowledge base based on the first entity, The ranker module ranks the multiple primitives based on their respective relevance to the natural language questions, Based on the aforementioned ranking, a subset of the higher-ranked primitives is selected, The generator generates the logical form executable on the database or knowledge base based on the natural language question and the subset of the plurality of primitives, The logical format is transmitted to a database system or knowledge base system via the aforementioned communication interface. A system comprising receiving query results in response to a natural language question based on the logical format via the communication interface.

10. The aforementioned operation is, The system according to claim 9, further comprising identifying the starting knowledge base entity based on matching the starting knowledge base entity with the first entity from the natural language question, wherein the starting knowledge base entity is used as a starting point for enumeration of the plurality of primitives.

11. The system according to claim 10, wherein the plurality of primitives include a knowledge base primitive located within a predetermined number of hops from the starting knowledge base entity.

12. The aforementioned plurality of primitives are, The column names of the tables in the aforementioned database, The system according to claim 9, comprising: the cell value in the database identified by matching the cell value with the first entity from the natural language query.

13. The aforementioned operation is, The system according to claim 11, further comprising providing the ranker module with the plurality of primitives, each of which has a natural language question and a special token indicating a first hop or a second hop concatenated to it by the enumerator.

14. The system according to claim 9, wherein the ranker module is trained to minimize contrast loss, and the contrast loss is based on positive primitive samples and negative primitive samples extracted from a ground truth logic form.

15. The aforementioned negative primitive sample is Select from the primitives connected within the knowledge base to the first hop entity extracted from the ground truth logical form, or The system according to claim 14, which is selected by at least one of selecting cell values ​​in the same table and column as the cell values ​​extracted from the Grand Truth logical form.

Images