System and method for semantic analysis using primitive level enumeration

Abstract

The embodiments described herein provide a semantic analysis framework that can be called Uni-Parser. The Uni-Parser framework may be applied to question answering in both a knowledge base and a database. The three main stages of the Uni-Parser framework are enumeration, ranking, and generation. In the enumeration stage, primitives are enumerated based on matching the question to a data structure. After enumerating the primitives, the Uni-Parser framework may rank the primitives using a trained ranker model. Then, the top-ranked primitives may be used as input to a generator, which is a sequence-to-sequence model that creates a logical form.

Description

Technical Field

[0001] This disclosure claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 63 / 355,438, filed Jun. 24, 2022, and to U.S. Non-Provisional Application No. 18 / 059,691, filed Nov. 29, 2022, which are hereby expressly incorporated by reference in their entireties.

[0002] Embodiments generally relate to natural language processing and machine learning systems, and more specifically, to systems and methods for semantic analysis using primitive-level enumeration for question answering.

Background Art

[0003] In database operations such as retrieving results in response to a search query, often a particular form of command is required. To write commands in a particular logical form, a user needs to acquire a high-level database language. Machine learning systems are widely used to convert natural language questions into queries in a format used by a database or knowledge base. In this way, a user may enter a natural language query such as "what is the month having the highest revenue in the past five years?". Parsing natural language questions into an executable logical form is a useful and interpretable way to perform question answering on structured data. Existing approaches enumerate executable logical forms, and when dealing with complex questions having multi-hop / multi-table relationships, the number of enumerated logical forms increases exponentially. Further, existing approaches rely on different model structures for different data modalities (e.g., knowledge base and database) rather than a unified adaptable structure. Thus, there is a need for improved systems and methods for semantic analysis.

Brief Description of the Drawings

[0004]

Figure 1

[0005]

Figure 2

[0006]

Figure 3

[0007]

Figure 4

[0008]

Figure 5

[0009]

Figure 6

[0010]

Figure 7

[0011]

Figure 8

[0012]

Figure 9

[0013]

Figure 10

[0014]

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

[0015]

Figure 16A

Figure 16B

[0016] The embodiments of the present disclosure and their advantages are best understood by referring to the following detailed description. It should be understood that like reference numerals are used to identify like elements illustrated in one or more of the drawings, and that the drawings are provided for the purpose of exemplifying the embodiments of the disclosure and not for the purpose of limiting the same.

DETAILED DESCRIPTION OF THE INVENTION

[0017] As used herein, the term "network" may include any hardware or software-based framework that includes any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented therein or implemented therewith.

[0018] As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, the module may be implemented on one or more neural networks.

[0019] Analyzing natural language questions to generate an executable logical form that can be executed on a database or knowledge base is a useful and interpretable way to perform question answering on structured data. For example, a natural language question such as "How many students are enrolled in Statistics?" can be answered if a database containing student data is provided. A semantic parser can generate a logical form that can be executed by the database to provide an answer. For example, SELECT count(*) FROM Courses JOIN Course_Attendance ON Courses.Courseld = Course_Attendance.CourseId WHERE Courses.CourseName = “Statistics”. Existing approaches for semantic parsing enumerate executable logical forms. For example, when data is structured as a knowledge base, entities may be interconnected by edges representing relationships. In existing approaches, starting from an entity, logical forms that include relationships of two "hops" within the structure are enumerated. Given M connections to entities one hop away from the starting entity and N connections from each of these entities, M*N logical forms are produced. For sufficiently complex data structures, this results in inefficient resource use and performance degradation. Furthermore, existing approaches do not provide a unified structure that is applicable to both knowledge bases and databases.

[0020] In view of the need for an improved system and method for semantic analysis, the embodiments described herein provide a semantic analysis framework (referred to as "Uni-Parser") that can be applied to question answering in both knowledge bases and databases. The Uni-Parser framework may generate one or more logical forms that can be executed on a data structure in response to a natural language input through enumeration, ranking, and generation. In the enumeration stage, primitives are enumerated based on matching a natural language question to a data structure (e.g., either a database or a knowledge base). For example, primitives in a database may be column names of a particular table, or values of a particular cell in the table (or an operation that provides cell values relevant as output). In a knowledge base, primitives may be represented by nodes in a graph. This is in contrast to a logical form composed of primitives and corresponding operations on the primitives, for example, in a database, it may be SELECT count(*) FROM Courses JOIN Course_Attendance ON Courses.CourseID = Course_Attendance.CourseID WHERE Courses.CourseName = “Statistics”. This logical form includes several primitives corresponding to column names of a particular table such as "Courses.CourseID" and cell values such as "Statistics". Instead of enumerating all possible logical forms using all different variations and / or combinations of primitives, Uni-Parser enumerates the primitives themselves. In this way, instead of enumerating M*N logical forms (where M is the number of first-hop entities and N is the number of second-hop entities from a particular entity in a natural language question within the knowledge base), Uni-Parser may enumerate M+N primitives.

[0021] After enumerating the primitives, the Uni-Parser framework may rank the primitives using a trained ranker model. The top-ranked primitives may then be used as input to a generator, which is a learned sequence-to-sequence model. The sequence-to-sequence model may then produce an output sequence in logical form in response to the input sequence of top-ranked primitives. Details of the enumeration, ranking, and generation stages are described in more detail below with respect to FIGS. 1-7.

[0022] The embodiments described herein provide several advantages. For example, the same semantic parsing framework may be used for both database and knowledge base question answering. By enumerating primitives instead of logical forms, the search space is significantly reduced, making candidate generation and ranking more efficient and scalable. Further, the composition of logical forms from primitives and operations is deferred to the generation stage. This leads to a more generalizable model that can operate on complex logical forms and generalize to questions with unseen schemas.

[0023] Figure 1 is a schematic diagram illustrating a semantic analysis framework according to some embodiments. Framework 100 includes an enumerator 104 operably connected to a ranker 106, which is operably connected to a generator 108. A natural language question 102 (e.g., "How many students are enrolled in Statistics?") may be an input to the enumerator 104. Given the question 102 and a known data structure (e.g., a knowledge base including a plurality of entities interconnected by edges representing relationships between entities, or a database including one or more tables, each table having respective columns and rows storing values, etc.), the enumerator 104 may enumerate some primitives. For example, for a given entity indicated by the question 102, the first-hop and second-hop entities from a given entity in the knowledge base may be enumerated as primitives. The enumeration of primitives is described in more detail with respect to Figure 2 for the knowledge base and Figure 3 for the database.

[0024] The enumerated primitives may be passed to a ranker 106 that ranks the primitives based on the question 102. The ranker 106 may be a trained model trained on a similar or identical data structure. The ranker 106 is described in more detail with respect to Figure 4 for the knowledge base and Figure 5 for the database.

[0025] The primitives ranked at the top may be passed to the generator 108. Based on the question 1-2 and the ranked primitives, the generator 108 may create a logical form 110. The logical form 110 may be executed on the data structure to provide an answer to the question 102. The generator 108 may be a trained model trained on a similar or identical data structure. The generator 108 is described in more detail with respect to Figure 6 for the knowledge base and Figure 7 for the database.

[0026] Figure 2 is a schematic diagram illustrating an enumeration of primitives for a knowledge base according to some embodiments. Based on the provided question 202, an enumerator (e.g., enumerator 104 of FIG. 1) may enumerate primitives from the knowledge base 204. The enumerator may start, for example, by detecting entities within the question 202 by using, for example, a named entity extraction (NER) algorithm. For example, named entities such as "currency", "New York", "Central Park", "construction", etc. may be identified within the question 202. In some embodiments, the NER algorithm identifies entities within the question 202 based on belonging to a predetermined class of entities (e.g., location, person, etc.).

[0027] The entities identified within the question 202 may then be matched (or fuzzy matched) with entities within the knowledge base 204. For example, "Central Park" may be selected as a starting entity based on matching this entity with the phrase "central park" within the question 202.

[0028] In some embodiments, only one entity is selected from the knowledge base 204, while in other embodiments, two or more entities may be selected. The matched entity(ies) may be used as a starting point for the enumeration. To reduce the problem of entity ambiguity, a ranker model may be used to select entity candidates based on the similarity between the question and the one-hop in-out relationships of the entities.

[0029] After selecting one or more starting entities, the enumerator may enumerate all primitives within two hops of the starting entities on the knowledge base. As reflected in the knowledge base 204, entities one hop away from "Central Park" shown and enumerated in the list 206 may include items such as "architecture.construction_cost central_park" and "architecture.landscape_project central_park". At two hops from "central park", additional entities such as "travel.get_destination" and "money_unit.currency" may be included. As shown in the list 206, the enumeration is done at the level of primitives, i.e., the basic units of the knowledge base, rather than queries composed of primitives and operations on primitives.

[0030] Figure 3 is a schematic diagram illustrating the enumeration of primitives for a database according to some embodiments. Based on the provided question 302, an enumerator (e.g., the enumerator 104 of FIG. 1) may enumerate primitives from the database 304. In the database, primitives for the input question may include column names or cell values. For example, in a database including one or more data tables, all column names of all data tables may be enumerated first, and only related cell values within one or more data tables may then be enumerated based on the input question. Since the number of column names is generally manageable even in a very large database, all column names of all tables in the database may be enumerated. For example, as shown in the enumeration list 310, the column names (associated with their tables) are enumerated including Students.StudentId, Students.StudentName (both from table 306), and although columns from the table 308 that would be included are not shown, column names from the table 310 are also included as exemplified by the exemplary Courses.CourseName entry.

[0031] For the enumeration of individual cells, the amount of individual cells is too large to be counted efficiently. Instead of enumerating all cells, the enumerator may detect the entities within question 302, for example, by using a named entity extraction algorithm. For example, "students", "course", "statistics" may be detected from question 302. Next, the entities 302 within the question may be matched (or fuzzy matched) with the cell values within database 304. For example, as illustrated, question 302 includes the word "statistics" that matches the cell value "Statistics" within the CourseName column of the table 308 of all courses. Thus, this maintains the relationship with table 308 and the CourseName column as "Courses.CourseName" <op>It is listed in the enumeration list 310 as "Statistics". In some embodiments, if the question 302 contains a numerical value and string matching cannot detect a match, the value is paired with all column names in the database.

[0032] FIG. 4 is a schematic diagram illustrating the ranking of primitives for a knowledge base according to some embodiments. The enumeration lists 402 and 404 are lists of first-hop and second-hop primitives as enumerated by an enumerator (e.g., enumerator 104) as described with reference to FIGS. 1-2. In some embodiments, the same ranker 106 model may be used for both classes of primitives (e.g., both first-hop and second-hop).

[0033] In some embodiments, as illustrated in FIG. 4, the enumeration list provided by the enumerator may be concatenated by the question, a special token indicating the first / second hop, and the primitive itself (by any of the enumerator, ranker, or another unit). For example, the enumeration list 404 starts with the text of the question 202 in FIG. 2 and a special first-hop token " <firsthop>It includes items having primitives such as ", and "architecture.construction_cost central_park". Table 402 includes question 202, the second hop token, and similar items with primitives concatenated thereafter. These are exemplary concatenation patterns, and other concatenation patterns may be used. The listed list is provided to ranker 106.

[0034] (As described in FIG. 1) Ranker 106 may be trained to exclude unrelated primitives by measuring the similarity between a question and an enumerated primitive candidate (e.g., as shown in lists 402, 404). In some embodiments, ranker 106 utilizes a cross-encoder architecture. Given a question X and a primitive p having a category token p c ranker 106 may use a BERT-base encoder that takes the concatenation of their vector representations as input and outputs a logit representing the similarity between the primitive and the question, i.e., [Number] where [Number] represents the concatenation operation. ψθ represents the [CLS] representation of the concatenated input after BERT embedding, and FNN is a projection layer that reduces the representation to a scalar similarity score. p c is a special token for distinguishing the category of the primitive (in the knowledge base, [Number] whereas in the database, [Number] ). The ranker is optimized to minimize the contrastive loss. That is, [Number] where p + is a positive primitive extracted from the grand-to-truth logic form, and P - is a set of negative primitives from the same category p c .

[0035] When training the ranker 106, negative sampling may be performed to pair negative primitive samples with positive samples. In some embodiments, random sampling is used to select negative samples. In other embodiments, the sampling strategy samples hard negative candidates to train the ranker 106. In the case of a knowledge base, the hard negative candidates for the second hop may be sampled from the primitives connected to the grand-to-truth first hop. Additionally, a bootstrap-type negative sampling strategy may be utilized. That is, the model may be recursively trained using the false positive candidates generated from the last training epoch.

[0036] During inference, ranker 106 may generate output values associated with each of the enumerated primitives. Based on these values, the primitives may be ranked. In some embodiments, the first-hop / second-hop primitives are ranked separately (or, in the case of a database, the column name / cell value). This is illustrated for the knowledge base as the ranked lists 406 and 408 of FIG. 4. In some embodiments, the top k primitives of each category may be selected and passed to a generator (e.g., generator 108). For example, k may be set to 5. In the knowledge base, the top-ranked first-hop primitives and second-hop primitives may be formed into two-hop paths by combining one first-hop primitive with each of the second-hop primitives. However, to provide valid primitive candidates to generator 108, they may be further filtered to remove second-hop primitives that cannot be reached from any of the first-hop primitives.

[0037] FIG. 5 is a schematic diagram illustrating the ranking of primitives for a database, according to some embodiments. Enumeration lists 502 and 504 are lists of column names and cell value primitives as enumerated by an enumerator (e.g., enumerator 104) as described with reference to FIGS. 1 and 3.

[0038] In some embodiments, as illustrated in FIG. 5, the enumeration list provided by the enumerator may include a question, a special token indicating whether the primitive is a column name or a cell value, and what the primitive itself is concatenated by (either by the enumerator, the ranker, or another unit). For example, enumeration list 502 may start with the text of question 302 of FIG. 3, the special cell value token "<tb_cl_vl>", and the primitive "Courses.CourseName" <op>It includes a single item with "Statistics". Table 504 includes question 202, column name tokens, and similar items with primitives concatenated thereafter. These are exemplary concatenation patterns, and other concatenation patterns may be used. The enumerated list is provided to ranker 106.

[0039] The training and utilization of ranker 106 may be substantially the same as described with reference to the database in FIG. 4. Instead of the enumerated knowledge base primitives, the enumerated database primitives are applied. When selecting hard negatives to train ranker 106 in the database, for the primitives of the <tb_cl> category, the training method may treat those having the same table name as the grand truth but different column names as hard negatives. Also, for the <tb_cl_vl> category, the training method may treat candidates having the same table and column names as the grand truth but different cell values as hard negatives.

[0040] The ranked lists 506 and 508 provide a ranking of database primitives in a similar manner as described for the ranked lists 406 and 408 of FIG. 4 with respect to the knowledge base.

[0041] FIG. 6 is a schematic diagram illustrating the generation of a logical form for a knowledge base according to some embodiments. In the knowledge base, generator 108 receives input 602 which is the primitives ranked highest from ranker 106 (concatenated with questions and special tokens). In some embodiments, the primitive order may be randomized such that generator 108 has to learn to use semantic meaning instead of misusing positional information to select the primitives ranked highest.

[0042] The generator 108 may learn to generate logical forms (queries composed of primitives and executable on corresponding data structures) by understanding the meaning of its elements (primitives and operations) and composing them. The generator 108 may be trained to generate an output logical form optimized by mutual entropy loss for each token. In inference, beam search may be used to decode the top-k target logical forms in an autoregressive manner.

[0043] An exemplary logical form output 604 is illustrated. As shown, the output 604 is (JOIN money_unit.currency (JOIN architecture.construction_cost central_park)). Note that the logical form is composed of primitives that include knowledge base entities and operations (e.g., JOIN) performed on those entities. This logical form is used as a query against the knowledge base to provide an answer to a question (e.g., question 102 or 202).

[0044] FIG. 7 is a schematic diagram illustrating the generation of logical forms for a database according to some embodiments. In general, the training and utilization of the generator 108 for the database is the same as that described for the knowledge base of FIG. 6.

[0045] In the database, the generator 108 includes primitives ranked top from the ranker 106, [Question; |table_name1| column_name1, column_name2 <op>Receive the formatted input 702 in the form of [[value...|table_name|2...]]. In the database, their order is naturally changed from the ranked order by formatting that organizes column names and cell values according to the tables to which they belong.

[0046] As an example, the output may be generated as output 704 such as SELECT count(*) FROM Courses JOIN Course_Attendance ON Courses.CourseId = Course_Attendance.CourseId WHERE Courses.CourseName = ”Statistics”. Similar to the logical form described in FIG. 6, this logical form output 704 may be used as a query to the database to provide an answer to a question (e.g., question 102 or 302).

[0047] FIG. 8 is a simplified diagram illustrating a computing device implementing the Uni-Parser framework described in FIGS. 1-7 according to one embodiment described herein. As shown in FIG. 8, the computing device 800 includes a processor 810 coupled to a memory 820. The operation of the computing device 800 is controlled by the processor 810. Also, although the computing device 800 is shown as having only one processor 810, the processor 810 can represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), etc. within the computing device 800. The computing device 800 may be implemented as a stand-alone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0048] Memory 820 may be used to store software executed by computing device 800 and / or one or more data structures used during operation of computing device 800. Memory 820 may include one or more types of machine-readable media. Some common forms of machine-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0049] Processor 810 and / or memory 820 may be arranged in any suitable physical configuration. In some embodiments, processor 810 and / or memory 820 may be implemented on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), etc. In some embodiments, processor 810 and / or memory 820 may include distributed, virtualized, and / or containerized computing resources. Without contradiction to such embodiments, processor 810 and / or memory 820 may be located in one or more data centers and / or cloud computing facilities.

[0050] In some examples, the memory 820 may include a non-transitory tangible machine-readable medium that includes executable code that, when operated on by one or more processors (e.g., processor 810), may cause the one or more processors to execute the methods described in more detail herein. For example, as illustrated, the memory 820 may include instructions for a Uni-Parser module 830 that may be used to implement and / or emulate systems and models and / or implement any of the methods further described herein. The Uni-Parser module 830 may receive an input 840, such as input training data (e.g., questions and corresponding logical forms), via a data interface 815 and may generate an output 850 that may be a logical form based on a question for a given data structure.

[0051] The data interface 815 may include a communication interface, a user interface (such as a voice input interface, a graphical user interface, etc.). For example, the computing device 800 may receive an input 840 (such as a training data set) from a networked database via the communication interface. Alternatively, the computing device 800 may receive an input 840, such as questions and / or data structures, from a user via the user interface.

[0052] In some embodiments, the Uni-Parser module 830 is configured to create a logical form for answering questions using the provided data structure. The Uni-Parser module 830 may further include an enumerator sub-module 831 (e.g., similar to the enumerator 104 in FIG. 1), a ranker sub-module 832 (e.g., similar to the ranker 106 in FIG. 1), and a generator sub-module 833 (e.g., similar to the generator 108 in FIG. 1). In one embodiment, the Uni-Parser module 830 and its sub-modules 831-833 may be implemented by hardware, software, and / or a combination thereof.

[0053] Some examples of computing devices, such as computing device 800, may include a non-transitory tangible machine-readable medium that includes executable code that, when operated on by one or more processors (e.g., processor 810), may cause the one or more processors to execute a process of a method. Some common forms of machine-readable media that may include a process of a method are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tapes, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read by a processor or computer.

[0054] FIG. 9 is a simplified block diagram of a networked system suitable for implementing the Uni-Parser framework described in FIGS. 1-7 and other embodiments described herein. In one embodiment, block diagram 900 includes a user device 910 that can be operated by a user 940, database vendor servers 945, 970, and 980, a server 930, and other forms of devices, servers, and / or software components that operate to execute various methodologies according to the described embodiments. Exemplary devices and servers may include devices that may be similar to the computing device 800 described in FIG. 8, stand-alone, and enterprise-class servers, and operate an OS such as a MICROSOFT® OS, UNIX® OS, LINUX® OS, or other suitable device and / or server-based OS. The devices and / or servers illustrated in FIG. 9 may be deployed in other ways, and the operations and / or services provided by such devices and / or servers may be combined or separated for a given embodiment, and it can be understood that they may be performed by a greater or lesser number of devices and / or servers. One or more devices and / or servers may be operated and / or maintained by the same or different entities.

[0055] The user device 910, database vendor servers 945, 970, and 980, and server 930 can communicate with each other via a network 960. The user device 910 may be utilized by a user 940 (e.g., a driver, system administrator, etc.) to access various functions available to the user device 910, which may include processes and / or applications associated with the server 930 in order to receive output data anomaly reports.

[0056] The user device 910, the database vendor server 945, and the server 930 each may include one or more processors, memory, and other suitable components for executing instructions such as program code and / or data stored on one or more computer-readable media to implement the various applications, data, and steps described herein. For example, such instructions may be stored on one or more computer-readable media such as internal and / or external memories or data storage devices of the various components of the system 900 and / or may be accessible via the network 960.

[0057] The user device 910 may be implemented as a communication device that utilizes suitable hardware and software configured for wired and / or wireless communication with the database vendor server 945 and / or the server 930. For example, in one embodiment, the user device 910 may be an autonomous vehicle, a personal computer (PC), a smartphone, a laptop / tablet computer, a wristwatch having suitable computer hardware resources, glasses having suitable computer hardware (e.g., GOOGLE GLASS (registered trademark)), other types of wearable computing devices, implantable communication devices, and / or other types of computing devices capable of transmitting and / or receiving data such as an IPAD (registered trademark) from APPLE (registered trademark). Although only one communication device is shown, multiple communication devices may function similarly.

[0058] The user device 910 in FIG. 9 includes a user interface (UI) application 912 and / or other applications 916, which may correspond to applications having executable processes, procedures, and / or associated hardware. For example, the user device 910 may receive a message from the server 930 indicating a logical form or a direct answer to a question, and display the message via the UI application 912. In other embodiments, the user device 910 may include additional or different modules having dedicated hardware and / or software as needed.

[0059] In various embodiments, the user device 910 includes other applications 916 if it is desired to provide functionality to the user device 910 in a particular embodiment. For example, the other applications 916 may include a security application for implementing client-side security features, a programmatic client application for interfacing with an appropriate application programming interface (API) via the network 960, or other types of applications. The other applications 916 may also include communication applications such as email, text, voice, social networking, and IM applications that enable the user to send and receive emails, phone calls, texts, and other notifications via the network 960. For example, the other applications 916 may be an email or instant messaging application that receives query result messages from the server 930. The other applications 916 may include a device interface and other display modules that can receive input and / or output information. For example, the other applications 916 may include a software program for asset management executable by a processor, including a graphical user interface (GUI) configured to provide an interface for the user 940 to view answers to questions.

[0060] The user device 910 may further include a database 918 stored in the temporary and / or non-temporary memory of the user device 910. This database stores various applications and data and may be utilized during the execution of various modules of the user device 910. The database 918 may store a user profile related to the user 940, predictions previously viewed or saved by the user 940, historical data received from the server 930, etc. In some embodiments, the database 918 may be local to the user device 910. However, in other embodiments, the database 918 may be external to the user device 910 and may be accessible by the user device 910 including a cloud storage system and / or a database accessible via the network 960.

[0061] The user device 910 includes at least one network interface component 917 configured to communicate with the database vendor server 945 and / or the server 930. In various embodiments, the network interface component 917 may include various other types of wired and / or wireless network communication devices including a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device, and / or microwave, radio frequency, infrared, Bluetooth, and near field communication devices.

[0062] The database vendor server 945 may provide a training data set including questions and corresponding logical forms to the server 930 corresponding to the server hosting the database 919. The database 919 may be implemented by one or more relational databases, distributed databases, cloud databases, etc.

[0063] The database vendor server 945 includes at least one network interface component 926 adapted to communicate with the user device 910 and / or the server 930. In various embodiments, the network interface component 926 may include various other types of wired and / or wireless network communication devices including a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device, and / or microwave, radio frequency, infrared, Bluetooth, and near field communication devices. For example, in one implementation, the database vendor server 945 may transmit asset information from the database 919 to the server 930 via the network interface 926.

[0064] The server 930 may house the Uni-Parser module 830 and its sub-modules described in FIG. 1. In some implementations, the Uni-Parser module 830 may receive data from the database 919 in the database vendor server 945 via the network 960 and generate a logical form and / or a direct answer to a question. The generated logical form and / or answer may be transmitted to the user device 910 via the network 960 for review by the user 940.

[0065] The database 932 may be stored in the temporary and / or non-temporary memory of the server 930. In one implementation, the database 932 may store data obtained from the database vendor server 945. In one implementation, the database 932 may store parameters of the Uni-Parser module 830. In one implementation, the database 932 may store previously generated logical forms, and corresponding input feature vectors.

[0066] In some embodiments, database 932 may be local to server 930. However, in other embodiments, database 932 may be external to server 930 and accessible by server 930 including a cloud storage system and / or database accessible via network 960.

[0067] Server 930 includes at least one network interface component 933 adapted to communicate with user device 910 and / or database vendor servers 945, 970, or 980 via network 960. In various embodiments, network interface component 933 may include various other types of wired and / or wireless network communication devices including, for example, a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device, and / or a microwave, radio frequency (RF), infrared (IR) communication device.

[0068] Network 960 may be implemented as a single network or as a combination of multiple networks. For example, in various embodiments, network 960 may include the Internet or one or more intranets, landline networks, wireless networks, and / or other suitable types of networks. Thus, network 960 may correspond to a small-scale communication network such as a private or local area network accessible by various components of system 900, or a large-scale network such as a wide area network or the Internet.

[0069] The knowledge base and / or database used for queries using the primitive enumeration and the generated logical form may be stored in one of the exemplary network devices such as the database vendor server 945, server 930, or user device 910. Such a knowledge base / database may or may not be associated with the databases described above (i.e., databases 918, 919, and 932), or may or may not be incorporated.

[0070] FIG. 10 is an exemplary logical flow diagram illustrating a method 1000 of semantic analysis based on the Uni-Parse framework shown in FIGS. 1-9 according to some embodiments described herein. One or more of the processes of method 1000 may be implemented in the form of executable code stored in a non-transitory tangible machine-readable medium that can cause one or more processors to execute one or more of the processes when executed, at least in part, by the one or more processors. In some embodiments, method 1000 corresponds to the operation of the Uni-Parser module 830 (e.g., FIGS. 8-9) that performs semantic analysis.

[0071] As illustrated, method 1000 includes several enumerated steps, but aspects of method 1000 may include additional steps before, after, and in between the enumerated steps. In some aspects, one or more of the enumerated steps may be omitted or executed in a different order.

[0072] In step 1001, the system receives a natural language question (e.g., question 102, 202, or 302) via a communication interface. For example, the question may be input by the user into the user interface. The communication interface may be, for example, a computer network interface, and the natural language question is received at the server (e.g., server 930) from the user device (e.g., user device 910) via the network interface (e.g., network interface 933).

[0073] In step 1002, the system identifies a first entity from the natural language question via a named entity recognition (NER) procedure. For example, the first entity may be a node in a knowledge graph such as "Central Park" described in FIG. 2. In the database, the first entity may be a cell value such as "Statistics" as described in FIG. 3.

[0074] In step 1003, the system enumerates a plurality of primitives representing entities or entity relationships in the database or knowledge base based on the first entity (e.g., by enumerator 104). For example, in the database, the primitives may include table column names and cell values (including cell values that match the first entity described in step 1002). For example, in the knowledge base, the primitives may include knowledge base entities and entity relationships (including knowledge base entities that match the first entity described in step 1002). In some embodiments, the knowledge base entity that matches the first entity is the starting entity, from which other primitives are enumerated, and the other primitives are one or two hops away from the starting entity.

[0075] In step 1004, the system ranks a plurality of primitives (e.g., by ranker 106) based on their respective relevance to the natural language question. In some embodiments, a separate ranked list is generated for each class of primitives (e.g., first hop and second hop, or column name and cell value). The primitives enumerated in step 1004 may be provided to the ranker in a form concatenated with special tokens that identify the natural language question and / or the class of primitives (e.g., first hop, second hop, column name, or cell value). The ranker may be a model (e.g., a neural network model) trained to minimize a contrastive loss. The contrastive loss may be based on positive primitive samples extracted from the ground truth logical form and negative primitive samples. The method for selecting negative primitive samples may be random, or they may be selected as "hard" samples that can improve training. For example, in a knowledge base, the negative samples may be selected from primitives connected within the knowledge base to the first-hop entities extracted from the ground truth logical form. In a database, the negative samples may be selected by choosing cell values within the same table and column as the cell values extracted from the ground truth logical form.

[0076] In step 1005, the system selects a subset of the top-ranked primitives based on the ranking. This may be performed by ranker 106, generator 108, or another component of the system. The amount of primitives selected may be based on a predetermined value. If the ranked list includes multiple ranked lists (e.g., a "first hop" list and a "second hop" list), each list may be ranked individually and the top-ranked primitives from each list may be selected.

[0077] In step 1006, the system generates a logical form executable on a database or knowledge base based on a natural language question and a subset of a plurality of primitives (e.g., by generator 108). The executable logical form may include at least one primitive from the subset. The logical form may be, for example, logical form 604 or 704.

[0078] In step 1007, the system transmits the logical form to a database system or a knowledge base system via a communication interface. For example, the database system or the knowledge base system may be a server such as the database vendor server 945 described in FIG. 9, and the logical form is transmitted to the server so that the server can execute the logical form on the database (or knowledge base).

[0079] In step 1008, the system receives, via a communication interface, query results in response to a natural language question based on the logical form. The answer may be displayed to the user, for example, on the same user interface used to input the question.

[0080] Figures 11 to 15 provide charts illustrating the exemplary performance of different embodiments described herein. Bert Ranking as described in Gu et al., Beyond iid: three levels of generalization for question answering on knowledge bases, In Proceedings of the Web Conference, pages 3477-3488, 2021, ReTrack as described in Chen et al., Retrack: a flexible and efficient framework for knowledge base question answering, in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 325-336, 2021, UnifiedSKG as described in Xie et al., Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models, arXiv: 2201.05966, 2022, RNG-KBQA as described in Ye et al., Rng-kbqa: Generation augmented iterative ranking for knowledge base question answering, arXiv:2109.08678, 2021, Topic Units as described in Lan et al., Knowledge base question answering with topic units, 2019, Yih et al., Semantic parsing via staged query graph generation: Question answering with knowledge base, STAGG as described in Proceedings of the Joint Conference of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing of the AFNLP, 2015, QGG as described in Lan et al., Query graph generation for answering multi-hop complex questions from knowledge bases, Association for Computational Linguistics, 2020, CBR as described in Das et al., Case-based reasoning for natural language queries over knowledge bases, arXiv: 2104.08762, 2021, ArcaneQA as described in Gu and Su, Arcaneqa: Dynamic program induction and contextualized encoding for knowledge base question answering, arXiv: 2204.08109, 2022, Global-GNN as described in Bogin et al., Global reasoning over database structures for text-to-sql parsing, arXiv: 1908.11214, 2019, EditSQL as described in Zhang et al., Editing-based sql query generation for cross-domain context-dependent questions, arXiv: 1909.00786, 2019, Scholak et al., Picard: Parsing incrementally for constrained auto-regressive decoding from language models, arXiv: 2109.05093, T5-Base as described in 2021, Wang et al., Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers, arXiv: 1911.04942, RAT-SQL as described in 2019, Lin et al., Bridging textual and tabular data for cross-domain text-to-sql semantic parsing, arXiv: 2012.12627, BRIDGE as described in 2020, Scholak et al., Picard: Parsing incrementally for constrained auto-regressive decoding from language models, arXiv: 2109.05093, T5-3B as described in 2021, Hwang et al., A comprehensive exploration on wikisql with table-aware word contextualization, arXiv: 1902.01069, SQLova as described in 2019, He et al., X-sql: reinforce schema representation with context, arXiv: 1908.08113, X-SQL as described in 2019, Ma et al., Mention extraction and linking for sql query generation, arXiv: 2012.10074, IE-SQL as described in 2020, Guo and Gao, Content enhanced bert-based text-to-sql generation, arXiv: 1910.A comparison was made with NL2SQL as described in 07179, 2019, HydraNet as described in Lyu et al., Hybrid ranking network for text-to-sql, arXiv: 2008.04759, 2020, and TAPEX as described in Liu et al., Tapex: Table pre-training via learning a neural sql executor, arXiv: 2107.07653, 2021.

[0081] The datasets used for the comparison include the test and dev datasets of GRAILQA as described in Gu et al., Beyond iid: three levels of generalization for question answering on knowledge bases, In Proceedings of the Web Conference, pages 3477-3488, 2021, Spider as described in Yu et al., Spider: A large-scale human labeled dataset for complex and cross-domain semantic parsing and text-to-sql task, arXiv: 1809.08887, 2018, and WikiSQL as described in Zhong et al., Seq2sql: Generating structured queries from natural language using reinforcement learning, arXiv: 1709.00103, 2017.

[0082] In the experiment, to construct the primitives of the <tb_cl_vl> category, relevant cell values related to the question were found. Given the question and the database, string matching was calculated between phrases of any length of the question and the list of cell values under each column of all tables. A fuzzy matching algorithm was used to match the question with possible cell values mentioned in the database. Also, numerical values in the question were detected, and all column names with those values as primitives were formed. Since column names in the WikiSQL dataset are ambiguous like "No", "Pick#", "Rank", cell values were used to complement the meaning of column names. Rows were identified using the matching cell values, and column names were matched with cell values in the same row. The primitive ranker was initiated using BERT-base-uncased. 48 negative candidates were sampled for each primitive category. The ranker was trained for 10 epochs using a learning rate of 1e-5 and a batch size of 8. Bootstrap-type hard negative sampling was performed every two epochs. Grand-to-truth entity linking was used to enumerate training candidates. The generator was trained using T5-base and T5-3B on the Spider dataset. The top 15 primitives of the <tb_cl> category were used, and the top 5 primitives of the <tb_cl_vl> category were returned by the ranker, and the T5-base model was fine-tuned for 200 epochs using a learning rate of 5e-5 and a batch size of 64. For the T5-3B model, it was executed on 16 A100 GPUs for 100 epochs using a batch size of 1024. Also, for the WikiSQL dataset, T5-base and T5-large were used, and the top 5 primitives of the <tb_cl> category and the top 3 primitives of the <tb_cl_vl> category were used as inputs to the generator. T5-base / large was fine-tuned for 20 epochs using a learning rate of 3e-5 and a batch size of 16.

[0083] Figures 11 - 12 illustrate the exact match (EM) and F1 scores for the test / dev split of GRAILQA, comparing alternative methods to the embodiment (Uni-Parse) of the method described herein. The reported models are based on the BERT-base model for the ranker and T5-base for the generator. The best results in dev are in bold, and test results better than dev are underlined. As shown, the tested embodiment (Uni-Parser) was generally superior to prior methods. Compared to methods that enumerate logical forms such as Bert Ranking and RNG-KBQA, the approach described herein achieves better performance in compositional reasoning ability and zero-shot settings. In particular, for F1 in the compositional reasoning ability setting, it presented a 3.1% improvement over the baseline. This matches the expectation that the generator learns primitive compositions.

[0084] Figures 13 - 14 summarize the results for the Spider and WikiSQL datasets respectively. For the Spider dataset, Uni-Parser achieves competitive performance against all baseline models. Compared to generation models that use the entire DB table schema as input, such as BRIDGE and UnifiedSKG, Uni-Parser achieves a 3% improvement, suggesting the advantages of this method. Compared to other T5-3B models, Uni-Parser achieves comparable performance with fewer training epochs. The upper block in Figure 14 shows the comparison among small pre-trained models, and the lower block shows the comparison among large pre-trained models.

[0085] Figure 15 illustrates the effect of the hard negative strategy in ranking for the WebQSP and WikiSQL datasets. Without CG means that primitives are not distinguished by category. Without CG shows lower performance than the setting using categories in the input. By comparing the settings with and without hard negatives (the two rightmost columns), the proposed hard negative sampling is shown to help the ranker better determine positive primitives from negative primitives.

[0086] Figure 16 provides an exemplary comparison between the output of the semantic analysis framework described in the embodiments of this specification and another semantic parser. The logical forms ranked top-5 using RNG-KBQA contain a lot of redundant information among themselves and none of them is equal to the ideal logical formula. In contrast, the formulas ranked top-5 by the Uni-Parser method find the correct first-hop and second-hop primitives and generate the correct logical form. Note that the generator has the ability to correctly select a suitable primitive even if it is not ranked top-1.

[0087] This description and the accompanying drawings, which illustrate aspects, embodiments, implementations, or applications of the invention, should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure the embodiments of the present disclosure. Similar numerals in two or more figures represent the same or similar elements.

[0088] In this description, specific details are set forth that describe several embodiments that do not conflict with the present disclosure. To provide a complete understanding of the embodiments, numerous details are set forth. It will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are illustrative but not limiting. Those skilled in the art may recognize other elements that are within the scope and spirit of the present disclosure but not specifically described herein. Additionally, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments unless specifically described otherwise in connection with those other embodiments or the one or more features render one embodiment non-functional.

[0089] Exemplary embodiments have been shown and described, but a wide range of modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some instances, some features of the embodiments may be employed without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention should be limited only by the following claims, and it is appropriate that the claims be broadly construed in a manner that does not conflict with the scope of the embodiments disclosed herein.< / op> < / op> < / firsthop> < / op>

Claims

1. A method for generating a logical form executable on a database or a knowledge base for a natural language question, comprising: receiving the natural language question via a communication interface; identifying a first entity from the natural language question via a named entity extraction procedure; enumerating, by an enumerator, a plurality of primitives indicating entities or entity relationships within the database or the knowledge base based on the first entity; ranking, by a ranker module, the plurality of primitives based on their respective relevance to the natural language question; selecting, based on the ranking, a subset of the primitives ranked at the top; generating, by a generator, the logical form executable on the database or the knowledge base based on the natural language question and the subset of the plurality of primitives; transmitting, via the communication interface, the logical form to a database system or a knowledge base system; receiving, via the communication interface, a query result in response to the natural language question based on the logical form.

2. The method according to claim 1, further comprising identifying the start knowledge base entity based on matching the start knowledge base entity with the first entity from the natural language question.

3. The method according to claim 2, wherein the plurality of primitives include primitives of the knowledge base within a predetermined number of hops from the start knowledge base entity.

4. The plurality of primitives include: column names of tables in the database; and cell values of the database identified by matching cell values with the first entity from the natural language question.

5. The method according to claim 1, further comprising providing, by the enumerator, to the ranker module the plurality of primitives each concatenated with a special token identifying the natural language question and a class of the primitive.

6. The ranker module is trained to minimize a contrastive loss, and the contrastive loss is based on positive primitive samples extracted from a ground truth logical form and negative primitive samples, the method according to claim 1.

7. The method according to claim 6, wherein the negative primitive samples are selected from the primitives connected in the knowledge base to a first-hop entity extracted from the ground truth logical form.

8. The method according to claim 6, wherein the negative primitive samples are selected by selecting cell values within the same table and column as the cell values extracted from the ground truth logical form.

9. A system for generating a logical form executable on a database or a knowledge base for a natural language question, a memory storing the database or the knowledge base and a plurality of processor-executable instructions; a communication interface for receiving the natural language question; one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations, the operations including: identifying a first entity from the natural language question via a named entity extraction procedure; enumerating, by an enumerator, a plurality of primitives indicating entities or entity relationships within the database or the knowledge base based on the first entity; ranking, by a ranker module, the plurality of primitives based on their respective relevance to the natural language question; selecting a subset of the primitives ranked at the top based on the ranking; generating, by a generator, the logical form executable on the database or the knowledge base based on the natural language question and the subset of the plurality of primitives; transmitting, via the communication interface, the logical form to a database system or a knowledge base system; receiving, via the communication interface, a query result in response to the natural language question based on the logical form, a system.

10. The operations include The system of claim 9, further comprising identifying the starting knowledge base entity based on matching the starting knowledge base entity with the first entity from the natural language question.

11. The system of claim 10, wherein the plurality of primitives includes the knowledge base primitives within a predetermined number of hops from the starting knowledge base entity.

12. The plurality of primitives includes column names of tables in the database, and cell values of the database identified by matching cell values with the first entity from the natural language question, the system of claim 9.

13. The operation further includes providing, by the enumerator, the plurality of primitives each concatenated with a special token identifying the natural language question and a class of primitives to the ranker module, the system of claim 9.

14. The ranker module is trained to minimize a contrastive loss, the contrastive loss being based on positive primitive samples and negative primitive samples extracted from a ground truth logical form, the system of claim 9.

15. The negative primitive samples are selected by selecting from the primitives connected in the knowledge base to a first-hop entity extracted from the ground truth logical form, or selected by performing at least one of selecting cell values in the same table and column as the cell values extracted from the ground truth logical form, the system of claim 14.

Images