System and method for semantic parsing using primitive level enumeration

Abstract

Embodiments described herein provide a semantic parsing framework that may be referred to as a unified parser. A unified parser framework can be applied to questions and answers on knowledge bases and databases. Three main stages of the unified parser framework are enumeration, sorting and generation. In the enumeration stage, primitives are enumerated according to matching of questions and data structures. After enumerating the primitives, the unified parser framework may sort the primitives using a trained sorter model. The primitives ranked in front may then be used as input to a generator that generates a learned sequence-to-sequence model in logical form.

Description

[0001] Inventors: Y. Liu, S. Yavuz, Zhou Yingbo, R. Meng

[0002] Cross-references

[0003] The present invention claims priority to U.S. non-provisional patent application No. 18 / 059,691 filed on November 29, 2022, which in turn claims priority to U.S. Provisional Application No. 63 / 355,438 filed on June 24, 2022 under 35 U.S.C 119, the entire contents of both of which are expressly incorporated herein by reference. Technical Field

[0004] Embodiments of the present invention relate generally to natural language processing and machine learning systems, and more particularly to systems and methods for semantic parsing using primitive-level enumeration for question answering. Background Art

[0005] Database operations (such as searching results in response to a search query) typically require commands in a specific form. Writing commands in a specific logical form requires the user to master a high-level database language. Machine learning systems have been widely used to convert natural language questions into queries in a format for a database or knowledge base. In this way, a user can enter a natural language query, such as "Which month has the highest income in the past five years?" Parsing natural language questions into executable logical forms is a useful and interpretable way to perform question and answer on structured data. Existing methods enumerate executable logical forms, where the number of enumerated logical forms grows exponentially when dealing with complex problems using multi-hop / multi-table relationships. In addition, existing methods rely on different model structures for different data modalities (e.g., knowledge bases and databases), rather than a unified adaptive structure. Therefore, improved semantic parsing systems and methods are needed. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a simplified diagram illustrating a semantic parsing framework according to some embodiments.

[0007] Figure 2 is a simplified diagram illustrating enumeration of primitives of a knowledge base according to some embodiments.

[0008] Figure 3 is a simplified diagram illustrating enumeration of primitives of a database according to some embodiments.

[0009] Figure 4 is a simplified diagram illustrating ordering primitives of a knowledge base according to some embodiments.

[0010] Figure 5 is a simplified diagram illustrating ordering primitives of a database according to some embodiments.

[0011] Figure 6 is a simplified diagram illustrating the logical form of generating a knowledge base according to some embodiments.

[0012] Figure 7 is a simplified diagram showing the logical form of generating a database according to some embodiments.

[0013] Figure 8 is a diagram showing an implementation according to an embodiment described herein. Figures 1 to 7 A simplified diagram of a computing device for the semantic parsing framework described in .

[0014] Fig. 9 is suitable for implementing Figures 1 to 7 A simplified block diagram of a networked system of the semantic parsing framework described in and other embodiments described herein.

[0015] Fig.10 is a diagram showing some embodiments described herein based on Figures 1 to 9 An example logical flow chart of the semantic parsing method of the framework shown.

[0016] Figures 11 to 15 Graphs are provided showing exemplary performance of various embodiments described herein.

[0017] FIG. 16 provides an exemplary comparison of the output of the semantic parsing framework described in the embodiments herein with another semantic parser.

[0018] Embodiments of the present invention and their advantages can be best understood by referring to the following detailed description. It should be understood that the same reference numerals are used to identify the same elements shown in one or more figures, where the illustrations are for the purpose of illustrating embodiments of the present invention and are not intended to limit the present invention. DETAILED DESCRIPTION

[0019] As used herein, the term "network" may include any hardware or software based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or in conjunction therewith.

[0020] As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, the module may be implemented on one or more neural networks.

[0021] Parsing natural language questions to generate executable logical forms that can be executed on a database or knowledge base is a useful and interpretable way to perform question answering on structured data. For example, given a database containing student data, a natural language question such as "How many students are enrolled in Statistics?" can be answered. A semantic parser can generate logical forms that can be executed by the database to provide an answer, for example, SELECT count(*)FROM Courses JOIN Course_Attendance ON Courses.Courseld=Course_Attendance.CourseId WHERE Courses.CourseName="Statistics". Existing semantic parsing methods enumerate executable logical forms. For example, when data is structured as a knowledge base, entities can be interconnected by edges representing relationships. Existing methods enumerate logical forms, starting with entities and including relationships that are two "hops" apart in the structure. Given M connections to entities that are one hop away from the starting entity, and N connections from each of those entities, this results in M*N logical forms. For sufficiently complex data structures, this can lead to inefficient use of resources and poor performance. Furthermore, existing approaches do not provide a unified structure that can be applied to both knowledge bases and databases.

[0022] In view of the need for improved systems and methods for semantic parsing, the embodiments described herein provide a semantic parsing framework (referred to as a "Uni-Parser") for question answering that can be applied to both knowledge bases and databases. The unified parser framework can generate one or more logical forms that can be executed on a data structure in response to natural language input, by enumeration, sorting, and generation. In the enumeration phase, primitives are enumerated based on matching natural language questions with data structures (e.g., databases or knowledge bases). For example, a primitive in a database may be a column name of a particular table, or a value of a particular cell in a table (or an operation that provides the relevant cell value as output), etc. In a knowledge base, a primitive can be represented by a node in a graph. This is in contrast to a logical form consisting of a primitive and a corresponding operation on the primitive. For a database, for example, the logical form may be: SELECT count(*)FROM Courses JOIN Course_AttendanceON Courses.CourselD=Course_Attendance. Course ID WHERE Courses.CourseName="Statistics". This logical form includes multiple primitives corresponding to column names and cell values ​​(such as Statistics) of a specific table (such as Courses.CourseID). Instead of enumerating all possible logical forms with all different primitive variations and / or combinations, the unified parser enumerates the primitives themselves. In this way, the unified parser can enumerate M+N primitives instead of M*N logical forms (where M is the number of first-hop entities and N is the number of second-hop entities from a specific entity in a natural language question in the knowledge base).

[0023] After enumerating the primitives, the unified parser framework can use the trained sorter model to sort the primitives. The top-ranked primitives can then be used as input to a generator, which is a learned sequence-to-sequence model. The sequence-to-sequence model can then produce an output sequence of logical form in response to an input sequence of top-ranked primitives. Figures 1 to 7 Describes the details of the enumeration, sorting, and generation phases in more detail.

[0024] Embodiment described herein provides many benefits.For example, the same semantic parsing framework can be used for question-answering of both database and knowledge base.By enumerating primitives instead of logical forms, the search space is greatly reduced, which makes candidate generation and sorting more efficient and scalable.In addition, the composition of logical forms from primitives and operations is postponed to the generation stage.This leads to a more general model, which can handle complex logical forms and generalize to problems involving patterns that have not been previewed.

[0025] Figure 1 is a simplified diagram illustrating a semantic parsing framework 100 according to some embodiments. The framework 100 includes an enumerator 104, which is operably connected to a sorter 106, and the sorter 106 is operably connected to a generator 108. A natural language question 102 (e.g., How many students are enrolled in Statistics?) can be an input to the enumerator 104. Given the question 102 and a known data structure (e.g., such as a knowledge base including multiple entities that are interconnected by edges representing relationships between the entities, or a database including one or more tables, each table having corresponding columns and rows for storing values), the enumerator 104 can enumerate multiple primitives. For example, for a given entity indicated in the question 102, the first-hop entity and the second-hop entity from the given entity in the knowledge base can be enumerated as primitives. Referring to Figure 2 (for knowledge base) and Figure 3 (For databases) An enumeration describing the primitive in more detail.

[0026] The enumerated primitives may be passed to a sorter 106, which sorts the primitives based on the question 102. The sorter 106 may be a learning model trained on a similar or identical data structure. Figure 4 (for knowledge base) and Figure 5 (For Databases) Sorter 106 is described in more detail.

[0027] The top ranked primitives may be passed to generator 108. Based on questions 1-2 and the ranked primitives, generator 108 may generate logical form 110. Logical form 110 may be executed on the data structure to provide an answer to question 102. Generator 108 may be a learning model trained on a similar or identical data structure. Figure 6 (for knowledge base) and Figure 7 (For Database) Describe Generator 108 in more detail.

[0028] Figure 2 is a simplified diagram illustrating an enumeration of primitives of a knowledge base according to some embodiments. Based on the provided question 202, an enumerator (e.g., Figure 1 The enumerator 104 of the question 202 can enumerate primitives from the knowledge base 204. The enumerator can begin by detecting entities in the question 202, such as by using a named entity recognition (NER) algorithm. For example, name entities such as "currency", "New York", "Central Park", "construction" can be identified in the question 202. In some embodiments, the NER algorithm identifies entities in the question 202 based on entities belonging to a predetermined category (e.g., location, person, etc.).

[0029] The entity identified in question 202 may then be matched (or fuzzy matched) with entities in knowledge base 204. For example, "Central Park" may be selected as a starting entity based on matching the entity with the word "Central Park" in question 202.

[0030] In some embodiments, only one entity is selected from the knowledge base 204, while in other embodiments two or more entities may be selected. The matching entity (or entities) may be used as a starting point for enumeration. To mitigate the entity disambiguation problem, a sorter model may be used to select entity candidates based on the similarity between the question and the entity's within / outside one-hop relationship.

[0031] After selecting one or more starting entities, the enumerator can enumerate all primitives within two hops of the starting entity on the knowledge base. As reflected in the knowledge base 204, the entities displayed in the list 206 one hop away from the enumerated "CentralPark" can include items such as "architecture.construction_cost central_park" and "architecture.landscape_project central_park". Additional entities such as "travel.get_destination" and "money_unit.currency" can be included at the second hop away from "centralpark". As shown in the list 206, the enumeration is done at the primitive level (the basic unit of the knowledge base), rather than on queries consisting of primitives and operations on primitives.

[0032] Figure 3 is a simplified diagram illustrating primitive enumeration of a database according to some embodiments. Based on the provided problem 302, an enumerator (e.g., Figure 1 The enumerator 104 of the embodiment of the present invention can enumerate primitives from the database 304. For a database, the primitives of any input question can include column names or cell values. For example, for a database including one or more data tables, all column names of all data tables can be enumerated first, and then only relevant cell values ​​within one or more data tables can be enumerated according to the input question. Since the number of column names can be managed as a whole even for very large databases, each column name of each table in the database can be enumerated. For example, as shown in the enumeration list 310, the column names (associated with their tables) are enumerated, including Student.StudentId, Students.StudentName (both from table 306), the columns from table 308 to be included are not shown, and the column names from table 310 are also included, as shown in the exemplary Courses.Course Name entry.

[0033] For enumeration of individual cells, the number of individual cells may be too large to be effectively enumerated. Instead of enumerating each cell, the enumerator can detect entities in question 302, for example, by using a named entity recognition algorithm, for example, "students", "course", "statistics" can be detected from question 302. The entities in question 302 can then be matched (or fuzzy matched) with cell values ​​in database 304. For example, as shown, question 302 contains the word "statistics", which matches the cell value "statistics" in the "CourseName" column of table 308 of all courses. Therefore, this is shown in enumeration list 310 as "Course.CourseName". <op>StatisticsCourses.Course Name <op>Statistics", which maintains its relationship with table 308 and the "CourseName" column. In some embodiments, if question 302 contains a numeric value and string matching fails to detect a match, the value is paired with all column names in the database.

[0034] Figure 4 is a simplified diagram illustrating the ordering of primitives in a knowledge base according to some embodiments. Enumerated lists 402 and 404 are as described in reference Figure 1 to Figure 2 The list of first-hop and second-hop primitives in question are enumerated by an enumerator (eg, enumerator 104). In some embodiments, the same sequencer 106 model may be used for both types of primitives (eg, both first-hop and second-hop).

[0035] In some embodiments, such as Figure 4 As shown, the enumeration list provided by the enumerator can be concatenated with the question, a special tag indicating the first / second hop, and the primitive itself (via the enumerator, the sequencer, or as another unit). For example, the enumeration list 404 includes the following Figure 2 Table 402 includes similar items concatenated with question 202, a special first hop marker "<|firsthop|>", and a primitive such as "architecture.construction_cost central-park". These are exemplary concatenation patterns, and other patterns may be used. The enumerated list is provided to sequencer 106.

[0036] Sequencer 106 (eg Figure 1 ) can be trained to filter out irrelevant primitives by measuring the similarity between the question and the enumerated primitive candidates (e.g., as shown in lists 402, 404). In some embodiments, the sorter 106 utilizes a cross encoder architecture. Given a question X and a class label p c For primitives p, the ranker 106 can use a BERT-based encoder that takes as input the concatenation of their vector representations and outputs a logit representing the similarity between the primitive and the question:

[0037]

[0038] Where ⊕ represents the concatenation operation. ψθ represents the [CLS] representation of the concatenated input after BERT embedding; FNN is a projection layer that simplifies the representation into a scalar similarity score. c is a special mark to distinguish the primitive category (in the knowledge base, p c ∈{<|firsthop|>,<|secondhop|>}, in the database, p c ∈{<|tb_cl|>, <|tb_cl_v1|>}). The ranker is optimized to minimize the contrastive loss:

[0039]

[0040] where p+ is a positive primitive extracted from a real logical form, and P- is a positive primitive from the same class p c The set of negative primitives.

[0041] In order to pair negative primitive samples with positive samples when training the sorter 106, negative sampling can be performed. In some embodiments, random sampling is used to select negative samples. In other embodiments, a sampling strategy samples hard negative candidates for training the sorter 106. In the case of a knowledge base, a hard negative candidate for the second hop can be sampled from a primitive connected to the true first hop. In addition, a bootstrapped negative sampling strategy can be utilized; that is, the model can be recursively trained using false positive candidates generated from the previous training epoch.

[0042] At inference, the sorter 106 may generate output values ​​associated with each enumerated primitive. Based on those values, the primitives may be sorted. In some embodiments, the first-hop / second-hop primitives are sorted individually (or column names / cell values ​​for databases). This is shown as Figure 4 406 and 408 of the knowledge base in the sorted list. In some embodiments, the top k primitives of each category can be selected to be passed to the generator (e.g., generator 108). For example, k can be set to 5. In the knowledge base, the top first-hop primitives and second-hop primitives can be formed into a two-hop path by combining one first-hop primitive with each second-hop primitive. However, in order to provide valid candidate primitives to the generator 108, they can be further filtered to remove second-hop primitives that cannot be reached from any first-hop primitive.

[0043] Figure 5 is a simplified diagram illustrating a sorting primitive for a database according to some embodiments. Enumeration lists 502 and 504 are as shown in reference Figure 1 and Figure 3 The list of column name and cell value primitives in question is enumerated by an enumerator (eg, enumerator 104).

[0044] In some embodiments, such as Figure 5 As shown, the enumeration list provided by the enumerator can be concatenated with the question, a special flag indicating whether the primitive is a column name or a cell value, and the primitive itself (through the enumerator, the sorter, or as another cell). For example, the enumeration list 502 includes the following: Figure 3 The text of question 302, the special cell value tag "<|tb_cl_vl|>" and the primitive "Course.Course Name <op>Statistics(Courses.CourseName <op>Statistics). Table 504 includes similar items concatenated with question 202, a primitive followed by a column name tag. These are exemplary concatenation patterns, and other patterns may be used. The enumerated list is provided to the sorter 106.

[0045] The training and use of the sequencer 106 can generally be compared with the reference Figure 4 The same as described in the database in . The enumerated database primitives will be applied instead of the enumerated knowledge base primitives. When selecting hard negative examples for training the sorter 106 on the database, for the <|tb_cl|> category primitives, the training method can consider those with real identical table names but different column names as hard negative examples. And for the <|tb_cl_vl|> category, the training method can consider candidates with real identical table names and column names but different cell values ​​as hard negative examples.

[0046] Sorted lists 506 and 508 similarly provide an ordering of database primitives, such as Figure 4 As discussed in the sorted lists 406 and 408 in FIG.

[0047] Figure 6 6 is a simplified diagram showing the logical form of generating a knowledge base according to some embodiments. For the knowledge base, the generator 108 receives input 602 from the sorter 106, which is the top-ranked primitives (concatenated with questions and special tokens). In some embodiments, the order of the primitives can be random so that the generator 108 does not abuse the position information to learn to select only the top-ranked primitives, but must learn to use semantics.

[0048] The generator 108 can learn to generate logical forms (queries composed of primitives and executable on corresponding data structures) by understanding the meaning of its elements (primitives and operations) and combining them. The generator 108 can be trained to generate output logical forms token by token and optimized by cross-entropy loss. At inference time, beam-search can be used to decode the top k target logical forms in an autoregressive manner.

[0049] An exemplary logical form output 604 is shown. As shown, the output 604 is: (JOIN money_unit.currency(JOINarchitecture.construction_cost central_park)). Note that the logical form consists of primitives that include entities of the knowledge base and operations (e.g., JOIN) performed on these entities. The logical form can be used as a query against the knowledge base to provide answers to questions (e.g., questions 102 or 202).

[0050] Figure 7 1 is a simplified diagram showing the logical form of generating a database according to some embodiments. In general, for a database, the training and utilization of the generator 108 is similar to Figure 6 Same as described in the knowledge base.

[0051] For a database, the generator 108 receives an input 702 that includes the sorted primitives from the sorter 106, but formatted as: [question; | table_name1 | column_name1, column_name2 <op>Value…|table_name|2】([Question;|table_name1|column_name 1,column_name2 <op>value...|table_name|2...]). For databases, the format of column names and cell values ​​organized according to the table they belong to will naturally change their sort order.

[0052] As shown, output such as output 704 may be generated, which is: SELECT count(*) FROM Courses JOIN Course_Attendance ON Courses.CourseId=CourseAttendance.CourseId WHERE Courses.Course Name="Statistics". Similar to Figure 6 The logical form described in , which logical form output 704 can be used as a query against a database to provide an answer to a question (e.g., question 102 or 302).

[0053] Figure 8 is a diagram showing an implementation according to an embodiment described herein. Figures 1 to 7 A simplified diagram of a computing device for the unified parser framework described in FIG. Figure 8 As shown, the computing device 800 includes a processor 810 coupled to a memory 820. The operation of the computing device 800 is controlled by the processor 810. In addition, although the computing device 800 is shown as having only one processor 810, it should be understood that the processor 810 can represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), etc. in the computing device 800. The computing device 800 can be implemented as a standalone subsystem, a board added to a computing device, and / or as a virtual machine.

[0054] The memory 820 may be used to store software executed by the computing device 800 and / or one or more data structures used during operation of the computing device 800. The memory 820 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media suitable for reading by a processor or computer.

[0055] The processor 810 and / or the memory 820 may be arranged in any suitable physical arrangement. In some embodiments, the processor 810 and / or the memory 820 may be implemented on the same board, in the same package (e.g., a system-in-package), on the same chip (e.g., a system-on-chip), etc. In some embodiments, the processor 810 and / or the memory 820 may include distributed, virtualized, and / or containerized computing resources. According to such embodiments, the processor 810 and / or the memory 820 may be located in one or more data centers and / or cloud computing facilities.

[0056] In some embodiments, the memory 820 may include a non-transitory tangible machine-readable medium including executable code that, when executed by one or more processors (e.g., processor 810), may cause the one or more processors to perform methods described in further detail herein. For example, as shown, the memory 820 includes instructions for a unified parser module 830 that may be used to implement and / or simulate systems and models, and / or implement any of the methods described further herein. The unified parser module 830 may receive input 840, such as input training data (e.g., questions and corresponding logical forms), via a data interface 815, and generate output 850, which may be a logical form based on a question for a given data structure.

[0057] The data interface 815 may include a communication interface, a user interface (such as a voice input interface, a graphical user interface, and / or the like). For example, the computing device 800 may receive input 840 (such as a training data set) from a networked database via the communication interface. Alternatively, the computing device 800 may receive input 840 from a user via the user interface, such as a question and / or a data structure.

[0058] In some embodiments, the unified parser module 830 is configured to generate a logical form for answering questions with a provided data structure. The unified parser module 830 may also include an enumerator submodule 831 (e.g., similar to Figure 1 ), the sorter submodule 832 (e.g., similar to Figure 1 ) and the generator submodule 833 (e.g., similar to Figure 1 In one embodiment, the unified parser module 830 and its submodules 831-833 may be implemented by hardware, software and / or a combination thereof.

[0059] Some embodiments of a computing device, such as computing device 800, may include a non-transitory tangible machine-readable medium including executable code that, when executed by one or more processors (e.g., processor 810), may cause the one or more processors to perform the processes of the method. Some common forms of machine-readable media that may include the processes of the method are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media suitable for reading by a processor or computer.

[0060] Fig. 9 is suitable for implementing Figures 1 to 7 900 shows a system that includes a user device 910 that can be operated by a user 940, data provider servers 945, 970 and 980, a server 930, and other forms of devices, servers and / or software components that operate to perform various methods according to the described embodiments. Exemplary devices and servers may include appliances, stand-alone and enterprise-level servers, which may be similar to Figure 8 The computing device 800 described in the embodiment of the present invention operates as follows: OS, OS, OS, or other suitable device and / or server-based operating systems. Fig. 9 The devices and / or servers shown may be deployed in other ways, and the operations performed and / or services provided by such devices and / or servers may be combined or separated for a given embodiment and may be performed by a greater or lesser number of devices and / or servers. One or more devices and / or servers may be operated and / or maintained by the same or different entities.

[0061] The user device 910, data provider servers 945, 970, and 980, and server 930 may communicate with each other via a network 960. The user device 910 may be used by a user 940 (e.g., a driver, a system administrator, etc.) to access various features available to the user device 910, which may include processes and / or applications associated with the server 930 to receive output data anomaly reports.

[0062] User device 910, data provider server 945, and server 930 may each include one or more processors, memory, and other appropriate components for executing instructions, such as program code and / or data stored on one or more computer-readable media, to implement the various applications, data, and steps described herein. For example, such instructions may be stored in one or more computer-readable media, such as memory or data storage devices internal and / or external to various components of system 900, and / or may be accessed via network 960.

[0063] The user device 910 may be implemented as a communication device that may utilize appropriate hardware and software configured to communicate wired and / or wirelessly with the data provider server 945 and / or the server 930. For example, in one embodiment, the user device 910 may be implemented as an autonomous vehicle, a personal computer (PC), a smartphone, a laptop / tablet, a watch with appropriate computer hardware resources, glasses with appropriate computer hardware (e.g., GOOGLE ), other types of wearable computing devices, implantable communication devices, and / or other types of computing devices capable of transmitting and / or receiving data, such as from of Although only one communication device is shown, multiple communication devices may operate similarly.

[0064] Fig. 9 The user device 910 may include a user interface (UI) application 912 and / or other applications 916, which may correspond to executable processes, procedures, and / or applications with associated hardware. For example, the user device 910 may receive a message indicating a logical form or a direct answer to a question from the server 930 and display the message via the UI application 912. In other embodiments, the user device 910 may include additional or different modules with specialized hardware and / or software as needed.

[0065] In various embodiments, the user device 910 includes other applications 916 that may be needed in a particular embodiment to provide features to the user device 910. For example, the other applications 916 may include security applications for implementing client-side security features, programming client applications, programming user applications for interfacing with appropriate application programming interfaces (APIs) through the network 960, or other types of applications. The other applications 916 may also include communication applications that allow users to send and receive emails, calls, text messages, and other notifications through the network 960, such as email, text messages, voice, social networking, and IM applications. For example, the other applications 916 may be email or instant messaging applications that receive query result messages from the server 930. The other applications 916 may include device interfaces and other display modules that can receive input and / or output information. For example, the other applications 916 may include software programs for asset management that can be executed by a processor, including a graphical user interface (GUI) configured to provide an interface to the user 940 to view the answer to a question.

[0066] The user device 910 may also include a database 918 stored in a temporary and / or non-temporary memory of the user device 910, which may store various applications and data and be utilized during the execution of various modules of the user device 910. The database 918 may store user profiles associated with the user 940, predictions previously viewed or saved by the user 940, historical data received from the server 930, etc. In some embodiments, the database 918 may be located locally on the user device 910. However, in other embodiments, the database 918 may be located external to the user device 910 and accessible by the user device 910, including a cloud storage system and / or database accessible via the network 960.

[0067] The user device 910 includes at least one network interface component 917 that is suitable for communicating with the data provider server 945 and / or the server 930. In various embodiments, the network interface component 917 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency, infrared, Bluetooth, and near field communication devices.

[0068] The data provider server 945 may correspond to a server that hosts the database 919 to provide a training data set including questions and corresponding logical forms to the server 930. The database 919 may be implemented by one or more relational databases, distributed databases, cloud databases, and the like.

[0069] The data provider server 945 includes at least one network interface component 926 adapted to communicate with the user device 910 and / or the server 930. In various embodiments, the network interface component 926 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency (RF), infrared (IR), Bluetooth, and near field communication devices. For example, in one implementation, the data provider server 945 may transmit asset information from the database 919 to the server 930 via the network interface 926.

[0070] The server 930 may accommodate Figure 1 In some embodiments, the unified parser module 830 may receive data from the database 919 at the data supplier server 945 via the network 960 to generate a logical form and / or a direct answer to the question. The generated logical form and / or answer may also be sent to the user device 910 via the network 960 for review by the user 940.

[0071] The database 932 may be stored in a temporary and / or non-temporary memory of the server 930. In one implementation, the database 932 may store data obtained from the data vendor server 945. In one implementation, the database 932 may store parameters of the unified parser module 830. In one implementation, the database 932 may store previously generated logical forms, and corresponding input feature vectors.

[0072] In some embodiments, database 932 may be located locally on server 930. However, in other embodiments, database 932 may be located external to server 930 and accessible by server 930, including a cloud storage system and / or database accessible via network 960.

[0073] The server 930 includes at least one network interface component 933 that is adapted to communicate with the user device 910 and / or the data provider server 945, 970, or 980 via the network 960. In various embodiments, the network interface component 933 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency (RF), and infrared (IR) communication devices.

[0074] The network 960 may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, the network 960 may include the Internet or one or more intranets, landline networks, wireless networks, and / or other suitable types of networks. Thus, the network 960 may correspond to a small-scale communication network, such as a private or local area network, or a larger-scale network, such as a wide area network or the Internet, that may be accessed by various components of the system 900.

[0075] A knowledge base and / or database for enumerating primitives and querying using the generated logical forms may be stored on one of the illustrated network devices, such as data provider server 945, server 930, or user device 910. Such a knowledge base / database may or may not be associated with or incorporated into the above-described databases (i.e., databases 918, 919, and 932).

[0076] Fig.10 is a diagram showing some embodiments described herein based on Figures 1 to 9 An example logic flow diagram of a method 1000 for semantic parsing of a Uni-Parse framework is shown. One or more processes of the method 1000 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors, may cause one or more processors to perform one or more processes. In some embodiments, the method 1000 corresponds to a unified parser module 830 (e.g., Figures 8 to 9 ) operation.

[0077] As shown, method 1000 includes a number of enumerated steps, but aspects of method 1000 may include additional steps before, after, and between the enumerated steps. In some aspects, one or more of the enumerated steps may be omitted or performed in a different order.

[0078] At step 1001, the system receives a natural language question (e.g., question 102, 202, or 302) via a communication interface. For example, the question may be input into the user interface by a user. For example, the communication interface may be a network interface of a computer, where the natural language question is received from a user device (e.g., user device 910) via a network interface (e.g., network interface 933) at a server (e.g., server 930).

[0079] At step 1002, the system identifies a first entity from a natural language question via a named entity recognition (NER) process. For example, the first entity can be a node in a knowledge graph, such as Figure 2 For a database, the first entity may be a cell value such as Figure 3 "Statistics" described in .

[0080] At step 1003, the system enumerates (e.g., via enumerator 104) a plurality of primitives indicating entities or entity relationships in a database or knowledge base based on the first entity. For example, in a database, primitives may include table column names and cell values ​​(including cell values ​​that match the first entity described in step 1002). For example, in a knowledge base, primitives may include entities and entity relationships of the knowledge base (including knowledge base entities that match the first entity described in step 1002). In some embodiments, the knowledge base entity that matches the first entity is the starting entity, from which other primitives that are one or two hops away from the starting entity are enumerated.

[0081] In step 1004, the system sorts multiple primitives based on the relevance of multiple primitives to natural language problems (for example, by sorter 106). In some embodiments, a separate sorting list is generated for every type of primitive (for example, the first jump and the second jump or the column name and the cell value). The primitives enumerated in step 1004 can be provided to the sorter in the form of their special markings (for example, the first jump, the second jump, the column name or the cell value) connected in series with the natural language problem and / or the identification primitive category. The sorter can be a model (for example, a neural network model) trained to minimize contrast loss. Contrast loss can be based on the positive primitive sample extracted from the real logical form, and the negative primitive sample. The method for selecting the negative primitive sample can be random, or can be selected as the "difficult" sample that can improve training. For example, in the knowledge base, negative samples can be selected from the primitives that are connected to the first jump entity extracted from the real logical form in the knowledge base. For the database, the cell value in the same table and column as the cell value extracted from the real logical form can be selected.

[0082] At step 1005, the system selects a subset of top-ranked primitives based on the ranking. This can be performed by the sorter 106, the generator 108, or another component of the system. The number of primitives selected can be based on a predetermined value. When the sorted list includes multiple sorted lists (e.g., a "first hop" list and a "second hop" list), each list can be sorted separately, and the top-ranked primitives can be selected from each list.

[0083] At step 1006, the system generates (e.g., by generator 108) a logical form executable on a database or knowledge base based on the natural language question and the subset of the plurality of primitives. The executable logical form may include at least one primitive from the subset. The logical form may be, for example, logical form 604 or 704.

[0084] In step 1007, the system transmits the logical form to a database system or knowledge base system via a communication interface. For example, the database system or knowledge base system may be such as Fig. 9 To a server such as the data provider server 945 described in , the logical form is transmitted to the server so that the server can execute the logical form on a database (or knowledge base).

[0085] At step 1008, the system receives, via the communication interface, query results responsive to the natural language question based on the logical form.The answers may be displayed to the user, for example, on the same user interface used to enter the question.

[0086] Figures 11 to 15 Graphs are provided that illustrate exemplary performance of various embodiments described herein. Compare to: Bert Ranking, as described in Gu et al., Beyond IID: three levels of generalization for question answering on knowledge bases, In Proceedings of the Web Conference, pp. 3477-3488, 2021; ReTrack, as described in Chen et al., Retrack: a flexible and efficient framework for knowledge base question answering, in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 325-336, 2021; UnifiedSKG, as described in Xie et al., Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models, arXiv:2201.05966, 2022; RNG-KBQA, as described in Ye et al., Rng-kbqa: Generation augmented iterative ranking for knowledge base question answering,arXiv:2109.08678, 2021; Topic Units, as described in Lan et al., Knowledge base question answering with topic units, 2019; STAGG, as described in Yih et al., Semantic parsing via staged query graph generation: Question answering with knowledge base, In Proceedings of the Joint Conference of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing of the AFNLP, 2015; QGG, as described in Lan et al., Query graph generation for answering multi-hop complex questions from knowledge bases, Association for Computational Linguistics, 2020; CBR, as described in Das et al., Case-based reasoning for natural language queries over knowledge bases, arXiv:2104.08762, 2021; ArcaneQA, as described in Gu and Su, Arcaneqa: Dynamic program induction and contextualized encoding for knowledge base question answering, as described in arXiv:2204.08109,2022; Global-GNN, as described in Bogin et al., Global reasoning over database structures for text-to-sql parsing, arXiv:1908.11214,2019; EditSQL, as described in Zhang et al., Editing-based sql query generation for cross-domain context-dependent questions, arXiv:1909.00786, 2019; T5-Base, as described in Scholak et al., Picard: Parsing incrementally for constrained auto-regressive decoding from language models, arXiv:2109.05093, 2021; RAT-SQL, as described in Wang et al., Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers, arXiv:1911.04942, 2019; BRIDGE, as described in Lin et al., Bridging textual and tabular data for cross-domain text-to-sql semantic parsing, arXiv:2012.12627, 2020; T5-3B, as described in Scholak et al., Picard: Parsing incrementally for constrained auto-regressive decoding from language models, arXiv:2109.05093, 2021; SQLova, as described in Hwang et al., A comprehensive exploration on wikisql with table-aware word contextualization, arXiv:1902.01069, 2019; X-SQL, as described in He et al., X-sql: reinforce schema representation with context, arXiv:1908.08113, 2019; IE-SQL, as described in Ma et al., Mention extraction and linking for sql query generation, arXiv:2012.10074, 2020; NL2SQL, as described in Guo and Gao, Content enhanced bert-based text-to-sql generation, arXiv:1910.07179, 2019; HydraNet, as described in Lyu et al., Hybrid ranking network for text-to-sql, arXiv:2008.04759, 2020; and TAPEX, as described in Liu et al., Tapex: Table pre-training via learning aneural sql executor, arXiv:2107.07653, 2021. .

[0087] The datasets used in the comparison include the following test and development (dev) datasets: GRA1LQA, as described in Gu et al., Beyond IID: three levels of generalization for question answering on knowledge bases, In Proceedings of the Web Conference, pages 3477-3488, 2021; Spider, as described in Yu et al., Spider: A large-scale human labeled dataset for complex and cross-domain semantic parsing and text-to-sql task, arXiv:1809.08887, 2018; and WikiSQL, as described in Zhong et al., Seq2sql: Generating structured queries from natural language using reinforcement learning, arXiv:1709.00103, 2017.

[0088] For the experiments, in order to construct the <|tb_cl_vl|> category primitives, relevant cell values ​​related to the question are found. Given a question and a database (DB), string matches between the phrase in question of arbitrary length and the list of cell values ​​under each column of all tables are computed. The question is matched with the possible cell values ​​mentioned in the database using a fuzzy matching algorithm. Numeric values ​​in the question are also detected and all column names are formed with this value as a primitive. The column names in the WikiSQL dataset are fuzzy, such as "No.", "Pick#," and "Rank," so the cell values ​​are used to supplement the meaning of the column names. The matched cell values ​​are used to locate the rows and match the column names with the cell values ​​in the same row. The primitive ranker is initialized using BERT-base-uncased. 48 negative candidates are sampled for each primitive category. The ranker is trained for 10 epochs using a learning rate of 1e-5 and a batch size of 8. Bootstrap hard negative sampling is performed after every two epochs. Ground truth entity links are used to enumerate training candidates. The generator is trained on the Spider dataset using T5-base and 3B. The T5-base model is fine-tuned for 200 epochs using the top 15 ranked <|tb_cl|> categorical primitives and the top 5 ranked <|tb_cl_vl|> categorical primitives returned by the ranker, with a learning rate of 5e-5 and a batch size of 64. For the T5-3B model, it runs on 16 A100 GPUs for 100 epochs using a batch size of 1024. And, on the WikiSQL dataset, T5-base and T5-large are used, and the top 5 ranked <|tb_cl|> categorical primitives and the top 3 ranked <|tb_cl_vl|> categorical primitives are used as the input to the generator. The T5-base / T5-large models are fine-tuned for 20 epochs using a learning rate of 3e-5 and a batch size of 16.

[0089] Figure 11 to Figure 12 Exact match (EM) and F1 scores for the test / development split of GRA1LQA are shown, comparing alternative methods with an implementation (Uni-Parse) of the method described in this paper. The reported models are based on the BERT-base model for the sorter and the T5-base model for the generator. The best development results are shown in bold, and the test results that are better than the development are underlined. As shown, the tested implementation (Unified Parser) generally outperforms existing methods. Compared with methods that enumerate logical forms (such as Bert Ranking and RNG-KBQA), the method described in this paper achieves better performance in the compositional setting and the zero-shot setting. In particular, in the compositional setting, a 3.1% improvement over the baseline is demonstrated in F1. This is in line with the expectation that the generator learns primitive compositions.

[0090] Figure 13 to Figure 14 Results on the Spider and WikiSQL datasets are summarized separately. On the Spider dataset, the unified parser achieves performance competitive with all baseline models. Compared to generative models that use the entire database table schema as input, such as BRIDGE and UnifiedSKG, the unified parser achieves a 3% improvement, demonstrating the strength of the approach. Compared to other T5-3B models, the unified parser achieves comparable performance with fewer training epochs. Fig.14 The upper box shows the comparison between small pre-trained models, and the lower box shows the comparison between large pre-trained models.

[0091] Fig.15 The effect of the hard negative strategy on ranking on the WebQSP and WikiSQL datasets is shown. No CG means that primitives are not distinguished by their categories. No CG shows lower performance than the setting that uses categories in the input. By comparing the settings with and without hard negative (the two rightmost columns), it is shown that the proposed hard negative sampling can help the ranker better determine positive primitives from negative primitives.

[0092] Figure 16 provides an exemplary comparison of the output of the semantic parsing framework described in the embodiments of this article with another semantic parser. The top five logical forms using RNG-KBQA contain a lot of redundant information between each other, and none of them can be compared with the golden logical expression. In contrast, the top five expressions of the unified parser method find the correct first-hop primitive and second-hop primitive and generate the correct logical form. This is not to say that even if the correct primitive is not ranked as the first place, the generator is capable of selecting it correctly.

[0093] The present specification and drawings describing aspects, embodiments, implementations or applications of the invention should not be considered as limiting. Various mechanical, compositional, structural, electrical and operational changes may be made without departing from the spirit and scope of the present specification and claims. In some cases, well-known circuits, structures or technologies are not shown or described in detail to avoid obscuring embodiments of the present invention. The same numbers in two or more drawings represent the same or similar elements.

[0094] In this specification, the specific details of describing some embodiments of the present invention are set forth. Many specific details are set forth in order to provide a thorough understanding of the embodiments. However, it is apparent to those skilled in the art that some embodiments may be put into practice in the absence of some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative rather than restrictive. Those skilled in the art may recognize that other elements, although not specifically described herein, are within the scope and spirit of the present invention. In addition, in order to avoid unnecessary repetition, unless specifically described in addition or if one or more features will make the embodiments inoperative, one or more features shown and described in conjunction with an embodiment may be incorporated into other embodiments.

[0095] Although exemplary embodiments have been shown and described, a wide range of modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without the corresponding use of other features. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications. Therefore, the scope of the present invention should be limited only by the appended claims, and the claims should be appropriately interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.< / op> < / op> < / op> < / op> < / op> < / op>

Claims

1. A method for generating a logical form executable on a database or knowledge base for a natural language question, the method comprising: Receiving the natural language question via a communication interface; identifying a first entity from the natural language question via a named entity recognition procedure; Enumerate, by an enumerator, a plurality of primitives representing entities or entity relationships in the database or the knowledge base based on the first entity; sorting, by a sorter module, the plurality of primitives based on a relevance of each of the plurality of primitives to the natural language question; Based on the ranking, selecting a subset of primitives ranked top; Generate, by a generator, the logical form executable on the database or the knowledge base based on the natural language question and the subset of the plurality of primitives; transmitting the logical form to a database system or a knowledge base system via the communication interface; as well as A query result responsive to the natural language question based on the logical form is received via the communication interface.

2. The method according to claim 1, further comprising: Based on matching a starting knowledge base entity with the first entity from the natural language question, the starting knowledge base entity is identified.

3. The method of claim 2, wherein the plurality of primitives comprises primitives of the knowledge base that are within a predetermined number of hops from the starting knowledge base entity.

4. The method of claim 1, wherein the plurality of primitives comprises: The column names of the tables in the database; and A cell value of the database is identified by matching the cell value with the first entity from the natural language question.

5. The method according to claim 1, further comprising: The plurality of primitives, respectively concatenated with the natural language question and a special token identifying a category of primitives, are provided to the sequencer module through the enumerator.

6. The method of claim 1, wherein the ranker module is trained to minimize a contrastive loss, wherein the contrastive loss is based on positive primitive samples extracted from true logical forms and on negative primitive samples. 7 . The method according to claim 6 , wherein the negative primitive samples are selected from primitives in the knowledge base connected to a first-hop entity extracted from the true logical form.

8. The method of claim 6, wherein the negative primitive samples are selected by selecting cell values ​​that are in the same table and column as the cell values ​​extracted from the true logical form.

9. A system for generating a logical form executable on a database or knowledge base for a natural language question, the system comprising: a memory storing the database or the knowledge base and a plurality of processor executable instructions; a communication interface that receives the natural language question; and one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising: identifying a first entity from the natural language question via a named entity recognition procedure; Enumerate, by an enumerator, a plurality of primitives representing entities or entity relationships in the database or the knowledge base based on the first entity; sorting, by a sorter module, the plurality of primitives based on a relevance of each of the plurality of primitives to the natural language question; Based on the ranking, selecting a subset of primitives ranked top; Generate, by a generator, the logical form executable on the database or the knowledge base based on the natural language question and the subset of the plurality of primitives; transmitting the logical form to a database system or a knowledge base system via the communication interface; and A query result responsive to the natural language question based on the logical form is received via the communication interface.

10. The system of claim 9, the operations further comprising: Based on matching a starting knowledge base entity with the first entity from the natural language question, the starting knowledge base entity is identified.

11. The system of claim 10, wherein the plurality of primitives comprises primitives of the knowledge base that are within a predetermined number of hops from the starting knowledge base entity.

12. The system of claim 9, wherein the plurality of primitives comprises: The column names of the tables in the database; and A cell value of the database is identified by matching the cell value with the first entity from the natural language question.

13. The system of claim 9, the operations further comprising: The plurality of primitives, respectively concatenated with the natural language question and a special token identifying a category of primitives, are provided to the sequencer module through the enumerator.

14. The system of claim 9, wherein the ranker module is trained to minimize a contrastive loss, wherein the contrastive loss is based on positive primitive samples extracted from true logical forms and on negative primitive samples. 15 . The system of claim 14 , wherein the negative primitive samples are selected from primitives in the knowledge base connected to a first-hop entity extracted from the true logical form.

16. The system of claim 14, wherein the negative primitive sample is selected by selecting a cell value that is in the same table and column as a cell value extracted from the true logical form.

17. A non-transitory machine-readable medium comprising a plurality of machine-executable instructions, which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising: Receiving the natural language question via a communication interface; identifying a first entity from the natural language question via a named entity recognition procedure; Enumerate, by an enumerator, a plurality of primitives representing entities or entity relationships in the database or the knowledge base based on the first entity; sorting, by a sorter module, the plurality of primitives based on a relevance of each of the plurality of primitives to the natural language question; Based on the ranking, selecting a subset of primitives ranked top; Generate, by a generator, the logical form executable on the database or the knowledge base based on the natural language question and the subset of the plurality of primitives; transmitting the logical form to a database system or a knowledge base system via the communication interface; as well as A query result responsive to the natural language question based on the logical form is received via the communication interface.

18. The non-transitory machine-readable medium of claim 17, the operations further comprising: Based on matching a starting knowledge base entity with the first entity from the natural language question, the starting knowledge base entity is identified.

19. The non-transitory machine-readable medium of claim 18, wherein the plurality of primitives comprises primitives of the knowledge base that are within a predetermined number of hops from the starting knowledge base entity.

20. The non-transitory machine-readable medium of claim 17, wherein the plurality of primitives comprises: The column names of the tables in the database; and A cell value of the database is identified by matching the cell value with the first entity from the natural language question.

Images