System and Method for Hierarchical Search of Semantic-Based Messages in Deep Learning

Abstract

The embodiments described herein provide dense hierarchical search for open domain questions and answers over a corpus of documents using dense document-level and node-level search models. Specifically, each document is considered as a structured collection with sections, subsections, and paragraphs. Each document may be divided into short-length nodes, where document-level and node-level search models can be applied to return a smaller set of filtered text. After encoding the question and documents and determining document relevance scores to the encoded question, top documents may be identified. Then, a set of top nodes is further identified based on encoding the nodes and determining node relevance scores to the encoded question. The document and node relevance scores may be used in combination to determine a final search ranking of the documents with the set of top nodes.

Description

Technical Field

[0001] [Cross - Reference] This application claims priority to U.S. Non - Provisional Application No. 17 / 533,613, filed on November 23, 2021, and U.S. Provisional Application No. 63 / 189,505, filed on May 17, 2021, the entire disclosures of which are hereby incorporated by reference in their entirety. [Copyright Notice] Part of the disclosure of this patent document contains materials subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the patent file or records of the Patent and Trademark Office, but reserves all copyrights otherwise in all cases.

[0002] [Technical Field] Embodiments generally relate to machine learning systems and deep learning, and more specifically to a hierarchical retrieval framework for semantic - based data.

Background Art

[0003] The subject matter discussed in the Background Art section should not be assumed to be prior art simply as a result of mention in the Background Art section. Similarly, problems mentioned in the Background Art section or related to the subject matter of the Background Art section should not be assumed to have been previously recognized in the prior art. The subject matter of the Background Art section merely represents a different approach and may itself be an invention.

[0004] Machine learning (ML) and neural network (NN) systems can be used, for example, to understand human speech and writing in order to understand the overall intent, syntax, and / or meaning of human communication. Such ML / NN systems may be trained using large amounts of training text, including different document corpora, which may or may not be pre-annotated with labels (supervised) or without pre-annotated labels (unsupervised). When training an ML system, different training data, including characters, words, phrases, Message (passages), and content from a document may be used. However, the training data and the specifications of such data vary in scope and may cause different predictions and classifications when using a large document corpus. Further, different uses of training data having different documents or from different documents, etc., may result in unpredictable and / or slower search results when the ML / NN model is trained. Message Recent research on dense neural retrievers has achieved promising results in open-domain question answering (QA) by ML / NN systems, where the latent representations of questions and

[0005] may be used for maximum inner product search in the retrieval process. However, training a dense retriever requires splitting the documents into short Message and its representations may contain local, partial, and sometimes biased content, so the training is highly dependent on the splitting process. As a result, the training results in hidden representations that are inaccurate and misleading within the model, thus degrading the final retrieval results by the ML / NN system. Message parts, Message and

Brief Description of the Drawings

[0006]

Figure 1

[0007]

Figure 2

[0008]

Figure 3

[0009]

Figure 4

[0010]

Figure 5

[0011]

Figure 6

[0012] In the drawings, elements having the same reference numerals have the same or similar functions.

DETAILED DESCRIPTION OF THE INVENTION

[0013] The present description and the accompanying drawings illustrating aspects, embodiments, implementations or applications should not be construed as limiting the claims defining the protected invention. Various mechanical, compositional, structural, electrical and operational changes may be made without departing from the spirit and scope of the present description and claims. In some instances, well-known circuits, structures or techniques have not been shown or described in detail because they are known to those skilled in the art. Like numbers in two or more drawings represent the same or similar elements.

[0014] In this description, specific details are described to describe some embodiments consistent with the present disclosure. Many specific details are described to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative, but not limiting. One skilled in the art can appreciate other elements that are not specifically described herein but are within the scope and spirit of the present disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated in other embodiments, unless otherwise noted or if one or more features render the embodiment non-functional.

[0015] Deep learning is widely used in ML and NN systems. In contrasting learning, open-domain QA is sometimes used to answer questions about factoids. Previously, dense networks were used to answer questions. Message Dense passage retrieval has sometimes been used. One common approach is to utilize a retriever-reader approach to provide answers. In such open-domain question answering, a question is given and a set of relevant contexts in a document corpus is predicted. However, extracting relevant contexts from a large document corpus like Wikipedia is difficult and suffers from weaknesses such as similar topics may be related to a particular question. Furthermore, the lack of information from documentsMessage may contain only local and specific information, leading to expressions that distract attention.

[0016] Instead, Dense Hierarchical Retrieval (DHR) may be used to generate an accurate dense representation of Message by leveraging both the macro meaning within the document and the micro meaning specific to each Message . First, for example, documents relevant to the question are retrieved based on encoding documents from a document corpus. Documents can be encoded at the document level using summaries, tables of contents, and / or other title lists within the document. Then, relevant Message can be retrieved by a retrieval model calibrated at the Message level with document-level relevance. To further enhance the global meaning, each Message is combined with a hierarchical list of titles. To better learn positive Message , two negative sampling strategies may be introduced, i.e., in-Doc negative samples and in-Sec negative samples within the document may be used as hard contrast samples. DHR is applied to large-scale open-domain QA datasets, where the dense hierarchical retrieval model may perform better than a dense Message retriever and helps an end-to-end QA system establish better results on multiple open-domain QA benchmarks.

[0017] As used herein, the term "network" may include any hardware or software-based framework that includes any artificial intelligence network or system, NN or system, and / or any training or learning model implemented on or with them.

[0018] As used herein, the term "module" can include any hardware or software-based framework that performs one or more functions. In some embodiments, the module can be implemented on one or more NNs.

[0019] Summary For a database system accessible by multiple distinct organizations, such as a multi-tenant database system, a method, data structure, and system are provided for processing a document corpus using a document-level retrieval model and Message a passage-level retrieval model. The database system stores a number of documents accessible by users of the database system, referred to as a document corpus or multiple document corpora. The documents can be generated by users or administrators (e.g., agents of an organization) of the database system based on, for example, inputs, articles, requests, and other documents that provide some information, such as information articles or encyclopedic entries, help requests, training manuals, pamphlets, or other articles related to a subject that provides information. At least some of the documents stored by the database system are associated with relevant text regarding the title or subject of the document. The documents in the document corpus can further include one or more document structures, including summaries, tables of contents (ToC), sections and corresponding section titles, subsections and corresponding subsection titles, lists of titles, paragraphs, sentences, and / or other text. Message associated therewith.

[0020] The embodiments described herein employ ML and NN techniques in a document-level retrieval model and Message a passage-level retrieval mode to derive meaning-based MessageA method, computer program product, and computer database system for hierarchically searching are provided. The online system provides users with access to online services and a document corpus. For example, the online system can be a web-based system that provides users with access to encyclopedic resources and / or customer relationship management (CRM) software applications. As part of providing services to users, the online system stores a document corpus that is accessible by and searchable using other search engines such as ML / NN processes and / or natural language processors that are trained and accessible by users of the online system. The document corpus can be generated by users or administrators of the online system, for example, based on document input and identification.

[0021] According to some embodiments, in a multi-tenant database system accessible by multiple distinct different organizations, considering the specificity of each document, document structure, and Message a neural network model is provided to process the document corpus and use DHR to provide relevant meaning-based Message thereby improving the experience of users associated with the organization, providing faster search results, and minimizing the time processing cost for text search.

[0022] Exemplary Environment The systems and methods of the present disclosure can be included, incorporated, or operate with or within a database environment, and in some embodiments, the database can be implemented as a multi-tenant cloud-based architecture. The multi-tenant cloud-based architecture was developed to improve collaboration, integration, and community-based cooperation among customer tenants without sacrificing data security. Generally speaking, multi-tenant refers to a system in which a single hardware and software platform simultaneously supports multiple user groups (also called "organizations" or "tenants") from a common data storage element (also called a "multi-tenant database"). The multi-tenant design offers many advantages over traditional server virtualization systems. First, the operator of a multi-tenant platform can often make improvements to the platform based on collective information from the entire tenant community. Further, since all users within a multi-tenant environment execute applications within a common processing space, it is relatively easy to grant or deny access to a particular dataset to any user within the multi-tenant platform, thereby improving collaboration and integration between applications and the data managed by various applications. Thus, the multi-tenant architecture enables convenient and cost-effective sharing of similar application functionality among multiple sets of users.

[0023] FIG. 1 shows a block diagram of an exemplary environment 110 according to some embodiments. The environment 110 may include a user system 112, a network 114, a system 116, a processor system 117, an application platform 118, a network interface 120, tenant data storage 122, system data storage 124, program code 126, and a process space 128 for executing database system processes and tenant-specific processes, such as executing an application as part of an application hosting service. In other embodiments, the environment 110 may not have all of the listed components and / or may have other elements instead of or in addition to the components listed above.

[0024] In some embodiments, the environment 110 is an environment in which an on-demand database service exists. The user system 112 may be any machine or system used by a user to access a database user system. For example, any of the user systems 112 may be a handheld computing device, a mobile phone, a laptop computer, a notebook computer, a workstation, and / or a network of computing devices. As illustrated in FIG. 1 (and in more detail in FIG. 2), the user system 112 may interact with an on-demand database service that is the system 116 via the network 114.

[0025] An on-demand database service that can be implemented using system 116 is a service made available to users external to the enterprise that owns, maintains, or provides access to system 116. As described above, such users do not necessarily need to be involved in the construction and / or maintenance of system 116. Instead, the resources provided by system 116 can be made available for use by such users when the users need the services provided by system 116, e.g., in response to a user's request. Some on-demand database services may store information from one or more tenants stored in tables of a common database image to form a multi-tenant database system (MTS). Thus, "on-demand database service 116" and "system 116" are used interchangeably herein. The term "multi-tenant database system" can refer to a system in which various elements of the hardware and software of the database system can be shared by one or more customers or tenants. For example, a given application server may process requests for a number of customers simultaneously, and a given database table may potentially store rows of data, such as feed items, for a much larger number of customers. A database image may include one or more database objects. A relational database management system (RDBMS) or equivalent may perform storage and retrieval of information for database objects.

[0026] The application platform 118 can be a framework that enables the execution of applications of the system 116, such as a hardware and / or software infrastructure, for example, an operating system. In one embodiment, the system 116 can include an application platform 118 that enables the creation, management, and execution of one or more applications developed by a provider of on-demand database services, a user accessing the on-demand database services via the user system 112, or a third-party application developer accessing the on-demand database services via the user system 112.

[0027] Users of the user system 112 may each have different capacities, and the capacity of a particular one of the user systems 112 may be completely determined by the permissions (permission levels) for the current user. For example, if a salesperson is using a particular user system 112 to interact with the system 116, that user system has the capacity assigned to that salesperson. However, while an administrator is using that user system 112 to interact with the system 116, that user system 112 has the capacity assigned to that administrator. In a system using a hierarchical role model, a user with a certain permission level may have access to applications, data, and database information that are accessible by users with lower permission levels, but may not have access to certain applications, database information, and data that are accessible by users with higher permission levels. Thus, different users have different capabilities with respect to access to and modification of applications and database information, depending on the security or permission level of the user.

[0028] Network 114 is any network or combination of networks of devices that communicate with each other. For example, network 114 can be any one or any combination of a local area network (LAN), a wide area network (WAN), a telephone network, a wireless network, a point-to-point network, a star network, a token ring network, a hub network, or other suitable configurations. The most common type of computer network currently in use is the Transmission Control Protocol and Internet Protocol (TCP / IP) network, such as the global Internet network often referred to as the "Internet" with a capital "I". However, while TCP / IP is a frequently implemented protocol, it should be understood that the networks that can be used in this embodiment are not limited thereto.

[0029] User system 112 communicates with system 116 using TCP / IP and, at a higher network level, can communicate using other common Internet protocols. Such as the Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Andrew File System (AFS), Wireless Application Protocol (WAP), etc. In an example where HTTP is used, user system 112 may include an HTTP client, commonly referred to as a "browser", for sending and receiving HTTP messages with an HTTP server in system 116. Such an HTTP server can be implemented as the only network interface between system 116 and network 114, but other technologies may also be used equally or instead. In some implementations, the interface between system 116 and network 114 includes a load sharing function such as a round-robin HTTP request distributor for balancing the load and evenly distributing incoming HTTP requests across multiple servers. At least, for each user accessing the server, each of the multiple servers has access to MTS data, but other alternative configurations may be used instead.

[0030] In some embodiments, the system 116 shown in FIG. 1 implements a web-based CRM system. For example, in one embodiment, the system 116 not only implements and runs a CRM software application, but also provides related data, code, forms, web pages, and other information between the user system 112, stores in and retrieves from database system-related data, objects, and web page content. It includes an application server configured to do so. In a multi-tenant system, data of multiple tenants can be stored in the same physical database object. However, tenant data is typically arranged such that the data of one tenant is logically separated from the data of other tenants, so that one tenant cannot access the data of another tenant unless such data is explicitly shared. In certain embodiments, the system 116 implements an application other than the CRM application, or implements applications in addition to the CRM application. For example, the system 16 can provide tenant access to a plurality of hosted (standard and custom) applications including the CRM application. User (or third-party developer) applications may or may not include CRM, but may be supported by the application platform 118, which manages the creation of applications in virtual machines within the process space of the system 116, the storage of applications to one or more database objects, and the execution of applications.

[0031] One configuration of the elements of system 116, including network interface 120, application platform 118, tenant data storage 122 for tenant data 123, system data storage 124 for system data 125 accessible to system 116 and possibly multiple tenants, program code 126 for implementing various functions of system 116, and process space 128 for executing MTS system processes and tenant-specific processes, such as running an application as part of an application hosting service, is shown in FIG. 1. Additional processes that may execute on system 116 include a database indexing process.

[0032] Some elements of the system shown in FIG. 1 include conventional well-known elements that are briefly described herein. For example, each of the user systems 112 can include a desktop personal computer, a workstation, a laptop, a notebook computer, a PDA, a cellular phone, or any other computing device that can interface directly or indirectly with any wireless access protocol (WAP)-enabled device or the Internet or other network connection. Each of the user systems 112 typically runs a browsing program such as an HTTP client, for example, Microsoft® Internet Explorer browser, Netscape Navigator browser, Opera browser, or in the case of a cellular phone, notebook computer, PDA, or other wireless device, a WAP-enabled browser, enabling a user of the user system 112 (e.g., a subscriber to a multi-tenant database system) to access, process, and view information, pages, and applications available from the system 116 via the network 114. Also, each of the user systems 112 typically includes one or more user interface devices, such as a keyboard, mouse, trackball, touchpad, touch screen, pen, etc., for interacting with the graphical user interface (GUI) provided by a browser on a display (e.g., a monitor screen, a liquid crystal display (LCD) monitor, a light emitting diode (LED) monitor, an organic light emitting diode (OLED) monitor, etc.) along with pages, forms, applications, and other information provided by the system 116 or other systems or servers. For example, the user interface device can be used to access data and applications hosted by the system 116, perform searches on stored data, or enable the user to interact with the various GUI pages that can be presented to the user. As described above, the embodiments are suitable for use with the Internet, which refers to a particular global Internetwork of networks.However, it should be understood that other networks such as an intranet, an extranet, a virtual private network (VPN), a non-TCP / IP-based network, any LAN or WAN, etc. can be used instead of the Internet.

[0033] According to one embodiment, each of the user systems 112 and all of its components can be operator-configurable using an application such as a browser, including computer code executed using a central processing unit such as an Intel Pentium® processor or the like. Similarly, all of system 116 (and additional instances of MTS if there are two or more) and its components can be operator-configurable using an application, including computer code executed using a central processing unit and / or multiple processor units such as a processor system 117 that may include an Intel Pentium® processor or the like. Embodiments of a computer program product include a machine-readable storage medium having instructions stored thereon / therein for programming a computer to execute any of the processes of the embodiments described herein. The computer code for operating and configuring system 116 to communicate and process web pages, applications, and other data and media content described herein is preferably downloaded and stored on a hard disk, although all or part of the program code may also be stored in any other well-known volatile or non-volatile memory medium or device such as read-only memory (ROM) or random access memory (RAM), or provided on any medium capable of storing program code, such as any type of rotating medium like a floppy disk, optical disk, digital versatile disk (DVD), compact disk (CD), microdrive, and magneto-optical disk, magnetic or optical card, nanosystem (including molecular memory integrated circuit (IC)), or any other type of medium or device suitable for storing instructions and / or data.Furthermore, all or part of the program code may be transmitted and downloaded from a software source via a transmission medium, for example, via the Internet or from another server as is well known, or transmitted via any other conventional network connection (for example, an extranet, a virtual private network (VPN), a LAN, etc.) using any communication medium and protocol (for example, TCP / IP, HTTP, HTTPS, Ethernet (registered trademark), etc.) as is well known. It will also be understood that the computer code for implementing the embodiments of the present disclosure can be implemented in any programming language executable on a client system and / or a server or server system, such as C, C++, HTML, any other markup language, Java (registered trademark), JavaScript, ActiveX, VBScript, and many other well-known scripting languages, and many other programming languages. (Java (registered trademark) is a trademark of Sun MicroSystems, Inc.)

[0034] According to one embodiment, system 116 is configured to provide web pages, forms, applications, data, and media content to user (client) system 112 and support access by user system 112 as a tenant of system 116. As such, system 116 provides a security mechanism for maintaining data separation for each tenant unless the data is shared. When two or more MTSs are used, they may be located in proximity to each other (e.g., within a server farm located in a single building or campus), or they may be dispersed in locations remote from each other (e.g., one or more servers located in City A and one or more servers located in City B). As used herein, each MTS can include one or more logically and / or physically connected servers that are local or dispersed across one or more geographical locations. Further, the term "server" is intended to include a computer system that includes processing hardware and process space, and associated storage systems and database applications (e.g., an object-oriented database management system (OODBMS) or a rational database management system (RDBMS)). It should also be understood that the terms "server system" and "server" are often used interchangeably herein. Similarly, database objects described herein can be implemented as a single database, a distributed database, a collection of distributed databases, a redundant online or offline backup, or other redundant databases, etc., and can include a distributed database or storage network and associated processing intelligence.

[0035] FIG. 2 also illustrates environment 110, which can be used to implement the embodiments described herein. FIG. 2 further illustrates elements of system 116 and various interconnections, according to some embodiments. FIG. 2 shows that each of user systems 112 can include a processor system 112A, a memory system 112B, an input system 112C, and an output system 112D. FIG. 2 shows network 114 and system 116. FIG. 2 also shows that system 116 can include tenant data storage 122, tenant data 123, system data storage 124, system data 125, user interface (UI) 230, application program interface (API) 232, PL / Salesforce.com object query language (PL / SOQL) 234, save routine 236, application setup mechanism 238, application servers 2001-200N, system process space 202, tenant process space 204, tenant management process space 210, tenant storage area 212, user storage 214, and application metadata 216. In other embodiments, environment 110 may not have the same elements as those enumerated above and / or may have other elements instead of or in addition to those enumerated above.

[0036] The user system 112, network 114, system 116, tenant data storage 122, and system data storage 124 were discussed above in FIG. 1. Regarding the user system 112, the processor system 112A may be any combination of one or more processors. The memory system 112B may be any combination of one or more memory devices, short-term memory, and / or long-term memory. The input system 112C may be any combination of input devices such as one or more keyboards, mice, trackballs, scanners, cameras, and / or interfaces to a network. The output system 112D may be any combination of output devices such as one or more monitors, printers, and / or interfaces to a network. As shown in FIG. 2, the system 116 may include a network interface 120 (of FIG. 1) implemented as a set of an HTTP application server 200, an application platform 118, tenant data storage 122, and system data storage 124. Also shown is a system process space 202 that includes individual tenant process spaces 204 and a tenant management process space 210. Each application server 200 may be configured to access tenant data storage 122 and tenant data 123 therein, as well as system data storage 124 and system data 125 therein, to respond to requests from the user system 112. The tenant data 123 may be divided into individual tenant storage areas 212, and the tenant storage areas 212 can be either a physical arrangement and / or a logical arrangement of data. Within each tenant storage area 212, user storage 214 and application metadata 216 may be similarly allocated for each user. For example, a copy of the user's most recently used (MRU) items may be stored in the user storage 214. Similarly, a copy of the MRU items for the entire organization that is a tenant may be stored in the tenant storage area 212.UI230 provides a user interface, and API232 provides an application programming interface to the system 116 resident process and to the users and / or developers of the user system 112. Tenant data and system data can be stored in various databases such as one or more Oracle® databases.

[0037] The application platform 118 includes an application setup mechanism 238 that supports the creation and management of applications by application developers, and such applications can be stored as metadata in the tenant data storage 122 by a save routine 236 for execution by subscribers as one or more tenant process spaces 204 managed, for example, by the tenant management process space 210. Calls to such applications can be coded using PL / SOQL 234, which provides a programming language style interface extension to the API 232. Some embodiments of the PL / SOQL language are discussed in more detail in U.S. Patent No. 7,730,478, entitled "Method and System For Allowing Access to Developed Applications Via a Multi-Tenant On-Demand Database Service," filed on September 21, 2007, which is incorporated herein by reference. Calls to an application may be detected by one or more system processes, which manage searching for application metadata 216 for a subscriber, making the call, and executing the metadata as an application within a virtual machine.

[0038] Each application server 200 may be communicatively coupled to a database system having access to system data 125 and tenant data 123, for example, via different network connections. For example, an application server 2001 may be coupled via a network 114 (e.g., the Internet), and another application server 200 N-1 may be coupled directly via a network link, and another application server 200 N may be coupled by yet different network connections. Transmission Control Protocol and Internet Protocol (TCP / IP) are typical protocols for communicating between the application server 200 and the database system. However, as will be apparent to those skilled in the art, other transport protocols may be used to optimize the system depending on the network connections used.

[0039] In one embodiment, each application server 200 is configured to process requests for any user associated with any organization that is a tenant. For any reason, it is desirable to be able to add or remove application servers from the server pool at any time, so it is preferred that there is no server affinity for a particular application server 200 with respect to users and / or organizations. Thus, in one embodiment, an interface system (e.g., an F5 Big-IP load balancer) that implements a load balancing function is communicatively coupled between the application server 200 and the user system 112 to distribute requests to the application server 200. In one embodiment, the load balancer uses a least connections algorithm to route user requests to the application server 200. Other examples of load balancing algorithms, such as round robin and observed response time, can also be used. For example, in one embodiment, three consecutive requests from the same user may hit three different application servers 200, and three requests from different users may hit the same application server 200. Thus, the system 116 is multi-tenant, where the system 116 processes the storage and access to different objects, data, and applications across separate users and organizations.

[0040] As an example of storage, one tenant can be a company that employs a sales force where each salesperson uses system 116 to manage their sales process and / or provide information to other users, agents, and administrators who may be able to search it. Thus, a user can maintain (e.g., within tenant data storage 122) all that is applicable to that user, such as contact data, lead data, customer follow-up data, performance data, goals and progress data, training materials, retrieved articles, etc. In an example of an MTS configuration, since all of the data and applications to be accessed, viewed, modified, reported on, sent, calculated, etc. are maintained and accessible by a user system that has nothing other than network access, a user can manage their information from any of many different user systems. For example, if a salesperson is visiting a customer and the customer has Internet access in the lobby, the salesperson can obtain important updated information about that customer while waiting for the customer to arrive in the lobby.

[0041] The data of each user may be separated from the data of other users, regardless of the employer of each user, although some data may be organization-wide data that is shared or accessible by multiple users of a given organization that is a tenant or all of the users. Accordingly, there are some data structures that are managed by the system 116 assigned at the tenant level, while other data structures may be managed at the user level. Since the MTS may support multiple tenants including potential competitors, the MTS should have security protocols that maintain separation of data, applications, and use of applications. Also, since many tenants may choose to access the MTS rather than maintain their own systems, redundancy, uptime, and backup are additional features that may be implemented at the MTS. In addition to user-specific data and tenant-specific data, the system 116 may also maintain system-level data that is available for use by multiple tenants or other data. Such system-level data may include industry reports, news, posts, etc. that are shareable among tenants.

[0042] In one embodiment, the user system 112 (which may be a client system) may need to communicate with the application server 200 to request and update system-level and tenant-level data from the system 116 that may require sending one or more queries to the tenant data storage 122 and / or the system data storage 124. The system 116 (e.g., the application server 200 within the system 116) automatically generates one or more Structured Query Language (SQL) statements (e.g., one or more SQL queries) designed to access the desired information. In other embodiments, other types of searches, such as a natural language processor or a machine learning engine, may be performed based on the input data. The system data storage 124 may generate a query plan for accessing the data requested from the database, and the query plan may include external objects based on references to objects within the document.

[0043] In a database system such as the system 116 shown and described with respect to FIGS. 1 and 2, data or information can be organized or arranged in categories or groupings. Each database can generally be considered a set of objects, such as a set of logical tables, that contain data conforming to pre - defined categories. A “table” is one representation of a data object and can be used herein to simplify the conceptual description of objects and custom objects. It should be understood that “table” and “object” can be used interchangeably herein. Each table typically contains one or more data categories that are logically arranged as columns or fields of a displayable schema. Each row or record of a table contains an instance of the data for each category defined by the fields.

[0044] In an encyclopedia system and / or a CRM system, for example, these categories or groupings can include various standard tables associated with a corpus of documents, such as a list of documents belonging to the corpus, and information associated with the search of those corpora presented to the system (e.g., encoded documents, summaries, ToC, text Message and additional aforementioned document text). For example, a database may include a table that describes a corpus of documents (e.g., one or more documents that can be searched for a topic or the system itself) and can include this text of the documents within the corpus. In some multi - tenant database systems, the tables and documents within the database may be provided for use by all tenants, or may be viewable only by some tenants and agents of the system (e.g., users and administrators).

[0045] In some multi-tenant database systems, a tenant may be allowed to create and store custom objects, or may be allowed to customize standard entities or objects by creating custom fields for standard objects, such as custom index fields. Systems and methods for creating custom objects and customizing standard objects in a multi-tenant database system are described in detail in U.S. Patent No. 7,779,039, entitled "Custom Entities and Fields in a Multi-Tenant Database System," filed on April 2, 2004, which is hereby incorporated by reference. In certain embodiments, for example, all custom entity data rows are stored in a single multi-tenant physical table, which may include multiple logical tables for each organization. It is apparent to a customer that multiple "tables" are actually stored in one large table, or that the customer's data may be stored in the same table as other customers' data.

[0046] The multi-tenant database system 116 described above can be accessed and used by multiple customers, clients, or other persons (generally "users") for searching and / or browsing encyclopedic entries, inquiries, problems, questions, issues, support-related matters, training or education, etc. However, in other embodiments, other types of search systems may also utilize the processes described herein to provide a semantically dense hierarchical search within documents. To facilitate interaction between the system 116 and the user, a search bar, voice interface, or similar user interface tool is provided. The search tool enables the user to query the database and access information or data related to or associated with various documents, objects, and / or entities relevant to the user. Message In some multi-tenant database systems, a tenant may be allowed to create and store custom objects, or may be allowed to customize standard entities or objects by creating custom fields for standard objects, such as custom index fields. Systems and methods for creating custom objects and customizing standard objects in a multi-tenant database system are described in detail in U.S. Patent No. 7,779,039, entitled "Custom Entities and Fields in a Multi-Tenant Database System," filed on April 2, 2004, which is hereby incorporated by reference. In certain embodiments, for example, all custom entity data rows are stored in a single multi-tenant physical table, which may include multiple logical tables for each organization. It is apparent to a customer that multiple "tables" are actually stored in one large table, or that the customer's data may be stored in the same table as other customers' data.

[0047] However, in the case of a large database having a large number of records and information, some or all of the documents may have one or more document structures (e.g., summaries, ToC, sections and corresponding section titles, subsections and corresponding subsection titles, table of titles, etc.) and Message (e.g., paragraphs, sentences and / or other text), there may be a large number of documents. For example, a document may include a document structure that specifies sections and Message and the corresponding text of Message . When searching for a document, conventional search techniques for open-domain QA (e.g., machine learning systems trained using character or word embeddings or vectors) can only search the content of the document by splitting the Message within the document and encoding the question and Message for searching. Therefore, the search index data of the database system may not be an accurate basis for predicting appropriate search results for a search query when not considering the document and document structure in addition to Message . Predicting and ordering search results for searches performed by users on a large document corpus is a difficult task. In a multi-tenant system such as Salesforce.com, a document may include a document structure, Message , etc. Continuing with this example, since the user may be most interested in the relevant search results for the query having all the returned data, for an optimal or enhanced user experience, it is desirable or preferable that the database system predicts the document most relevant or applicable to the user's search or query, and as a result, the desired information or data is presented to the user with the fewest number of keystrokes, mouse clicks, user interface, etc. Therefore, according to some embodiments, a system and method are provided for predicting and returning search results using a document-level search model and encoder and Message one or more dense hierarchical search models that may include a level search model and encoder.

[0048] Dense Hierarchical Search Model According to some embodiments, in a multi-tenant database system accessible by a plurality of distinct different organizations, such as the system 116 shown and described with respect to FIGS. 1 and 2, in consideration of the document-level data and structure having level data of one or more corpora of documents, a dense hierarchical search model is provided for an intelligent search process that returns to the database the results most relevant to a given query, thereby providing an improved user experience. Message

[0049] FIG. 3 is a simplified diagram of a computing device implementing a hierarchical search of semantic-based training data for deep learning according to some embodiments described herein. As shown in FIG. 3, the computing device 300 includes a processor 310 coupled to a memory 320. The operation of the computing device 300 is controlled by the processor 310. Also, although the computing device 300 is shown as having only one processor 310, it is understood that the processor 310 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), etc. within the computing device 300. The computing device 300 may be implemented as a stand-alone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0050] ​Memory 320 can be used to store software executed by computing device 300 and / or one or more data structures used during the operation of computing device 300. Memory 320 may include one or more types of machine-readable media. Some common forms of machine-readable media include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAMs, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0051] Processor 310 and / or memory 320 may be arranged in any suitable physical configuration. In some embodiments, processor 310 and / or memory 320 may be implemented on the same board, within the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), etc. In some embodiments, processor 310 and / or memory 320 may include distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, processor 310 and / or memory 320 may be arranged within one or more data centers and / or cloud computing facilities.

[0052] In some examples, memory 320 may include a non-transitory tangible machine-readable medium and / or a medium including executable code that, when executed by one or more processors (e.g., processor 310), can cause the one or more processors to execute methods described in more detail herein. For example, as illustrated, memory 320 may include instructions for deep learning module 330 that can be used to implement and / or emulate systems and models and / or to implement any method further described herein. In some examples, deep learning module 330 may receive an input 340, such as a question about a corpus of documents, via data interface 315. Deep learning module 330 may also receive and / or access one or more document corpora for the question. Data interface 315 may be either a user interface that receives questions for a QA dataset or a communication interface that can receive or retrieve previously requested questions stored by another system and / or in a database. Deep learning module 330 may generate an output 350, such as an answer result from a document corpus, based on the question of input 340. The answer result may include one or more documents determined using deep learning module 330 and / or from the documents Message and may include these, which may be ranked, listed, classified, and / or scored based on their relevance to questions determined using a document-level search model and Message a passage-level search model.

[0053] In some embodiments, deep learning module 330 may further include a dense hierarchical search module 331 and a document and Message passage encoder module 332. The dense hierarchical search module 331 and the document and Message passage encoder module 332 may be dense MessageBy using the DHR methodology that considers questions using a dense document-level retrieval model in combination with a level retrieval model, it can be used to provide better results for open-domain questions. Questions for open-domain QA on a document corpus can be received and encoded by a dense hierarchical retrieval module 331 for the document corpus identified to search using that question. The dense document-level retrieval model may use the encoded documents (e.g., based on its summary, ToC, and / or other document structures) that are encoded and indexed by an encoder module 332. The dense hierarchical retrieval module 331 may utilize the encoder module 332 to identify one or more documents having Message from the document corpus. Irrelevant documents are filtered by the dense hierarchical retrieval module 331, and one or more documents for retrieval and / or ranking may be identified. Message using the encoder module 332, Message The dense level retrieval model of the document and

[0054] document and Message the dense Message of the encoder module 332 can be used when encoding Message from the document identified from the document-level retrieval model and the encoded question. The encoded Message and the question may be used by the dense Message level retrieval model to obtain the top-rated Message from the top-rated documents, and these Message are scored or ranked to return to the question from the open-domain QA on the document corpus. The relevance score of the returned document and / or Message may be determined based on a combined relevance score of the document relevance score from the dense document-level retrieval model and the Message relevance score from the dense Message level retrieval model. The dense hierarchical retrieval module 331 and the document and MessageFurther functionality of the encoder module 332 is described in further detail with respect to FIGS. 4-6. In some examples, the deep learning module 330 and sub-modules 331-332 may be implemented using hardware, software, and / or a combination of hardware and software.

[0055] According to some embodiments, the functionality of the computing device 300 including the deep learning module 330 may be implemented or incorporated into a Search Model Service (SMS) plugin. SMS is a gRPC microservice that hosts and executes machine learning models. SMS takes in parameters via a Protobuf file and uses those input parameters to execute the model. SMS returns a response to a client (e.g., a user device) according to defined response parameters. In some embodiments, the SMS implementation is run in an environment for a containerized application (e.g., Salesforce Application Model (SAM)) that is separate or independent from the core application of a multi-tenant database system such as system 116. SMS can provide for faster deployment of the model. SMS also provides for the isolation / containerization of the Java Virtual Machine (JVM), such that crashes associated with its model execution do not affect or are not affected by the application server of the database system. The SMS plugin is separate code that can initialize model data for a particular model type, execute model-specific feature processing, and execute the model based on feature vectors and other parameters. The plugin architecture provides various advantages, including the ability to make changes to the plugin code without making changes to the model executor code, and the alleviation or elimination of concerns regarding load balancing, routing, and parallelization.

[0056] FIG. 4 shows a meaning-based in deep learning according to some embodiments MessageExemplary documents within a document corpus that can be retrieved using hierarchical search, and Message shows a schematic diagram of FIG. 4. Document 400 of FIG. 4 is a text in a document corpus, such as an article, an encyclopedic entry, training materials, customer help requests and responses, and other documents in a database system. Message displays the document structure and structural elements that make up a document having. Thus, document 400 is a document encoding based on one or more document structures, as well as a short-length segmented Message from Message that can be used to generate an encoding.

[0057] Document 400 includes a document title 402, an abstract 404, a table of contents (ToC) 406, a first section 408, and a second section 410. Document 400 may correspond to a document in a document corpus, such as an online encyclopedia or an encyclopedic entry for other searchable databases and platforms. In this regard, document 400 may be regarded as a structured document, where different inputs and data are extracted and used as inputs for a document-level search model and Message a level search model. These models of DHR can be used to determine a combination of relevance scores to better retrieve, score, and / or rank documents and Message for questions presented for open-domain QA. In this regard, for document-level search of documents from a corpus, a dense document-level search model can utilize the encoded documents from the index. The encoded documents may be encoded from the document corpus based on one or more document structures and indexed in an offline environment. Thus, when a question is presented and queried at runtime, the index can be accessed and used together with the encoding of the question for document-level search.

[0058] For document-level search, it is necessary to encode one or more document structures for each document in the document corpus. Encoding may correspond to creating an embedded or vector representation (e.g., having n features, variables, or attributes) that represents a document based on the component parts of the document structure used to encode the document. In this regard, document 400 may include a ToC 406 that corresponds to the document structure used to encode document 400 for document-level search. In some embodiments, document title 402 and / or summary 404 may also be used for document-level search encoding. Document 400 Message for document 400 for Message level search may further include summary 404, first section 408, and second section 410 that may be used when encoding

[0059] ToC 406 includes a list 412 of section and / or subsection titles, which is then extracted using extraction operation 414 and cleaned to generate a hierarchical title list 416. Hierarchical title list 416 may then be used to encode document 400 for a document-level search model and a document-level searcher when a query is queried and encoded. Thus, each document may be considered as a structural set having sections S, subsections Ss, and their corresponding paragraphs. Each section or subsection has a corresponding title. Thus, each document Di may also include a unique nested table of titles [[T S1 [T Ss1 ;...;TS sn ;...;[T Sm [T Ss1 ;...;T Ssl . T D may be considered as unique identification information for distinguishing documents in the corpus.

[0060] Furthermore, for Message performing level search on document 400, summary 404, first section 408, second section 410, and / or other text of document 400 may be of short lengthMessage may be split. In this regard, in-section split Message text (in-section split passage text) 418 can be generated when splitting the abstract 404, the first section 408, and the second section 410 into short lengths Message . This can only be done by splitting the paragraphs under the title of the same section or subsection into limited lengths Message . Each Message may correspond to a sequence of tokens with nested titles. In the corpus C = {P1; P2;,...,; P M}}, there may be a total of M Message , and for each Message P i , the corresponding document to which it belongs can be determined by looking at T D . Therefore, if f: p → d is defined as a function that maps a given Message p to its document d, then g: d → P maps a given document d to its Message P. The Message generated from the abstract 404, the first section 408, and the second section 410 further includes section title information, and during training, in-Doc and / or in-Sec negative training can be used to train the Message level search model. The extraction of the hierarchical title list 416 and / or the in-section split Message text 418 can use the WikiExtractor code to extract the clean text part of the article and remove semi-structured data such as tables, infoboxes, lists, and / or disambiguation pages. However, the hierarchical title list 416 is retained for the document 400. Further, the text under the same section may be concatenated, and each section may be split into a plurality of mutually prime text blocks with a maximum length not exceeding 100 words.

[0061] FIG. 5 shows a meaning-based in deep learning according to some embodiments MessageShows a simplified diagram of exemplary components for hierarchical search. In some embodiments, the dense hierarchical search using the dense document level search model and the dense Message level search model can be implemented as the deep learning module 330 of the computing device 300.

[0062] The component 500 in FIG. 5 can correspond to the system diagram for dense hierarchical search using the document level search model and Message the level search model to obtain meaning-based features and Message from the document corpus for dense hierarchical search. For example, the question 501 can be provided to the open domain QA system and encoded as E Q (q) using the question encoder. To provide a better search for the question 501, a document level searcher 502 may be used, which may correspond to the document level search model used with the encoded document E D (d) (e.g., based on their encoded document structure and elements). The document may correspond to a document within the document corpus 504, which can be searched using document level encoding and / or embedding of the document. The document corpus 504 may correspond to a large set of documents that may include different topic documents. Within the document corpus 504, each document is Message associated with the Message which can be separated from the document corpus 504 for further Message level encoding of the level search model. Message level encoding.

[0063] The top k1 documents 506 are determined based on the document level search model from the encoded documents and the question. Irrelevant documents may be filtered, and one or more top-rated or ranked documents may be returned based on the trained model for the document level searcher 502. Once the top k1 documents 506 are determined, Message the level searcher 508 Message executes the level search model to perform top ranking or rating on those documents Messagecan be determined. However, first, using the document- Message mapping, it can be determined for the top k1 documents 506 the Message that are mapped to the encoded documents retrieved. The mapped Message can be identified as 510 from the top k1 documents. Once the 510 from the top k1 documents are determined, these Message are encoded and Message can be provided to the level searcher 508. Message Message

[0064] The encoding of the query 501 may also be used with the level searcher 508, where Message the level search model encodes the 510 from the top k1 documents to determine the scored Message 512 from the top k1 documents. Message Message When applied to E Message (p) by the model of the level searcher 508, the scored Message 512 from the 510 encoded from the top k1 documents can then be determined as a smaller filtered subset of the documents and corresponding Message P that may be relevant to the query 501. However, since the relevance scores of the top k1 documents 506 determined from the document level search model may be related to the ranking and output of the documents and Message a re-ranking can be performed to combine the relevance scores of the top k1 documents 506 and the scored Message 512 from the top k1 documents. The re-ranking combines the scores based on the combined relevance scores from the document level searcher 502 and Message the level searcher 508 to obtain the re-ranked top k2 Message 514. This is where the DHR is related to the document level search model and Message Message Message ​​​​​​Enable the use of both level search models to search question 501 using document corpus 504.

[0065] Train the models of document-level searcher 502 and Message level searcher 508 to generate encodings of encoded document E D (d) and encoded Message E P (p). A training operation can be performed on DHR to generate the encodings. In some embodiments, the first section of a document in the document corpus may be a description and / or summary of the document that contains information central to the topic within the document, such as an encyclopedia entry. This may include a summary, and the document may further include a ToC that emphasizes sections and subsections within the document. The nested table of contents can be linearized by using commas or special tokens [SEP], such as [[T table =T S1 ,T Ss1 ,...,T Ssl or T table =T S1 [SEP]T Ss1 [SEP]...[SEP]T Ssl to be of the form [[T S1 [T Ss1 ;...;T Ssn ;...;[T Sm [T Ss1 ;...;T Ssl . The final document D may be represented as [CLS]T D [SEP]W D [SEP]T table [SEP].

[0066] Dense document-level retrieval can use a question encoder and a document encoder based on the BERT (Bidirectional Encoder Representations from Transformers) deep neural network (NN) model. BERT corresponds to a language representation deep learning model that enables training of deep bidirectional representations in the NN model layer. The question and the document may be encoded as dense representation vectors, and the relevance score of the document to the question may be calculated by an inner product: Sim(q, d) = <E Q (q), E D (d)>, where q and d may be low-dimensional vectors from the question and document encodings respectively, and <.> may represent the dot product.

[0067] When training the encoder of the dense document-level retrieval model, a QA dataset for the training data may be used. These include Natural Questions (NQ), which have questions mined from actual Google (R) searches and corresponding answers from encyclopedic articles identified by annotators, TriviaQA, which has a set of trivia questions with answers collected from the web, questions selected using the Google (R) Suggest API, WebQuestions, which have questions and answers corresponding to Freebase entities, and / or CuratedTREC (TREC), which has questions from the TREC QA track and various web sources for open-domain QA from unstructured text, including standardized open-domain QA evaluation datasets. When selecting positive Message , question-and-answer pairs may be provided in TREC and TriviaQA. Thus, the top-ranked Message determined using a deep learning model of Best Matching 25 (BM25) that includes the answer may correspond to positive Message . If none of the top 100 retrieved Message contain an answer, the question may be discarded. Additionally, negative documents and Message (e.g., positive Messagealthough it appears as (but does not include the answer), negative sampling and training regarding it may also be used for model training, where Message the ranking of Message may affect the training of a dense model based on training data. This Message may include the use of in-Doc and in-Sec negatives for Message , which Message may be more biased or weighted more heavily based on the proximity of in-Doc and / or in-Sec negatives to the positive Message that includes the answer.

[0068] For example, when training a dataset, in a dataset that contains gold titles (e.g., positive and / or most-matching titles) for a given question, the positive documents can be the documents that have the gold titles. In other datasets, when using BM25, the top 1 document that includes the answer for the entire document text is retrieved as the positive document. Then, for training, three different types of negatives are used. The intro negative may represent each document using the first section, and then BM25 can be used to retrieve the top documents, but the entire document text does not include the answer. The full-text negative may represent each document using the entire document text, and then BM25 can be used to retrieve the top documents, but the entire document text does not include the answer. Additionally, in-batch negatives can be used from Message pairs with other questions that appear in the training dataset.

[0069] Message The level searcher 508 Message may further require the encoding of Message a level search model trained for level search using Message This may further require the encoding of Message . Dense MessageWhen training the encoder of the level search model, a subtitle list (e.g., a list of titles of sections and / or subsections) can be considered together with the document title. Message P is expressed as [CLS] title [SEP] subtitle 1, subtitle 2,..., subtitle n [SEP] Message [SEP] Different E Q (.) can be used in the dense document level search model and the dense Message level search model. Therefore, the relevance score for a question can be calculated by the inner product: Sim(q, d) = <E Message (q), E Q (p)>, where q and p can be low-dimensional vectors from the encoding of the question and D the document, respectively. Message

[0070] Positive and negative Message can be determined for training in a similar way to dense Message passage retrieval (DPR). For example, in a dataset with a gold (e.g., best or top 1) context for a given question, positive Message can be the mapping of Message with the gold context within the set {P}. For other datasets, BM25 can be used to search for the top 1 Message containing the answer. Also, BM25 negatives and in-batch negatives can be used. Furthermore, in-Doc negatives and in-Sec negatives can be used for the retrieved Message to improve the model's ability to find positive Message given a positive document from document level search. An in-Doc negative can be a Message that does not contain the answer within the same document as the positive Message while an In-Sec negative can be another Message that does not contain the answer within the same section as the positive Message Message

[0071] Therefore, during the inference time, the document-level searcher 502 is applied to select the top k1 documents 506. Document- Message mapping is used to select the Message from the top k1 documents, which are scored Message 512, and sent to the Message level searcher 508 to determine the scored Message 512 from the top k1 documents. The scored Message 512 from the top k1 documents are re-ranked using the combination of the relevance scores of the documents and Message to obtain the top k2 re-ranked D 514. Before the inference time, the document encoder E Message is encoded offline from the documents in the document corpus 504. When the query q is given at runtime, the embedding is derived, and the top k1 documents 506 with the embeddings closest to the query q are retrieved. All Message from the top k1 documents 506 are retrieved from the mapping, and P the encoder E Message is applied to all the retrieved Message 512. The scored Message 512 from the top k1 documents, and the ranking or relevance scores from the document-level search and Message level search are used to re-rank

[0072] Therefore, the search rankings and / or relevance scores from both the dense document search and the dense Message search contribute to the final ranking of the top k2 re-ranked Message 514. To do this, the document relevance score is calculated as Sim(q, D j ) + λ * Sim(q, P i ), where P i ∈ D j is calculated by MessageCombined with a relevance score, where λ is a coefficient used between the two scores. The scores may be substantially similar, and thus, λ may be close to or equal to 1. Further, iterative training is applied to train both the document-level search model and Message the level search model. For example, after initial training, re-training using the dataset and positive / negative can be used to further improve the predictive decision-making and document / Message search by the model.

[0073] FIG. 6 shows a simplified diagram of a flowchart for semantic-based Message hierarchical search in deep learning using the document-level searcher and Message the level searcher described in FIGS. 3, 4, and 5 according to some embodiments. One or more of processes 602-614 of method 600 may be implemented in the form of executable code stored in a non-transitory tangible machine-readable medium that, when executed by one or more processors, can cause the one or more processors to execute one or more of processes 602-614. In some embodiments, method 600 can be executed by one or more computing devices within environment 110 of FIGS. 1 and 2.

[0074] The model of the deep learning module 330 uses data analysis, extraction, encoding, transformation, and QA prediction processes to perform a semantic-based Message and / or fine-grained hierarchical search of documents within a database system (e.g., system 116) based on the document-level search model and Message the level search model. In some embodiments, these include documents and corpora of documents (e.g., articles, encyclopedia entries, training materials, customer help requests and responses, and other documents that may be related to a particular database system) that are standard for the database system and may be provided to customers of a CRM or other system.

[0075] To achieve this, referring to FIGS. 4 and 5, method 600 starts from process 602. In process 602, deep learning module 330 receives a question about a document corpus, where the documents in the corpus are associated with respective sets of Message . The document corpus may correspond to document corpus 504 and may include documents similar to document 400, such as information articles, encyclopedic entries, help requests, training manuals, brochures, or other articles on topics providing information. Question 501 may correspond to an input question that is a query for open-domain QA. In process 604, an index of the document corpus and the encoded documents is accessed. For example, a dense document-level retrieval model may be used to generate an encoding of the documents, which may be designated as E D (d) and may include document 400 and / or be from document corpus 504. In this regard, document-level retriever 502 may search for E D (d) after encoding the dense model.

[0076] In process 606, the question is encoded. Question 501 may be encoded as E Q (q), which may be encoded using a question encoder that can be utilized together with the document encoder that generates E D (d) for example for document-level retriever 502. In process 608, a document relevance score for the question is determined using a document-level retrieval model. The document-level retriever may search for the top k1 documents 506 based on the encoding E Q (q) of question 501. The document relevance score may correspond to a scored value, ranked value, or other weighted value for determining the relevance of the top k1 documents 506 to question 501. In this regard, the top k1 documents 506 may be scored and / or ordered based on their encoding E D (d), and the encoding E D(d) can be determined from one or more document structures of document 400 and / or documents from document corpus 504.

[0077] In process 610, irrelevant documents are filtered from the documents based on a document relatedness score. For example, the top k1 documents 506 may correspond to a subset of the filtered documents from document corpus 504 based on their corresponding relatedness scores. Further, the mapping of documents - Message is required to determine E D from (d). Message This enables returning 510 from the top k1 documents. In process 612, (based on the filtered and returned documents and Message ) at least one document among the documents Message is encoded using a Message level search model. Message The Message level search 508 can encode 510 from the top k1 documents to generate the encoded Message E P (p). Message This is possible.

[0078] In process 614, top - ranked Message for the question is obtained. Using EP(p) encoded with EQ(q) (e.g., the encoding of question 501), the scored Message 512 from the top k1 documents can be returned by Message the Message level search 508. This can be determined using the corresponding dense model based on the encoding of the question and Message . However, before only returning the scored Message 512 from the top k1 documents, the relatedness scores from the top k1 documents 506 and the scored Message 512 from the top k1 documents are combined and / or processed to re - rank the top k2 Message514 can be obtained. These top-ranked Message can be provided as the output of question 501 for open-domain QA on the document corpus 504.

[0079] For the above process, one or more neural network models may be trained based on training data. In some embodiments, for training, the neural network may perform preprocessing on the training data for, e.g., each word, part of a word, or character in the training text. The embeddings are encoded, e.g., in one or more encoding layers of the neural network, to generate respective vectors. The preprocessing layer generates an embedding for each word in the text input sequence. Each embedding can be a vector. In some embodiments, these can be word embeddings obtained, e.g., by performing methods such as word2vec, FastText, or GloVe, which define ways to learn word vectors each having useful properties. In some embodiments, pre-trained vectors of a specific dimension may be used. In some embodiments, the embeddings may include partial word embeddings related to parts of words. For example, the word "where" includes the parts "wh", "whe", "her", "ere", and "re". Partial word embeddings help enrich word vectors with subword information / FastText. Similarly, when applying the preprocessing layer to words and / or phrases from the training data, a sequence of word vectors may be generated based on the sequence of words in the document and document structure. In some examples, e.g., the text input sequence used for training may contain a small number of words, in which case the embedding output from the preprocessing layer can be "padded" with, e.g., zeros. The mask layer masks such numbers, such that they are ignored or not processed in subsequent layers, e.g., to help shorten the training time.

[0080] The symbolic layer learns high-level features from the words of the text input sequence. Each symbolic layer generates an encoding (e.g., a vector) that maps the words within the text input sequence into a higher-dimensional space. The encoding can encode the semantic relationships between words. In some embodiments, the symbolic layer or encoder stack is implemented with a recurrent neural network (RNN). The RNN is a deep learning model that processes variable-length vector sequences. This makes the RNN suitable for processing sequences of word vectors. In some embodiments, the symbolic layer can be implemented with one or more gated recurrent units (GRUs). A GRU is a specific model of a recurrent neural network (RNN) and is intended to perform machine learning of tasks using connections through a sequence of nodes. The GRU helps to adjust the input weights of the neural network to solve the vanishing gradient problem, which is a problem common to RNNs. In some embodiments, the symbolic layer can be implemented with one or more long short-term memory (LSTM) encoders.

[0081] Multiple GRUs may be arranged in rows. The first row of GRUs looks at or acts on the information (e.g., embedding or encoding) of each word in the text input sequence in the first (e.g., "forward") direction, and each GRU generates a corresponding state vector and passes that vector to the next GRU in that row (as indicated by an arrow pointing from left to right, for example). The second row of GRUs looks at or acts on the information (e.g., embedding or encoding) of each word in the input sequence in the second (e.g., "reverse") direction, and each GRU generates a corresponding hidden state vector and passes that vector to the next GRU in that row. The weights (values) of the embedding matrix are initialized randomly and / or individually and may be updated / learned using backpropagation during training.

[0082] According to some embodiments, the embedding can be learned end-to-end while training a machine learning engine and / or a neural network model (with other features) on its classification task. As a result of this training, one vector is obtained for each character, word, phrase, or sentence, and the vectors are clustered. For example, two characters, words, phrases, or sentences with similar embeddings will ultimately have similar vectors that are closer to each other than to dissimilar embeddings. Next, the embeddings are flattened by respective flatteners and / or concatenated by respective concatenators.

[0083] The model of the neural network is trained using concatenated features or vectors. For training, the neural network may include or be implemented using a multi-layer or deep neural network or neural model having one or more layers. According to some embodiments, examples of multi-layer neural networks include ResNet-32, DenseNet, PyramidNet, SENet, AWD-LSTM, AWD-QRNN, and / or similar neural networks. The ResNet-32 neural network is described in more detail in "Deep Residual Learning for Image Recognition" by He et al., arXiv:1512.03385, presented on December 10, 2015; the DenseNet neural network is described in more detail in "Densenet: Implementing Efficient Convnet Descriptor Pyramids" by Iandola et al., arXiv:1404.1869, presented on April 7, 2014; the PyramidNet neural network is described in more detail in "Deep Pyramidal Residual Networks" by Han et al., arXiv:1610.02915, presented on October 10, 2016; the SENet neural network is described in more detail in "Squeeze-and-Excitation Networks" by Hu et al., arXiv:1709.01507, presented on September 5, 2017; the AWD-LSTM neural network is described in more detail in "Quasi-Recurrent Neural Networks" by Bradbury et al., arXiv:1611.0157, presented on November 5, 2016, each of which is incorporated herein by reference.

[0084] Each neural network layer can operate or process features or vectors, for example, performing regularization (such as L2 and L1 regularization, early stopping, etc.), normalization, and activation. In some embodiments, each neural network layer may include a dense layer, batch normalization, and dropout for deep learning. In some embodiments, each rectifier linear unit (ReLU) at the end of each layer executes the ReLU activation function. The output layer of the neural network executes the softmax function to create or generate a single model for all contexts. The global model predicts a case object or test case object for the current query to a database system such as system 116. In some embodiments, the model includes or represents a probability distribution of embeddings within a document and / or document structure (either standard or custom) with respect to a given training document and / or document structure (e.g., Message and one or more corpora of documents having the document structure). With respect to the distribution, each embedding has a corresponding numerical value that represents or indicates the relevance of such an embedding to the current search. In some embodiments, to alleviate the softmax bottleneck problem, the softmax layer can be implemented with a high-rank language model called mixture of softmaxes (MOS).

[0085] As described above and further emphasized here, FIGS. 3, 4, 5, and 6 are merely examples of the deep learning module 330 for training and use and the corresponding method 600, which do not unduly limit the claims. Those skilled in the art will recognize many variations, alternatives, and modifications.

[0086] Some examples of computing devices, such as computing device 300, may include a non-transitory tangible machine-readable medium that, when executed by one or more processors (e.g., processor 310), causes the one or more processors to execute the processes of method 600. Some common forms of machine-readable media that may include the processes of method 600 are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tapes, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium to which a processor or computer is adapted to read.

[0087] Exemplary embodiments have been shown and described, but extensive modifications, changes, and substitutions may be contemplated in the foregoing disclosure, and in some instances, some features of an embodiment may be used without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of this application should be limited only by the following claims, and the claims should be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A system for dense hierarchical search in deep learning, comprising: a non-transitory memory storing machine-executable code; and one or more hardware processors coupled to the non-transitory memory, the one or more hardware processors being configured to: receive a query regarding a corpus of documents, wherein the documents within the corpus are each associated with a respective set of messages; access the corpus of documents and an index of encoded documents for the documents, wherein the encoded documents are based on at least one of a summary of the documents or a linearized table of contents of the documents; encode the query with respect to a document-level search model for the documents and a message-level search model for the messages; use the document-level search model to determine a document relevance score for the documents with respect to the query, the document relevance score being based on the encoded query and the encoded documents; use the document relevance score to filter out irrelevant documents from the documents to identify at least one document among the documents; use the message-level search model for the messages to encode the messages within the at least one document among the documents, the step of encoding the messages further using at least one document structure of the at least one document among the documents; use the message-level search model, and using the document relevance score and the encoded messages, to obtain top-rated messages for the query from the at least one document among the documents, the step of obtaining the top-rated messages comprising: determining a message relevance score for the query using the message-level search model and the encoded messages; determining a set of filtered messages from the documents using a combined relevance score of the document relevance score and the message relevance score; and one or more hardware processors configured to execute the machine-executable code to perform the operations including the steps above; A system comprising the above.

2. The message is extended before encoding the message using the at least one document structure, and the at least one document structure includes at least one of a summary, a table of contents, one or more section titles, one or more sub-section titles, or a list of one or more paragraph titles of a corresponding one of the documents in the documents. The system according to claim 1.

3. The step of determining the document relevance score is further based on the ranking of the message for training data provided during training of the document level search model. The system according to claim 1.

4. Before receiving the question, the machine-executable code causes the one or more hardware processors to determine at least one of the summary or the linearized table of contents of the document in the corpus; encode the document using a first deep learning model based on the at least one of the summary or the linearized table of contents; perform an offline indexing of the encoded documents in the index; The system according to claim 1, further causing the operations to be performed.

5. The machine-executable code causes the one or more hardware processors to search for a set of top messages for training questions using a second deep learning model; train the message level search model using a negative sampling operation on document-level negative samples and section-level negative samples from the set of retrieved top messages for the training questions; The system according to claim 4, further causing the operations to be performed.

6. The negative sampling operation applies a bias weighted based on the proximity of the negative message to the positive message in the corresponding document or corresponding section from the set of retrieved top messages. The system according to claim 5.

7. A method performed by a system for dense hierarchical search in deep learning, the method comprising: receiving a question about a corpus of documents, wherein the documents in the corpus are each associated with a respective set of messages; Accessing the corpus of the document and an index of the encoded document for the document, wherein the encoded document is based on at least one of a summary of the document or a linearized table of contents of the document; Encoding the question for the document-level retrieval model of the document and the passage-level retrieval model of the passage; Determining a document relevance score of the document for the question using the document-level retrieval model, wherein the document relevance score is based on the encoded question and the encoded document; Filtering out irrelevant documents from the documents using the document relevance score to identify at least one document among the documents; Encoding the passages in the at least one document among the documents using the passage-level retrieval model of the passage, wherein the step of encoding the passages further uses at least one document structure of the at least one document among the documents; Using the passage-level retrieval model, and using the document relevance score and the encoded passages, to obtain top-rated passages for the question from the at least one document among the documents, wherein the step of obtaining the top-rated passages: Determining a passage relevance score for the question using the passage-level retrieval model and the encoded passages; Determining a set of filtered passages from the documents using a combined relevance score from the document relevance score and the passage relevance score; Steps including; A method including. Claim 8 The passage is expanded before encoding the passage using the at least one document structure, and the at least one document structure includes at least one of a summary, a table of contents, one or more section titles, one or more sub-section titles, or a list of one or more paragraph titles of a corresponding one of the documents among the documents; The method according to claim 7. Claim 9 The step of determining the document relevance score is further based on the ranking of the messages for the training data provided during the training of the document-level retrieval model. The method according to claim 7.

10. Before receiving the question, the method determining at least one of the summary or the linearized table of contents of the document in the corpus; encoding the document using a first deep learning model based on at least one of the summary or the linearized table of contents; performing offline indexing of the encoded documents in the index; The method according to claim 7, further comprising.

11. searching for a set of top messages for the training question using a second deep learning model; training the message-level retrieval model using a negative sampling operation on the in-document negative samples and in-section negative samples from the set of retrieved top messages for the training question; The method according to claim 10, further comprising.

12. The negative sampling operation applies a weighted bias based on the proximity of the negative message to the positive message in the corresponding document or corresponding section from the set of retrieved top messages. The method according to claim 11.

13. A non-transitory machine-readable medium storing instructions configurable to perform a method of dense hierarchical retrieval in deep learning, the instructions causing a machine to receive a question about a corpus of documents, wherein the documents in the corpus are associated with respective sets of messages; access the corpus of documents and an index of encoded documents for the documents, wherein the encoded documents are based on at least one of a summary of the documents or a linearized table of contents of the documents; encoding the question for the document-level retrieval model of the document and the message-level retrieval model of the messages; Determining a document relevance score for the document with respect to the question using the document-level search model, wherein the document relevance score is based on the encoded question and the encoded document; Filtering out irrelevant documents from the documents using the document relevance score to identify at least one of the documents; Encoding the message within at least one of the documents using a message-level search model for the message, wherein the step of encoding the message further uses at least one document structure of the at least one of the documents; Using the message-level search model and using the document relevance score and the encoded message to obtain a highly-rated message for the question from the at least one of the documents, wherein the step of obtaining the highly-rated message: Determining a message relevance score for the question using the message-level search model and the encoded message; Determining a set of filtered messages from the documents using a combined relevance score from the document relevance score and the message relevance score; Including the steps; A non-transitory machine-readable medium including machine-executable code for causing a machine to perform operations including the steps.

Images