System for query processing using neural network based language model

Abstract

Disclosed is a method of processing a query using a trained artificial neural network (ANN) implementing a large language model (LLM). A database of documents is maintained for use in processing queries, the database defining multiple repositories, each containing one or more documents. The documents in the database are processed to generate, for each repository, a vector store comprising vector embeddings encoding information obtained from documents of the repository. A user inputs a query string and a selection of one or more of the repositories to be used to process the query. A query embedding corresponding to the query string is generated and the vector stores corresponding to each selected repository are searched using the query embedding to identify one or more vectors that are relevant to the query. An LLM query is formulated to include the query string, a query context comprising information determined based on the identified relevant vectors and a predefined LLM prompt. The LLM is invoked with the LLM query as input whereby the LLM query is processed using the ANN to generate an LLM output. A query response is provided to the user based on query response data received from the LLM.

Description

FIELD OF THE INVENTION The present application relates to methods and systems for processing a query using a trained artificial neural network (ANN) implementing a large language model (LLM). BACKGROUND OF THE INVENTION Advances in artificial neural networks have enabled the creation of sophisticated large language models that can respond to natural language queries. Some LLMs, such as OpenAI’s ChatGPT, are made available for use by end users and / or for integration into third-party systems via APIs. Such LLMs generally answer queries based on a large corpus of information on which the LLM was trained. However, this makes it difficult to know what information is being used to answer queries or to ensure that only relevant information is used. Prompt engineering can address this to some extent but does not fully overcome the problem. Furthermore, users may wish to use the power of the LLMs to answer queries based on information not included in the training corpus, e.g. proprietary information. While some LLMs can be used to answer queries about specific information sources (e.g. about a particular document) on an ad hoc basis, it would be advantageous to be able to integrate pre-trained LLMs with a variety of data sources (including proprietary data sources) in a more structured, flexible and efficient manner. SUMMARY OF THE INVENTION Aspects of the invention are set out in the independent claims. Certain preferred features are set out in the dependent claims. Disclosed herein is a method of processing a query using a trained artificial neural network (ANN) implementing a large language model (LLM), comprising: maintaining a database of documents for use in processing queries, the database defining a plurality of repositories, each containing one or more documents; processing the documents in the database to generate, for each repository, a vector store comprising vector embeddings encoding information obtained from documents of the repository; receiving from a user, via a query interface: a query string for a query; and a selection of one or more of the plurality of repositories to be used to process the query; generating a query embedding corresponding to the query string; searching the vector stores corresponding to each selected repository using the query embedding to identify one or more vectors that are relevant to the query, wherein the relevant vectors are identified based on comparison of the query embedding to the vectors in the vector stores; formulating an LLM query comprising: the query string; a query context comprising information determined based on the identified relevant vectors; and a predefined LLM prompt; invoking the LLM with the LLM query as input whereby the LLM query is processed using the ANN to generate an LLM output; receiving query response data from the LLM based on the LLM output; and outputting a query response to the user based on the query response data. The vector embeddings in the vector stores preferably comprise embeddings in a semantic vector space (also referred to as a latent space) defined such that a distance between two vectors in the vector space is indicative of the semantic similarity or dissimilarity of the information represented by the vectors. Searching the vector stores correspondingto the selected repositories preferably comprises: determining distances between the query embedding and vectors in the vector stores, and selecting one or more vectors based on the distances. This may include selecting the one or more relevant vectors in dependence on comparing the distances to a threshold and / or selecting one or more vectors having a lowest distance to the query embedding. Preferably, searching the vector stores comprises: for each selected repository, searching the vector store corresponding to the repository using the query embedding to identify a set of vectors relevant to the query from that vector store (e.g. based on vector distances as noted above); and merging the sets of identified relevant vectors from each vector store to produce the relevant vectors. Merging the sets of identified relevant vectors may comprise selecting a most relevant set of vectors from across the sets of vectors identified from respective vector stores, the most relevant set preferably comprising vectors having lowest distances to the query embedding (e.g. the n lowest-distance vectors may be used and / or any vectors with distances below some threshold). Thus, merging may or may not retain all vectors identified from the respective vector stores. In an embodiment, the searching may comprise: creating a Langchain knowledge retriever for each selected vector store to retrieve results relevant to the query from the vector store, and invoking a Langchain merger retriever to obtain a merged set of results from the selected vector stores using the knowledge retrievers. The method may comprise creating the query context by a process that includes: retrieving document content of documents in the database that corresponds to the identified relevant vectors; and including the retrieved information in the query context. The retrieved information preferably comprises text chunks corresponding to the identified relevant vectors. Processing the documents in the database to generate a vector store for a repository preferably comprises generating vector embeddings for each of a plurality of text chunks extracted from documents of the repository. The prompt preferably includes an instruction to the LLM to generate the query response based only on information in the context and excluding other information available to the LLM or on which the LLM was trained. A plurality of document repositories are preferably maintained in the database, each corresponding to a respective document topic and / or associated with a respective user or user group. The method may comprise restricting the selection of one or more repositories by the user in dependence on access rights defined for the user and / or the repositories. Restricting selection may comprise allowing a user to select only repositories for which access rights have been granted to the user. Preferably, the LLM is accessed with content moderation and / or abuse monitoring functions disabled. While described examples use a neural network based LLM other forms of language model (preferably machine learning models) may be used. Also disclosed are a system having means, optionally comprising a computer system or device having a processor with associated memory, for performing any method as set out herein and a computer program or computer readable medium comprising software code adapted, when executed by a data processing system, to perform any method as set out herein. Features of one aspect or example may be applied to other aspects or examples, in any combination. For example, method features may be applied to system or computer program aspects or examples (and vice versa). BRIEF DESCRIPTION OF THE FIGURES Certain embodiments of the invention will now be described by way of example only, in relation to the Figures, wherein: Figure 1 illustrates a system for query processing using an ANN-based LLM; Figure 2 illustrates processing of a query in overview; Figure 3 illustrates retrieval of information from a modular knowledge based for use in processing a query; Figure 4 illustrates processing of a query in detail; and Figure 5 illustrates a computer device for implementing the described query processing system. DETAILED DESCRIPTION Embodiments of the invention provide a system for utilizing a trained artificial neural network (ANN) implementing a large language model (LLM) to generate query responses from documents in a modular document database. A system 100 for implementing the described techniques is illustrated in Figure 1. The system includes a client device 102 accessing a knowledge base application via a backend system 106. The application is preferably implemented as a web application accessed using a browser 104 of the client to provide the application front end. The knowledge base application maintains a modular knowledge base 107 including a document store 108 storing a set of documents that may be queried via the knowledge base application. The backend system 106 extracts information from the documents in the document store 108 and stores the extracted information in vectorized from in a vector database 110 to support queries. The backend system 106 receives a query from a user via the client device 102 and the application front end. The query specifies both a query string and identifies the parts of the knowledge base to be used to answer the query. Based on this information, the backend system formulates a query for submission to the LLM platform 112. The document store comprises a number of distinct document repositories each comprising a group of documents. Each repository together with its associated information in the vector database 110 can be considered as a distinct module of the modular knowledge base 107 that can be independently managed and queried. Document repositories may for example, be defined by document topic, document owner and in various other ways. In an example implementation, the backend system includes a number of components, including: • API 120 - used for general processing (submitting raw documents for storage in the document store, listing available repositories / vector stores, mediation between the front end and other services) • Document Import service 122 - used for reading documents from the document store 108, generating vector embeddings of the information in the documents and storing the resulting vectors in the vector database 110 • Knowledge Service 124 - Used when retrieving information from the vector database 110 • Identity Service 126 - Used for managing user accounts (e.g. creating new accounts, setting access permissions, configuring / changing user credentials such as passwords etc.) The backend system may be implemented on a single server or with the above components distributed across multiple servers or deployed to a cloud platform. The query process is shown in overview in Figure 2. In step 202, a user query is received. The user query specifies which of the available document repositories are to be used to answer the query. In step 204, the backend system searches the stored information in the vector database 110 that corresponds to the selected document repositories to identify the information that appears to be most relevant to the query. This search is based on semantic matching of the query string against the vectorised information in the vector database. The identified information is then used to form a query context which is submitted in step 206 to a pre-trained LLM provided by LLM platform 112 along with a predefined prompt string for the LLM. The LLM generates a query response based on the provided prompt and the context and the response is received by the backend system and output to the client in step 208. The processing of a query will now be described in more detail with reference to Figures 3 and 4, with Figure 3 illustrating software modules and data elements involved in processing of queries and Figure 4 illustrating the processing steps performed (divided into steps occurring respectively at the client device and backend system). As shown in Figure 3, the modular knowledge base 107 includes the document store 108 storing a variety of documents. The term “document” may refer to any file or other data unit comprising information that a user may wish to query. For example, documents could be word processing files, PDF (Portable Document Format) documents, presentations, text files, spreadsheets etc. The document store 108 is divided into a number of document repositories 310, 312, 314. Each repository may correspond to a particular topic or subject and / or may be associated with a particular user, user group or organisation. In one example, the document store may store documents to support specialist organisations operating in a particular field of business or other endeavour. The document store may include document repositories for various topics relevant to the subject area in question. This may include one or more repositories provided by an operator of the knowledge base (e.g. an information management service provider) as well as further repositories created in the system by users and organisations using the knowledge base service. All service users may have access to the operator-provided repositories, whilst user- or organisation-specific repositories may only be accessible to the users / organisations who created and / or own those repositories. For example, in a knowledge base for teachers, the operator may include standard documents such as curricula, lesson plans and past exam papers in one or more fixed data repositories (e.g. one per school subject area), while individual schools or teachers using the service may create separate document repositories storing their own materials, lesson plans etc. Typically, service users may modify the contents of their own repositories, e.g. by adding or deleting documents, but may not modify the contents of other users’ repositories or the provider-managed repositories. While Figure 3 illustrates distinct document repositories 310, 312, 314, this distinction may be a logical rather than physical distinction, and the documents may be stored in a single repository with metadata identifying document owners and individual repositories to which documents belong. In one implementation, the document store is implemented using blob storage, separated at the container level into tenants, with documents grouped by user organisations. The system keeps track of the documents that are uploaded to blob storage in an SQL database, which also records the document repository to which each document belongs. In an offline step (i.e. performed independently of query processing), data from documents in each document repository is extracted and converted into a semantic vector representation. Specifically, text from the documents is split into information units or “chunks” (e.g. a chunk may be a sentence or series of sentences, and there may be a degree of overlap between chunks). Each chunk is mapped to a semantic embedding in a vector space defined such that a distance between two vectors in the vector space is indicative of the semantic similarity or dissimilarity between the chunks represented by the vectors. In one implementation, the embedding used is text-embedding-ada-002 (details are available at: https: / / openai.com / blog / new-and-improved-embedding-model). However, any suitable embedding may be used. Thus, each document repository produces a set of vectors encoding information from the documents in the repository (typically multiple vectors per document for each of the multiple documents in the repository), each vector representing a text chunk of a source document. These are notionally grouped here as a “vector store” (320, 322, 324) in the vector database 110 for each repository. The vector store links vectors back to the source documents and repositories allowing different repositories to be searched individually and selectively and allowing search results to be linked back to the source documents. Together, each document repository and associated vector store effectively forms a separate knowledge base module of the knowledge base 107. With reference to Figures 3 and 4, processing of a query starts with the user accessing the knowledge base application and selecting (step 402) one or more of the available knowledge base modules, which may include modules made available to all users by the service operator and / or modules corresponding to data repositories maintained by the service user. The list of selected modules is shown as element 303 in Figure 3. The selection is made via a suitable user interface in which the user can select or deselect modules to be used in the query (e.g. via buttons, check boxes etc.) Access to knowledge base modules is limited by user permissions. Thus, after a user logs into the system using relevant user credentials, only the knowledge base modules available to that specific user and / ortheir organisation are made available for selection. Those may include modules created by that organisation and / or modules provided by the service provider and for which access has been granted to the user or their organisation by the service provider. In general, access rights may be defined for repositories and / or users, determining which repositories a user has access to and may use for processing of queries. Access rights may be user-specific or defined for a user’s organisation. Aside from limiting users to access only their own organisation’s repositories (in addition to any platform operator’s repositories being made available) this approach can allow one organisation to grant (or deny) access to their repositories to another organisation or its users (individually or collectively). Rights for adding, modifying or deleting documents in a repository may be defined separately from query rights, allowing an organisation to maintain document repositories on the platform that can be included in outside users’ queries (but without allowing those users to modify the contents of the repositories). In step 404, the user inputs a query string 302. The client front-end formulates a request containing the query string and identifiers of the knowledge base modules to be queried, which is then transmitted to the backend system (406). The request is received at the backend system in step 408 where the selected knowledge base modules are mapped to the vector store identifiers of the corresponding vector stores in the vector database. An API endpoint is invoked with a payload containing the query string and the list of one or more vector store identifiers identifying the particular vector stores to be searched. In the Figure 3 example, three vector stores 320, 322, 324, corresponding respectively to three document repositories 310, 312, 314 have been selected, but any number of vector stores (including a single store) could be selected. The vector stores are used by a context selector module 304 to obtain context information for the LLM module. In an embodiment, the backend knowledge service is implemented using the LangChain LLM application framework. Langchain abstracts the creation and searching of vector information away from a specific implementation of vector stores into a "VectorStore" base class. To invoke the methods for retrieving information from the specific implementation of the vector store (in a particular implementation, PGVector may be used, see https: / / python.langchain.com / docs / integrations / vectorstores / pgvector) an instance object of that class is created. Thus, in step 410 a LangChain vector store is instantiated for each vector store ID within the payload and the vector stores are converted to LangChain knowledge retrievers 330, 332, 334. Each retriever identifies information from its associated vector store that is relevant to the query based on semantic matching of an embedding of the query string to the embeddings stored in the vector store. Specifically, this involves generatingthe vector embeddingof the query string, searching the vector store based on the query embedding to find similar embeddings in accordance with a semantic matching criterion and identifying the text chunks correspondingto the most similar embeddings. The semantic matching criterion maybe based on a similarity metric indicating the semantic similarity / dissimilarity between two embeddings, for example, in the form of a distance measure between vectors (e.g. vector difference / Euclidean distance). In an example, the search may identify embeddings in the vector store for which the distance measure between the query embedding and the vector store embedding is below some threshold and / or may identify the n embeddings from the vector store with lowest distance to the query embedding. Those embeddings are then considered to be semantic matches for the query. In step 412, the LangChain Merger Retriever 336 is then run using the generated retrievers 330, 332, 334. This results in the individual retrievers being run to retrieve the most relevant results from their respective vector stores. The merger retriever 336 combines all vector store results into the most relevant set of results across all vector stores included in the search. For example, the m lowest distance vectors across the results from all vector stores may be selected as the final set of query-relevant vectors. The results from the Merger Retriever comprise the chunks of the original source documents and may include chunks from multiple documents. Each chunk has associated metadata that points back to the original document in the document store should the user wish to access it. In step 414, an LLM query is formulated. This includes the query string 302 and a query context 340. The query context is formulated to include information based on the results of the merger retriever 336. Specifically, for each of the selected vectors identified by the merger retriever, the corresponding text chunk encoded by that vector is obtained. For example, these may be retrieved from the source documents based on references stored in the vector store, or the text chunks may have been cached in (or alongside) the vector stores to provide immediate access. The text chunks corresponding to the selected query-relevant vectors are then combined to form the query context 340, e.g. by string concatenation of the chunks. The query further includes a predefined prompt 342. In an example, the predefined prompt may take the following form: “You are MyRTO, a helpful Al assistant for the education industry created by RTO Materials. You provide answers to questions in the style of the provided context. You MUST answer questions based on the context. You CANNOT provide additional detail to the answers using your own knowledge. If you do not know the answer from the context, respond starting with "I do not know the answer." All answers must be factually correct, DO NOT make up answers. ” Thus, the prompt instructs the LLM to use only the information in the context (and not any other information it may have acquired during training of the LLM or may have access to in some other way) to answer the query. In step 416, the query comprising the query string 302, context 340 and prompt 342 are submitted to the LLM. In an example, ChatGPT-3.5-Turbo provided via the OpenAI service is used as the LLM. However, any suitable trained LLM may be used. In preferred embodiments, to improve data security and privacy, the OpenAI service is invoked with content moderation turned off so that none of the data (context and query) are stored / screened by the OpenAI service. User data is only sent to the LLM as part of the query context. Furthermore, in an example implementation, Azure OpenAI us used as a layer in front of the LLM which offers a more private deployment of the OpenAI GPT-3.5-Turbo model. Azure OpenAI may store some data for abuse monitoring so preferred implementations may opt out of that functionality. These measures can improve privacy by preventing data leakage, given that the user-specific document stores may include confidential documents. In step 418, the backend system receives a query response from the LLM, formats this as necessary, and sends the response to the client. In step 420, the client receives the query response and displays it to the user. Processing device Figure 5 illustrates a processing device 500 suitable for implementing processing elements of the system, such as the backend system 106 of Figure 1. The processing device 500 may be based on conventional workstation or server hardware and as such includes one or more processors 508 together with a main memory 502 (e.g. volatile / random access memory) for storing temporary data and software code being executed. An input / output subsystem 506 includes one or more I / O interfaces for communicating with external devices and peripherals, such as displays, input devices (e.g. keyboard, mouse), external storage devices and the like. A network interface 510 is provided for communication with external systems via network 120 (encompassing e.g. Local and / or Wide Area Networks, including private networks and / or public networks such as the Internet, cellular telephony networks etc.) For example, the processing device 500 may communicate with client devices 102 and the LLM platform 112 (e.g. OpenAI) via the network. Persistent storage 504 (e.g. in the form of hard disk storage, optical storage and the like) persistently stores software and data for performing various described functions (e.g. for performing server-side steps 408-418 of the Figure 4 process). The persistent storage further includes a computer operating system and any other software and data needed for operating the processing device. The device may include other conventional hardware components as known to those skilled in the art. The various components are interconnected by one or more data buses 512 (e.g. system / memory bus and one or more I / O buses). While a specific architecture is shown and described by way of example, any appropriate hardware / software architecture may be employed to implement the backed system. Furthermore, functional components indicated as separate may be combined and vice versa. The backend functions may be performed by a single device or may be distributed across multiple devices (e.g. in a server cluster). It will be understood that the present invention has been described above purely by way of example, and modification of detail can be made within the scope of the invention.

Claims

1. A method of processing a query using a trained artificial neural network (ANN) implementing a large language model (LLM), comprising:maintaining a database of documents for use in processing queries, the database defining a plurality of repositories, each containing one or more documents;processing the documents in the database to generate, for each repository, a vector store comprising vector embeddings encoding information obtained from documents of the repository;receiving from a user, via a query interface:a query string for a query; anda selection of one or more of the plurality of repositories to be used to process the query;generating a query embedding corresponding to the query string;searching the vector stores corresponding to each selected repository using the query embedding to identify one or more vectors that are relevant to the query, wherein the relevant vectors are identified based on comparison of the query embedding to the vectors in the vector stores;formulating an LLM query comprising:the query string;a query context comprising information determined based on theidentified relevant vectors; anda predefined LLM prompt;invoking the LLM with the LLM query as input whereby the LLM query is processed using the ANN to generate an LLM output;receiving query response data from the LLM based on the LLM output; and outputting a query response to the user based on the query response data.

2. A method according to claim 1, wherein the vector embeddings in the vector stores comprise embeddings in a semantic vector space defined such that a distance between two vectors in the vector space is indicative of the semantic similarity or dissimilarity of the information represented by the vectors.

3. A method according to claim 2, wherein searching the vector stores corresponding to the selected repositories comprises:determining distances between the query embedding and vectors in the vector stores, andselecting one or more vectors based on the distances.

4. A method according to claim 3, comprising selecting the one or more relevant vectors in dependence on comparing the distances to a threshold and / or selecting one or more vectors having a lowest distance to the query embedding.

5. A method according to any of the preceding claims, wherein searching the vector stores comprises:for each selected repository, searching the vector store corresponding to the repository using the query embedding to identify a set of vectors relevant to the query from that vector store; andmerging the sets of identified relevant vectors from each vector store to produce the relevant vectors.

6. A method according to claim 5, wherein merging the sets of identified relevant vectors comprises selecting a most relevant set of vectors from across the sets of vectors identified from respective vector stores, the most relevant set preferably comprising vectors having a lowest distance to the query embedding.

7. A method according to claim 6 or 7, wherein the searching comprises: creating a Langchain knowledge retriever for each selected vector store to retrieve results relevant to the query from the vector store, andinvoking a Langchain merger retriever to obtain a merged set of results from the selected vector stores using the knowledge retrievers.

8. A method according to any of the preceding claims, comprising creating the query context by:retrieving document content of documents in the database that corresponds to the identified relevant vectors; andincluding the retrieved information in the query context.

9. A method according to claim 8, wherein the retrieved information comprises text chunks corresponding to the identified relevant vectors.

10. A method according to any of the preceding claims, wherein processing the documents in the database to generate a vector store for a repository comprises generating vector embeddings for each of a plurality of text chunks extracted from documents of the repository.

11. A method according to any of the preceding claims, wherein the prompt includes an instruction to the LLM to generate the query response based only on information in the context and excluding other information available to the LLM or on which the LLM was trained.

12. A method according to any of the preceding claims, wherein a plurality of document repositories are maintained in the database each corresponding to a respective document topic and / or associated with a respective user or user group.

13. A method according to any of the preceding claims, comprising restricting the selection of one or more repositories by the user in dependence on access rights defined for the user and / or the repositories.

14. A method according to claim 13, wherein restricting selection comprises allowing a user to select only repositories for which access rights have been granted to the user.

15. A method according to any of the preceding claims, wherein the LLM is accessed with content moderation and / or abuse monitoring functions disabled.

16. A system having means, optionally comprising a computer device having a processor with associated memory, for performing a method according to any of the preceding claims.

17. A computer program or computer readable medium comprising software code adapted, when executed by a data processing system, to perform a method as set out in any of claims 1 to 15.

Images