Graph-based memory extension for large language models

Abstract

To provide assistance systems and methods with memory engines that store and retrieve information described and accessed as natural language text.SOLUTION: This method comprises: obtaining and processing, by a natural language processing module, instructions in natural language; generating chunks of text based on the obtained instructions; generating, by a text embedding module (TE), vectors based on the chunks of text generated from the obtained instructions; storing, by a vector embedding memory (VEM), the generated vectors in association with the generated chunks of text; searching the vector embedding memory to determine nodes corresponding to matching vectors; activating corresponding nodes stored in a memory (GBM); selecting the activated nodes for further processing; determining chunks of text corresponding to the activated nodes; generating a response to a query based on the determined chunks of text; and outputting the response.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure is in the field of assistance systems and methods using a memory engine that stores and retrieves information described and accessed in natural language. In particular, techniques are disclosed for providing the ability to generate responses to queries in a dynamic conversational manner. [Background technology]

[0002] General purpose memory engines are used in systems that store, retrieve, and use information that may be described and accessed in natural language, e.g., assistance systems. Current implementations of such systems often use deep neural networks (DNNs), specifically large-scale language models (LLMs), i.e. DNNs trained on large corpora of natural language (NL) text. LLMs have a large amount of knowledge of general facts and common sense statements available from initial training on sources of natural language text. The strength of LLMs lies in their natural language user interface and their ability to incorporate human common sense to resolve language-based ambiguities and fill in the gaps that may come with poorly constrained queries from users.

[0003] However, current LLMs have several problems. They exhibit deficiencies in terms of accuracy, reliability, transparency, privacy, and flexibility of the information used within them and output by them in response to user-provided queries. They exhibit weaknesses, especially in terms of managing additional data, such as personal and proprietary data. Because LLMs' processing of natural language-based texts uses a kind of statistically inferred likelihood-based imputation, LLMs do not always work for critical applications.

[0004] Specialized extensions of LLM are currently being used for use cases that involve dynamically generating additional data, e.g., by measurements in the real world, for example, in augmenting the agent's memory.

[0005] One available extension uses the LLM as an interface to connect a backend database that contains additional data, e.g., sensitive information, in a conventional memory structure. Using a conventional backend database in combination with the LLM combines the strengths of the LLM, including natural language user interface, language-based ambiguity resolution, and incorporation of human common sense to fill gaps inherent in queries, with the strengths of a more formal, schema-based database. The backend database has the advantages of high reliability, traceability, and unique access to the sensitive information. Using the LLM as an interface to a backend database as well, e.g., to generate queries in a language specific to the backend database containing the additional data, can introduce bottlenecks due to limited pre-built queries or due to limitations in query generation.

[0006] In an alternative approach that extends LLM to address at least some of the deficiencies of current LLMs, a mixed structure based on a less canonical database uses another natural language oriented memory storage system that includes linguistic embeddings of text chunks. This approach uses linguistic embeddings to find a pre-selection of relevant text chunks (fragments) based on a determined similarity between the query and the chunks of text in the embedding space. The pre-selection provides identifiable chunks of text that contain text that is semantically similar to the query and thus deemed plausible for answering the query. The (pre-)selected chunks of text together with the initial query are passed to the LLM to generate an answer to the query. This known mixed structure approach has the problem of basing the pre-selection of plausible text chunks solely on similarity of embeddings. Other memory items that may be relevant to the query but do not contain semantically similar words, for example because they contain supplementary information, are not considered to generate a response.

[0007] It is desirable to continuously and incrementally improve the growing proprietary memory without requiring costly retraining and fine-tuning of typical LLMs. Summary of the Invention [Problem to be solved by the invention]

[0008] In view of the above considerations, there is a need to address the problem of improving natural language based information retrieval and reasoning from memories containing unique relational information. [Means for solving the problem]

[0009] The present computer-implemented method and corresponding system provide an advantageous solution to this and similar problems. The dependent claims define further advantageous embodiments.

[0010] A computer-implemented method generates a response to a query from a user or agent in a system by an agent. The system includes a vector embedding memory configured to store vectors (embedding vectors) and chunks of natural language text, a graph-based memory configured to store information in the form of nodes interconnected by links, and a processor. The method includes searching for matching vectors from the vector embedding memory based on semantic closeness between the vectors stored in the vector embedding memory and the vectors generated based on the chunks of natural language text of the retrieved query, determining and activating nodes stored in the graph-based memory based on their correspondence with the matching vectors, and determining and activating connected nodes based on links between the activated nodes. The method further includes generating a response to the query based on the chunks of text determined based on the semantic closeness between the matching vectors from the vector embedding memory and the vectors generated based on the chunks of natural language text of the query. The generating response generates a response to the query based further on the determined chunks of text corresponding to the activated nodes from the graph-based memory.

[0011] The computer-implemented method according to the first aspect has several advantages: in particular, the method provides the ability to make additional information available to the question answering system, giving the question answering system full access to unique relevant information without the need to extensively retrain the question answering system.

[0012] Embedded systems known in the art consider each memory item as independent of each other memory item, thus constituting a traditional indexed memory. The present computer-implemented method overcomes this by providing a memory that relates to real-world memory items using a graph structure that provides a web of interconnected concepts. In an example inspired by human memory, a particular event such as a concert may be associated with a date through the date of the concert, a place through the location of the concert, a mood, an experience, musical preferences, terrible weather, to the friends that the person went with to attend the concert. All these elements and other similar elements are associated with each other in the memory, but each element represents a semantically distinct and seemingly unrelated memory item. Similarly, a note in an individual's knowledge graph may be linked to various associated nodes, and those associated notes provide background information related to the note. As a result, the present computer-implemented method provides a unique advantage over existing indexed memories in that the quality of the searched memory items, which are in the form of chunks of text, that are later responsible for generating responses to queries, is improved by taking into account the associative relationships between memory items and chunks of text. Instead of an index memory as in known approaches, the computer-implemented method uses an extensible knowledge graph over the sensitive memory to generate responses to queries, and follows relevant relationships in the graph to search for further relevant information.

[0013] Current state-of-the-art methods for extending LLMs through additional training or fine-tuning essentially incorporate implicit information about the knowledge network that is static (fixed) and predetermined solely by the text and language statistics used when training the LLM. Currently, it is prohibitively expensive to retrain or fine-tune an LLM with unique information for custom purposes, so interfacing with external memory is currently a viable approach.

[0014] The computer-implemented method according to the first aspect provides a solution that overcomes the limitations of known approaches. Unique external information does not simply consist of isolated pieces of information and chunks of text. The relationships that connect the pieces of information capture a high percentage of the unique external information. Returning to the example of a person's personal memory of a concert that they went to with friends, the entire description of this event, the people who went, their relationships, etc. may be contained in a single chunk of text. If this chunk is converted into an embedding vector, this information is likely to be found for queries like "Who did X go to event Y with?". In real-world applications, the relevant information may be expected to be distributed across chunks of text including, for example, (i) friends and their relationships, (ii) concerts, performers, locations, dates, etc., (iii) plans to go to a concert on that particular day, and (iv) plans to spend time with that friend on that particular day, to name a few. By using a graph-based memory in addition to a vector embedding memory to connect all these pieces of information, it becomes possible to infer that if you are going to a concert on that day and you have information that you are planning to spend time with someone on that day, and that someone is a friend, then it must be that friend that you are going to the concert with. The web of links and nodes in the graph-based memory, covering the relationships between individual pieces of information, makes it possible to infer this conclusion, benefiting from the additional information contained in the graph-based representation that is stored in and retrieved from the graph-based memory and that is not contained in a single piece of information or chunk of text.

[0015] A unique memory based current index may be sufficient when the granularity of the chunk of text matches the granularity of the query. This effectively means that most of the information required to answer the query is contained in a single chunk of text. When a query requires gathering information that is distributed across different chunks of text, not all of which have embedding vectors to the embedding of the query, the graph based representation of the computer implemented method is advantageous in providing a more satisfying response by basing the information retrieval process on the representation.

[0016] A computer-implemented method according to one embodiment includes augmenting the generated response with reference to the chunk of text used as the basis for generating the response to allow for full traceability.

[0017] An embodiment defined by the dependent claims includes a computer implemented method for generating a response to a query from a user or agent according to the first aspect, the method comprising the steps of: obtaining and processing natural language instructions from a user to generate chunks of text by a natural language processing module; generating vectors based on the chunks of text by a text embedding module; storing, by a vector embedding memory, the vectors generated by the text embedding module in association with the generated chunks of text; searching, by a processor, the vector embedding memory to determine matching vectors based on similarity to the vector of the obtained query from the user or agent; and searching, by a vector search module, the stored vectors in the vector embedding memory and the vectors generated by the text embedding module based on the chunks of natural language text of the query obtained and processed by the natural language processing module to generate a first ranked list of the determined matching vectors. and determining nodes stored in the graph-based memory based on a correspondence between the determined vector of the first ranked list of matching vectors to generate a second ranked list of corresponding nodes, the graph-based memory storing information in the form of nodes interconnected by links, each node including a particular chunk of text representing a concept, each link being disposed between a source node and a target node of the nodes, each link representing a relationship between the concept of the source node and the concept of the target node, the method further comprising: activating, by the processor, the corresponding node stored in the graph-based memory and activating connected nodes based on links between the activated node and other nodes in the graph-based memory by applying a graph traversal method, e.g., a graph traversal algorithm such as random walk or personalized page rank; and, by the processor,generating a third ranking list including the activated nodes stored in the graph-based memory, selecting the activated nodes of the third ranking list for further processing, and determining chunks of text corresponding to the activated nodes of the third ranking list; receiving, by a natural language processing module, the determined chunks of text corresponding to the activated nodes of the third ranking list, and generating a response to the query based on the chunks of text determined based on a semantic closeness between the stored vectors of the vector embedding memory and the vectors generated by the text embedding module based on the chunks of natural language text of the query, and the determined chunks of text corresponding to the activated nodes of the third ranking list for further processing;

[0018] The agent may be an agent that constitutes the system. Alternatively, the agent is another agent that is distinct from the agents that constitute the system.

[0019] Thus, the computer-implemented method provides a response to a query by a user or agent based on the determined chunk of text that corresponds to an activated node in the third ranked list for further processing, which is additional background information.

[0020] A computer-implemented method for generating a response to a query from a user or agent according to one embodiment includes retrieving, by a vector search module, vectors from a vector embedding memory based on semantic closeness of stored vectors in the vector embedding memory in combination with keyword-based searching.

[0021] According to one embodiment of a computer-implemented method for generating responses to queries from a user or agent, the method includes that a vector embedded memory and a graph-based memory are pre-populated in parallel during a training phase of the system.

[0022] Alternatively, a computer-implemented method for generating responses to queries from a user or agent includes dynamically populating the vector-embedded memory and the graph-based memory during the operational phase of the system.

[0023] In a computer-implemented method according to one embodiment, each node contains a chunk of natural language text that represents an abstract concept.

[0024] A computer-implemented method according to one embodiment includes continuously maintaining, by a processor, a correspondence between vectors stored in a vector embedded memory and corresponding nodes stored in a graph-based memory.

[0025] According to one embodiment of a computer-implemented method, the nodes stored in the graph-based memory correspond to concepts that are a superset of the concepts that correspond to the vectors stored in the vector-embedded memory, and the graph-based memory stores additional information about the set of concepts (a general set of concepts) in the form of relationships between the concepts of the set of concepts and additional concepts for which there are no corresponding vectors in the vector-embedded memory.

[0026] In mathematics, a set A is a subset of set B if every element of A is also an element of B, and then B is a superset of A. A and B can be equal, and if they are not equal, A is a proper subset of B. The relationship in which one set is a subset of another is called subsumption or sometimes containment. A being a subset of B can also be expressed as B containing A, or A being contained in B. A k-subset is a subset with k elements.

[0027] A computer-implemented method according to one embodiment includes storing additional information in a graph-based memory regarding additional concepts beyond the concepts corresponding to the vectors stored in the vector-embedded memory.

[0028] A computer-implemented method according to one embodiment includes analyzing, by a natural language processing module, a question, query, or instruction obtained from a user or agent, and determining based on the analysis whether there is an intent of the user to access stored memory contents of the knowledge base and store new memory contents in the knowledge base, i.e., isolating the new memory contents of the user, wherein the text embedding module is further configured to convert the generated chunks of text describing the new memory contents into vectors describing the new memory contents.

[0029] Thus, this embodiment can determine whether a user or agent may wish to retrieve information from memory or store additional information in memory. The system bases the determination of intent on analysis of the query or instruction, for example, by a natural language processing module. The method may be used both to add information to a knowledge base and to retrieve information from a knowledge base.

[0030] A computer-implemented method according to one embodiment includes analyzing, by a natural language processing module, a question, query, or instruction obtained from a user or agent. In an activating step by the processor, corresponding nodes stored in a graph-based memory are activated based on the analysis, and connected nodes are activated based on links between the activated nodes and other nodes in the graph-based memory. In particular, activating corresponding nodes stored in the graph-based memory based on the analysis, and activating connected nodes based on links between the activated nodes and other nodes in the graph-based memory includes constraining activation of connected nodes to a selected subset of link types and node types based on the analysis.

[0031] In this embodiment, if the analysis of the query gives the result that a particular type of question is obtained, e.g., regarding place, time, person, etc., the system determines the nodes in the graph-based memory and activates them, e.g., by focusing on the spatial, temporal, or personal type of links and nodes, respectively. Thus, the retrieval of information from the graph-based memory is made in a more targeted manner, speeding up the search or reducing the required processing resources and improving the results.

[0032] The natural language processing module's analysis of the question, query, or command obtained from the user or agent may include looking for keywords including words such as "where," "when," "who," "how," etc. to determine if a particular type of question related to a place, time, or person is indicated. According to one embodiment of the computer-implemented method, the method includes generating a first ranked list of matching vectors for responding to the obtained query from the user or agent from the vectors stored in the vector embedded memory, the first ranked list including, for each matching vector, a measure of match or a measure of appropriateness or a similarity rating.

[0033] A computer-implemented method according to one embodiment uses a vector embedded memory that stores content of at least one further modality, and the method may include generating a response to the query based on the matching vectors retrieved from the vector embedded memory and further based on the stored content of the at least one further modality.

[0034] Thus, optionally, multi-modal search results may be provided by storing the contents of additional modalities in memory and retrieving results of various modalities from the vector embedding memory by creating embedding vectors in the same embedding space.

[0035] Thus, using an appropriate embedding model, the method also allows embedding inputs of modalities other than textual content, e.g. image content, audio recordings, etc. This allows finding and retrieving not only chunks of text, but also chunks of information of different modalities.

[0036] A computer-implemented method according to one embodiment includes dynamically constructing, by a perception module, a knowledge database stored in a vector-embedded memory and a graph-based memory, the knowledge database including a plurality of chunks of text stored in the vector-embedded memory and a plurality of nodes and links stored in the graph-based memory.

[0037] A computer-implemented method according to one embodiment includes performing, by a processor, a synchronization process, the synchronization process including at least one of comparing representations stored in the vector-embedded memory with corresponding representations stored in the graph-based memory and deriving, from one predefined data source, the representations stored in the vector-embedded memory and the corresponding representations stored in the graph-based memory.

[0038] A computer-implemented method according to one embodiment includes determining, by a processor, a type of query based on an analysis of a chunk of text, and activating, by the processor, nodes stored in the graph-based memory further based on the determined type of query. In particular, the method may include activating, by the processor, selected link types or node categories stored in the graph-based memory further based on the determined type of query.

[0039] A computer-implemented method according to one embodiment includes determining, by a processor, constraints for refining a query based on an analysis of a chunk of text, and activating, by the processor, nodes stored in the graph-based memory further based on the determined type of query. In particular, the method may include activating, by the processor, nodes stored in the graph-based memory that are connected via selected link types or node categories further based on the refined query.

[0040] Thus, by performing pre-processing including intent analysis of the retrieved query and then appropriately adjusting the graph-based augmentation using the knowledge representation stored in the graph-based memory, the graph-based search for suitable information pieces can be more targeted. Thus, the processing is more efficient and at the same time improves the quality of the information retrieved from the graph. For example, if the query asks a question that includes a "where" clause, the type of the determined query may be a location query. Then, the search in the graph-based memory may be constrained to the spatial relationship among the links.

[0041] According to one embodiment, the computer-implemented method includes dynamically adjusting, by the processor, a number of chunks of text retrieved when activating a corresponding node stored in the graph-based memory. In particular, the method may dynamically adjust a number of chunks of text retrieved from the representation stored in the graph-based memory to avoid reaching a token limit. The method may include determining, by the processor, whether information retrieved from the graph-based memory and the vector-embedded memory is sufficient to respond to a query. The method may then dynamically adjust a number of chunks of text retrieved from the graph-based memory if the processor determines that information retrieved from the graph-based memory and the vector-embedded memory is insufficient to respond to the query.

[0042] Thus, the post-hoc LLM inference is optimized, resulting in reduced processing complexity and adapting the response to the requirements of the response, for example, the method can avoid reaching the token limit or by separately analyzing whether the retrieved information is sufficient to answer the query or whether it is recommended to retrieve more information to adequately answer the query.

[0043] A computer-implemented method according to one embodiment includes determining, by a processor, a size of the retrieved chunk of text from the representations stored in the graph-based memory and the vector-embedded memory, and if the determined size exceeds a predefined threshold, applying a process to summarize the retrieved chunk of text based on a large-scale language model, in particular based on a MapReduce algorithm.

[0044] This limits the amount of data to be collected and based on which a response can be generated, thereby limiting the computational complexity of subsequent processing steps.

[0045] MapReduce refers to a programming model and associated implementation for processing and generating large data sets using independent and potentially parallel processing algorithms on a group of processing devices. MapReduce involves mapping the same function onto several inputs, allowing for independent and potentially parallel processing, followed by a summarization step. The summarization step may be performed in parallel on several Graphics Processing Units (GPUs). Alternatively, variants of MapReduce may be used to retrieve more information from a knowledge base and incrementally add information until it determines that the generated answer to a query is accurate.

[0046] According to one embodiment, a computer-implemented method includes determining, by a processor, a measure representative of a mapping of chunks of text to a representation stored in a graph-based memory, and adapting the process of generating chunks of text and determining the measure of mapping until the determined measure of mapping satisfies a predetermined termination criterion.

[0047] The system according to the second aspect provides the corresponding advantages discussed with respect to the computer-implemented method according to the first aspect.

[0048] In a second aspect, a system of an agent for generating a response to a query from a user or agent includes a vector embedding memory configured to store vectors and chunks of generated natural language text, a graph-based memory configured to store information in the form of nodes connected to each other by links, and a processor. The processor is configured to search for matching vectors from the vector embedding memory based on semantic closeness between the vectors stored in the vector embedding memory and the vectors generated based on the chunks of natural language text of the retrieved query, determine and activate nodes stored in the graph-based memory based on their correspondence with the matching vectors, determine and activate connected nodes based on links between the activated nodes, and generate a response to the query based on the chunks of text determined based on the semantic closeness between the matching vectors from the vector embedding memory and the vectors generated based on the chunks of natural language text of the query. The processor further generates the response based on the determined chunks of text corresponding to the activated nodes of the graph-based memory.

[0049] Also advantageous is a system comprising an agent and a system according to the second aspect for generating a response to a query from the agent, wherein the agent is an autonomous agent.

[0050] In particular, the autonomous agent is configured to generate and output queries based on the large-scale language model, and to perform behavior planning for the autonomous agent based on responses received from the system to generate responses to the autonomous agent's queries.

[0051] Thus, responses to queries received from the system may include, for example, information regarding the preferences of relevant stakeholders and the causal chain of possible actions planned for execution by the autonomous agent.

[0052] In this embodiment, the autonomous agent is configured to generate and internally process queries based on the large language model, and to plan its behavior based on responses received from the system (which acts as the autonomous agent's memory) to generate responses to the autonomous agent's queries. The memory can store information related to decision-making and robot planning, such as a memory for actions capturing causal chains, particularly but not limited to, via a graph-based memory, as well as information about possible outcomes, capabilities, means, or agents involved in an action, particularly but not limited to, via a text-based description.

[0053] In this case, the agent is not using the proposed system to generate a response to a user or another agent that posed a query, but rather the autonomous agent is consulting its own internal memory as a kind of internal monologue to retrieve specific information for reasoning, planning, or taking action.

[0054] Thus, the system and corresponding method are particularly useful in applications for planning the behavior of autonomous agents, e.g., robots. The technique may help retrieve relevant information for the robot's decision-making process. For example, the preferences of a human user may be stored in a particular text (e.g., "Paul likes to snuggle with cats"). When the robot presents a query to the system, corresponding to "how can I cheer up Paul", the response generated and output to the system may be based on the preferences in the stored text.

[0055] Many interconnected memories relate to activity-related or behavior-related memories. These interconnected memories may store information that represents causal chains. Examples of such causal chains may include inferences similar to "something was observed, then something else happened, which resulted in...". Causal chains stored in the interconnected memories may also include direct information about outcomes, capabilities, means, or agents that may be involved in an action. The computer-implemented method is suitable for storing information about such causal chains and retrieving such information, which is essential for making predictive inferences, including action plans for autonomous agents.

[0056] The following description of the embodiments refers to the following figures: [Brief description of the drawings]

[0057] [Figure 1] 1 is a schematic flow diagram of a computer-implemented method according to one embodiment. [Diagram 2] FIG. 1 illustrates an architectural overview of the graph-vector-text memory of the proposed system and its population and usage in one embodiment. [Diagram 3] FIG. 1 is a schematic diagram of an architecture for a search process that leverages relationship information in one embodiment. [Figure 4] FIG. 1 is a high-level abstraction overview of the architecture of the proposed system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0058] In the figures, corresponding elements have the same reference numbers. The description of the figures will avoid, where possible, the description of the same reference numbers in different figures without negatively affecting comprehension and will avoid unnecessary repetition for the sake of brevity.

[0059] This disclosure refers to vectors, and in particular to dense vectors. Dense vectors are vectors with high dimensions, for example including three orders of magnitude of dimensions. Each dimension contains relevant information, which can be determined by neural networks. Due to their high dimensionality, it is more difficult to compress these vectors, and therefore they typically require more memory than sparse vectors.

[0060] The vectors define an embedding vector space adapted to capture the semantic proximity of the vectors to each other.

[0061] This application uses word tokens to represent words in a natural language utterance that are mapped to vectors of embedding values ​​of nodes in a knowledge graph. A token can be a part of a word, a word, or a sequence of words. Typically, a token corresponds to four characters.

[0062] The present application uses known word token representations as embedding vectors for natural language text. A token may be a part of a word, a word, or a sequence of words. In particular, a token may refer to a part of a word. Before processing the prompt, the input is broken into tokens. The tokens are not cut exactly where a word begins or ends. Tokens may include trailing spaces and even partial words. A token often refers to four letters in English, so that one token corresponds to roughly three-quarters of a word, or 100 tokens correspond to about 75 words.

[0063] The term token limit refers to the maximum number of tokens that an LLM can process. The token limit is the combination of input tokens and output tokens.

[0064] In knowledge representation and reasoning, a knowledge graph is a knowledge base that integrates data using a graph-structured data model or topology. In general, a knowledge graph can store interlinked descriptions of entities, i.e. objects, events, situations, or abstract concepts, while encoding the meanings underlying the terms used. In particular, a knowledge graph can represent a digital structure that represents knowledge as concepts and the relationships (facts) between those concepts. A knowledge graph can contain explicit specifications of concepts that allow both humans and machines to understand and reason about its content.

[0065] The computer-implemented method may be particularly advantageous for agents, including virtual agents, which are computer-generated agents capable of interacting with human users. Task-oriented virtual agents can communicate with human users in natural language and collaborate with or assist the users in performing various tasks.

[0066] Natural language understanding systems interpret sequences of words in a user's speech and are used by task-oriented virtual agents.

[0067] The present computer-implemented method and system extends the benefits of graph databases in storing information in the form of heterogeneous graphs, where nodes (concept nodes) represent entities or concepts, and edges represent relationships between entities or concepts. In graph databases, relationships, e.g., heterogeneous connections between entities, are first-class citizens. With their graph structure and flexible schema, graph databases enable more efficient and expressive ways than relational databases to handle high-order relationships between distant entities, particularly for navigating data structures that contain multi-hop hierarchies. Whereas traditional databases require expensive join operations to look up information, graph databases can use the graph database's adjacency matrix to directly traverse the graph and navigate the links more efficiently.

[0068] More and more personal information is available in digital form. Access to this information source has been greatly facilitated by recent improvements in LLM and augmentation techniques such as retrieval augmented generation (RAG), which allows the development of chatbots for personal or private data. However, even when combined with keyword search, techniques built around semantic similarity using embeddings are inherently limited by those search techniques that do not consider relational information. In contrast, the disclosed computer-implemented method provides such relational information and leverages it to identify additional relevant context, enabling deeper answers to questions that go well beyond the content of individual documents or isolated chunks of text and reveal improved understanding of the subject of the query.

[0069] Examples of knowledge management systems include, but are not limited to, systems for answering natural language queries using a graph-based knowledge base, e.g., Notion, and chatbots for searching information about research papers, which can exploit information not explicitly contained in the paper itself. For example, relational information about the affiliations of the authors may provide further insight. The following diagrammatic description focuses on advantageous implementations of the method for answering queries. Further advantageous application areas are summarized at the end.

[0070] FIG. 1 illustrates a schematic flow diagram of a computer-implemented method according to one embodiment for generating a response to a query.

[0071] The method starts at step S1 with obtaining and processing natural language instructions from a user by a natural language processing module NLP to generate chunks of text based thereon. The natural language instructions may correspond to a query or queries from a user or an agent. The agent may be a virtual or embodied agent.

[0072] A natural language processing module NLP may combine several modules, the natural language processing module NLP being configured to analyze and process text in a natural language to generate input for the other modules.

[0073] The natural language processing module NLP may be based on an LLM, such as ChatGPT, which may be suitable for performing tasks involving natural language processing for intent classification. For intent classification, the natural language processing module NLP identifies the intent associated with the natural language (NL) query. In particular, the natural language processing module NLP mainly separates the questions in the query that should receive a response. Furthermore, the natural language processing module NLP may identify statements that should be stored in the memory 23 of the system 20.

[0074] In addition, the natural language processing module NLP may perform information extraction by processing the NL statements, for example via named entity recognition and named entity linking, in order to store the statements in memory 23.

[0075] The natural language processing module NLP may also perform query formulation (NLPQP). To perform NLPQP, the natural language processing module NLP takes as input a NL text and then converts it into a query, for example a Cypher.

[0076] The system 20 may include a chunk processing module (CH) that divides the text input into manageable chunks of text (text chunks), each of which is of a smaller size that is optimized for the token limitations of the text embedding module TE and the response generation module RGM and their respective capabilities to capture the meaning of the text and generate response text.

[0077] The chunk processing module CH further performs a concatenation of the chunk of text generated from a query received from a user or agent with the selected natural language chunk and passes the concatenated chunk of text to the response generation module RGM.

[0078] In step S2, the method continues with the generation of vectors by the text embedding module TE based on the chunks of text generated based on the retrieved instructions.

[0079] The text embedding module TE receives as input natural language text and processes the natural language text based on statistical correlations, which may be implicit in the neural network (NN) representation. The text embedding module TE generates vector embeddings (VEs), or vectors, which can be stored in memory and compared to other vectors for similarity.

[0080] In step S3, the method may store the vectors generated by the text embedding module TE in a vector embedding memory VEM in association with the chunks of generated text.

[0081] Although not explicitly shown in FIG. 1, the method may store the generated vectors in a vector embedded memory VEM and further store the corresponding nodes and the links between those corresponding nodes in a graph based memory GBB.

[0082] The structure of the vector embedded memory VEM will be explained in detail with reference to the graph vector text memory GVT in FIG.

[0083] In step S4, the method includes searching, by the processor 21, the vector embedding memory VEM to determine matching vectors based on their similarity with the vector of the retrieved query from the user or agent and retrieving, by the vector search module VR, stored vectors from the vector embedding memory VEM based on semantic closeness between the stored vectors of the vector embedding memory VEM and the vectors generated by the text embedding module TEM based on the chunks of natural language text of the query retrieved and processed by the natural language processing module NLP, in order to generate a first ranked list of the determined matching vectors.

[0084] Step S4 is followed by step S5, which includes the processor 21 determining, based on a correspondence between the determined vectors of the first ranked list of matching vectors, the nodes stored in the graph-based memory GBM to generate a second ranked list of corresponding nodes. The graph-based memory GBM stores information in the form of nodes interconnected by links, each node containing a particular chunk of text representing a concept, each link being located between a source node and a target node of the nodes, each link representing a relationship between a concept of the source node and a concept of the target node.

[0085] In step S6, the method continues with the processor 21 activating the corresponding node stored in the graph-based memory GBM and activating further connected nodes based on links between the activated node and other nodes in the graph-based memory GBM by applying a graph traversal algorithm, which may include random walk or personalized page rank.

[0086] In step S7, the processor 21 further generates a third ranked list including the activated nodes stored in the graph based memory GBM, selects the activated nodes of the third ranked list for further processing, and determines chunks of text that correspond to the activated nodes of the third ranked list.

[0087] In step S8, the method continues with receiving, by the natural language processing module NLP, the determined chunk of text corresponding to the activated node of the third ranking list, which generates a response to the query based on the chunk of text determined based on the semantic closeness of the stored vectors of the vector embedding memory VEM and the vectors generated by the text embedding module TEM based on the chunk of natural language text of the query, and the determined chunk of text corresponding to the activated node of the third ranking list for further processing.

[0088] In particular, a question answering module QA implemented using a natural language processing module NLP generates an answer to the query based on a given context of the chunk of text corresponding to the activated node of the third ranked list. The question answering module QA may be implemented using a specially trained language model or even a generic LLM.

[0089] Because the original entry point of the search and the search propagation process are known, responses can be easily extended with source citations, allowing full traceability.

[0090] The generated response is then output to the user who posed the original query or to the agent providing the query. The response is in natural text, particularly in an audible or visible form.

[0091] Figure 2 shows an architectural overview of the graph-vector-text memory GVT of the proposed system in one embodiment, as well as its population and usage.

[0092] In the center of Figure 2, the graph-vector-text memory GVT is depicted.

[0093] The graph-vector-text memory GVT includes a text chunk memory TCM, a vector embedded memory VEM, and a graph-based memory GBM.

[0094] The GVT may be implemented as part of the memory 23 of the system 20. Alternatively or additionally, at least a part of the graph-vector-text memory GVT, the text chunk memory TCM, the vector embedded memory VEM and the graph based memory GBM may be implemented on one or more servers accessible via the network 27 of the system 20.

[0095] Text Chunk Memory (TCM) is a memory for storing and retrieving text chunks.

[0096] A graph-based memory GBM is a database that stores knowledge graphs and retrieves concepts from them. A graph-based memory GBM stores, for example, a graph with nodes that may be labeled and optionally labeled links. The labeled links may, for example, include the nature of the relationship on which the link is based. The nodes of the graph represent memory items, and the links between the nodes of the graph represent associations between the nodes, with each link being between a start node and a target node.

[0097] In particular, a graph-based memory GBM stores information in the form of nodes interconnected by links, each node containing a particular chunk of text representing a concept, each link being placed between a source node and a target node among the nodes, each link representing a relationship between the concept of the source node and the concept of the target node.

[0098] Graph-based memory GBM may be implemented using a knowledge graph (KG) in a graph database such as Neo4j.

[0099] The Vector Embedding Memory VEM is a vector storage for storing and retrieving vectors (embedding vectors). Each vector stored in the Vector Embedding Memory VEM maintains a reference to the text chunk from which it was generated, as well as a reference to the corresponding node in the graph-based memory GBM.

[0100] Thus, vectors stored in the vector embedded memory VEM can reference, eg trigger, nodes stored in the graph based memory GBM.

[0101] An implementation of the vector embedded memory VEM may be based on existing vector stores or vector databases, including ChromaDB and Pinecone.

[0102] In the proof of concept for storing small chunks of text, the graph-vector-text memory GVT, which includes the text chunk memory TCM, the vector embedding memory VEM, and the graph-based memory GBM, as well as the TCM and VEM modules, are realized through Neo4j. Neo4j includes a graph database that supports the storage of vector embeddings and enables exploration by embedding similarities using an implementation of the HNSW algorithm, as disclosed in Hasbe, S.: Neo4j's Vector Search: Unlocking Deeper Insights for AI-Powered Applications, 2023 (https: / / neo4j.com / blog / vector-search-deeper-insights).

[0103] The queries obtained by the system 20 include natural language text (plain text) that represents the input to the system 20 .

[0104] The text embedding module TE generates a vector (embedding vector) based on the obtained query.

[0105] The generated vector provides the basis for similarity searches within a vector representation containing multiple vectors stored in a vector embedded memory VEM.

[0106] The results of the similarity search in the vector representation include vectors that satisfy the similarity criterion of the similarity search: on the one hand, the resulting similar vectors correspond directly to (are associated with) text chunks stored in the text chunk memory TCM, and on the other hand, the resulting similar vectors correspond to (are associated with) nodes of the graph representation stored in the graph based memory GBM.

[0107] The nodes of the graph-based representation associated with the obtained similarity vectors form the basis of an association propagation search on the graph representation stored in the graph-based memory GBM.

[0108] The association propagation search on the graph representation may be performed iteratively until a termination criterion is met, which may be based on determining that sufficient information has been retrieved to answer the query.

[0109] The association propagation search returns further nodes that enable the system 20 to retrieve from the text chunk memory TCM further related chunks of text that are stored in the text chunk memory in association with the further nodes.

[0110] All retrieved chunks of text are then provided to the collection and response generation module RG. The total retrieved chunks of text includes the chunks of text directly retrieved by the similarity search performed on the embedded memory VEM (hereafter referred to as (retrieved) chunks of text) and the additionally retrieved related chunks of text determined by graph propagation using the graph-based memory GBM (hereafter referred to as further (retrieved) chunks of text).

[0111] The collection and response generation module RG collects all the evidence, including in particular the chunks of retrieved text and further chunks of retrieved text identified as relevant by the association propagation search process. The collection and response generation module RG (response generation module RG) then generates a response to the retrieved query and outputs the generated response to the user or agent submitting the query.

[0112] The generated response is natural language text.

[0113] FIG. 3 shows an architectural overview of a search process that leverages relationship information in one embodiment of a computer-implemented method.

[0114] Once a query to the system 20 is formulated by a user or agent and received by the system 20, the received query is processed by a natural language processing module NLP.

[0115] The natural language processing module NLP passes the obtained text, which includes one or more chunks of text, to the text embedding module TE, which generates embedding vectors (vectors) based on the chunks of text. The geometry of the embedding vector space is suitable for capturing the semantic closeness of the vectors that make up the vector space. The generated vectors are then used to query the vector embedding memory VEM for similarity with the vectors stored in the vector embedding memory VEM. The vector embedding memory VEM returns a ranked list of the most similar vectors stored in the vector embedding memory VEM. The returned most similar vectors represent the vectors or concepts that are semantically closest to the vector generated based on the text chunks of the original query. Each vector returned by the vector embedding memory VEM contains a reference to a corresponding node in the graph-based memory GBM. In particular, each vector returned by the vector embedding memory VEM contains a reference to a corresponding node in the graph-based memory GBM, thereby triggering a specific concept that corresponds to a node. Thus, the nodes corresponding to the most relevant concepts to the original query can be retrieved from the graph-based memory GBM based on the results of the similarity search performed on the vectors stored in the vector embedding memory, and starting from the node corresponding to the most relevant concept, further related nodes and links are retrieved from the graph-based memory GBM using the link structure stored in the graph-based memory GBM.

[0116] The most relevant nodes, including nodes selected directly based on a similarity search performed on the vectors stored in the vector embedded memory VEM and further nodes retrieved by searching along the links of the graph based memory GBM, are then used to retrieve related text chunks from the text chunk memory TCM as further retrieved chunks of text.

[0117] The text of the retrieved text chunks is then compiled as natural text background evidence, combined with the original query text and metadata from the graph-based memory GBM for further traceability of the retrieved text chunks, and passed to the response generation module RG for generating a response thereon. In particular, the text of the retrieved text chunks forms the basis for assembling an answer to the original query provided to the system 20.

[0118] In addition, the system 20 may augment the generated response with reference to the chunk of text used as a basis for generating the response to allow full traceability of the chunk of text and to the further chunk of text used as a basis for generating the response. This reference may form augmentation information that may include metadata of the further retrieved chunk of text, for example, an identifier associated with the respective chunk of text and a source of the respective chunk of text. This reference is advantageous for traceability of the further retrieved chunk of text, and metadata such as identifiers and sources of the chunk of text may be output to the response generation module RG to augment the generated response. The bottom part of FIG. 3 provides further details of the search process for searching for related concepts in the graph-based memory GBM.

[0119] The search process first determines nodes stored in the graph-based memory GBM based on their correspondence with the determined vector in the first ranked list of matching vectors to generate a second ranked list of corresponding nodes.

[0120] Then, the search process continues by activating the corresponding nodes stored in the graph-based memory GBM and also activating the connected nodes based on the links between the activated nodes and other nodes in the graph-based memory GBM, for example by applying a random walk algorithm or a personalized page rank algorithm, although other kinds of graph algorithms such as graph traversal algorithms or spreading activation algorithms are also applicable.

[0121] The search process then continues by generating a third ranked list containing the activated nodes stored in the graph-based memory GB, selecting the activated nodes of the third ranked list for further processing, and determining chunks of text that correspond to the activated nodes of the third ranked list.

[0122] Ranking the determined vectors of the activated nodes of the first ranking list among the second ranking list and the third ranking list may be performed by calculating their respective importance scores and sorting the first, second and third ranking lists based on the calculated importance scores.

[0123] The importance score may include a numerical value that describes the relevance of a vector or node for answering the query currently being processed.

[0124] The third ranked list then represents input for determining relevant chunks of text corresponding to activated nodes in the third ranked list for further processing, in particular for gathering information for formulating a response to a query.

[0125] FIG. 4 shows a high-level abstraction overview of the proposed system 20 architecture.

[0126] The system 20 of FIG. 4 includes a processor 21, a data storage 23 (memory 23), an input / output interface 24, and a network interface 25, which are connected by a data bus 22.

[0127] The input / output interface 24 may provide, among other things, the ability to receive queries from other agents or human users via voice or text, and thus may represent an interface for connecting any combination of input / output devices 26, including, but not limited to, a keyboard, a mouse, a pointing device, a display, a microphone, and a loudspeaker.

[0128] The input / output interface 24 may be implemented, at least in part, in software modules executed on the processor 21 .

[0129] The processor 21 may be any type of controller or processor, and may be embodied as one or more processors 21 adapted to perform the functions described herein. As the term processor is used herein, the processor 21 may include the use of a single integrated circuit (IC), or may include the use of multiple integrated circuits or other components connected, arranged, or grouped together, such as controllers, microprocessors, digital signal processors (DSPs), parallel processors, multi-core processors, custom ICs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), and the like, and may further include adaptive computing ICs and associated memory, such as RAM, DRAM, and ROM, and other ICs and components. Thus, the term processor 21 should be understood to equally mean and include a single IC, or a configuration of custom ICs, ASICs, processors, microprocessors, controllers, FPGAs, adaptive computing ICs, or other groupings of integrated circuits that perform the functions described with respect to the computer-implemented method, together with associated memory, such as microprocessor memory or additional RAM, DRAM, SDRAM, SRAM, MRAM, ROM, FLASH, EPROM, or E2 PROM. The processor 21, together with its associated memory, may be adapted or configured through programming, FPGA interconnects, or hardwiring to perform the methodology of the computer-implemented method. For example, the method may be programmed and stored in the processor 21, together with associated memory or memory 23 and other equivalent components, as a set of program instructions or other code for later execution when the processor 21 is operational, e.g., powered on and functioning.

[0130] The memory 23 may include a data repository or database and may be embodied in any number of forms, including within any computer or other machine-readable data storage medium, memory device, or other storage or communication device for storing or communicating information, including, but not limited to, a memory integrated circuit (IC), or a memory portion of an integrated circuit (such as resident memory within the processor 21), which may be volatile or non-volatile, removable or non-removable, and includes, without limitation, RAM, FLASH, DRAM, SDRAM, SRAM, MRAM, FeRAM, ROM, EPROM or E2 PROM, or any other form of memory device, such as a magnetic hard drive, optical drive, magnetic disk or tape drive, hard disk drive, other machine-readable storage or memory medium, such as a floppy disk, CD-ROM, CD-RW, digital versatile disk (DVD) or other optical memory, or any other type of memory, storage medium, or data storage device or circuitry. Additionally, such computer readable media includes any form of communication media that embodies computer readable instructions, data structures, program modules or other data in a data signal or modulated signal, such as an electromagnetic or optical carrier wave or other transport mechanism, and includes any information delivery medium that may encode data or other information in a wired or wireless signal, including electromagnetic, optical, acoustic, RF, or infrared signals. The memory 23 may be adapted to store various look-up tables, parameters, coefficients, other information and data, programs or instructions for the software of the present disclosure, as well as other types of tables, such as database tables.

[0131] The memory 23 may in particular implement at least a portion of the graph vector text memory GVT, the text chunk memory TCM, the vector embedding memory VEM, and the graph based memory GBM, however, in an alternative embodiment of the present disclosure, the text chunk memory TCM, the vector embedding memory VEM, and the graph based memory GBM are not separate modules of a computer program, but are implemented as a single graph vector text memory GVT that collectively stores the graph, text chunk, and embedded vector information.

[0132] The processor 21 is programmed using the software and data structures of the disclosed computer-implemented method, for example, to perform the methodology of the present disclosure. Thus, the system and computer-implemented method of the present invention may be embodied as software providing programming or other instructions as described above, such as a set of instructions and / or metadata embodied in a computer-readable medium. In addition, metadata may be utilized to define various data structures of a lookup table or database. Such software may be in the form of source code or object code, for example, without limitation. The source code may further be compiled into some form of instructions or object code (including assembly language instructions or configuration information). The software, source code or metadata of the present invention may be embodied as any type of code, such as C, C++, Python, SystemC, LISA, XML, Java, Brew, SQL and its variants (e.g., SQL99 or proprietary versions of SQL), DB2, Oracle, or other types of programming languages ​​that perform the functions described herein, including various hardware definition or hardware modeling languages ​​(e.g., Verilog, VHDL, RTL) and the resulting database files (e.g., GDSII). As a result, "structure," "program structure," "software structure," or "software," as used equivalently herein, means and refers to any programming language of any kind, with any syntax or signature, that provides or is interpreted to provide a specified associated function or methodology (e.g., when instantiated or loaded and executed on a processor or computer, including processor 21).

[0133] The software, metadata, or other source code and resulting bit files (object code, databases, or lookup tables) of the present invention may be embodied as computer readable instructions, data structures, program modules or other data in any tangible storage medium, such as any of the computer or other machine-readable data storage media, as described above with respect to memory 175, such as, for example, a floppy disk, CD-ROM, CD-RW, DVD, magnetic hard drive, optical drive, or any other type of data storage device or medium.

[0134] The processor 21 may in particular execute software that controls the processor 21 to perform the functions of, for example, a natural language processing module NLP, a text embedding module TE, a chunk processing module CH and a question answering module QAM.

[0135] Network interface 25 provides system 20 with the ability to connect to external databases and servers via communication network 27. Network interface 25 allows, inter alia, for system 20 to be implemented in a spatially distributed manner, with at least some of the individual method steps being performed at least in part remotely from system 20.

[0136] 1-3 present features of the disclosure with reference to a particular embodiment of a system 20 for generating natural language responses to queries posed in natural language by a user or agent. The described embodiment represents one particular application.

[0137] Agents operating in the real world may have a digital form (virtual agents) or an embodied form (physical agents). Both virtual and physical agents require the ability to store knowledge of their environment, common sense knowledge, and personal memories and beliefs. The system 20 of the present disclosure is capable of storing information specific to each agent. This information may come from the hardware, but also from the personal preferences of the agent's human owner (user).

[0138] In addition, virtual and physical agents benefit from the ability to continually update their knowledge during the operational phase after an initial training phase. Information for updating an agent's knowledge base is often available in text or may at least be expressed in textual form, often also in natural language text. Information for updating an agent's knowledge base may include at least a portion of descriptions of objects that are useful or at least relevant to addressing the task and their objectives, descriptions of humans including their properties and preferences, and a history of actions that have occurred in the agent's environment.

[0139] The computer-implemented method and system 20 may be used to incrementally update the knowledge base of an agent, whether a physical or virtual agent. The proposed memory structure architecture and associated memory architecture management assists in incrementally updating the pre-trained knowledge base during the agent's operational phase.

[0140] Examples of such agents include, but are not limited to, digital agents, such as Apple's Siri, as well as agents with robotic embodiments, such as personal assistant robots.

[0141] The computer-implemented method and system 20 aid in the process of automating workflows that humans perform repetitively using their computers, and also enhance the ability of embodied agents to assist people with mental or physical disabilities by anticipating their intentions and proactively performing useful actions after performing a planning process of the assistive behavior. These intelligent agents, both virtual and physical, can benefit from incremental updates of their knowledge bases. Incorporating the system 20 and computer-implemented method into such agent and assistance system embodiments provides advantageous application areas of the present disclosure.

[0142] All steps performed by various entities described in this disclosure, and functions described as being performed by various entities, are intended to mean that the respective entities are adapted or configured to perform the respective steps and functions.

[0143] In the claims and the detailed description, the word "comprising" does not exclude the presence of other elements or steps. The indefinite articles "a" or "an" do not exclude a plurality.

[0144] A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that different dependent claims recite certain means and functions of a control circuit does not exclude that a combination of these means and functions cannot be combined in an advantageous implementation.

[0145] Terms, abbreviations and definitions ChatGPT Chatbot Generative Pre-trained Transformer CH chunk processing DNN Deep Neural Network EM Embedding Model: A model, e.g. a neural network, specially trained to create vector embeddings for text EV embedding vector GBM Graph-Based Memory GNN Graph Neural Network GPT Generative Pre-trained Transformer: A type of LLM GVT Graph Vector Text (memory) KG Knowledge Graph LLM Large Scale Language Model LTM Long-term memory NEL Entity Linking NER named entity recognition NL natural language NLP Natural Language Processing NLPRP Natural Language Processing for Response Creation NLPQP: Natural Language Processing for Query Creation NN Neural Network MTM Medium Term Memory QA questions and answers RAG Search Enhancement Generation RGM Response Generation Module STM Short-term memory TCM Text Chunk Memory TE Text Embeddings VE Vector Embedding VEM Vector Embedded Memory VS Vector Store

Claims

1. 1. A computer-implemented method for generating responses to queries from a user or agent in a system by an agent, the system comprising: a Vector Embedded Memory (VEM) configured to store vectors and chunks of generated natural language text, a Graph Based Memory (GBM) configured to store information in the form of nodes interconnected by links, and a processor (21), said method comprising: searching for matching vectors from the vector embedding memory (VEM) based on semantic closeness between the vectors stored in the vector embedding memory (VEM) and vectors generated based on the chunks of the natural language text of the retrieved query; determining and activating nodes stored in the graph-based memory (GBM) based on their correspondence with the matching vectors, and determining and activating connected nodes based on links between the activated nodes; generating a response to the query based on chunks of text determined based on the matching vectors from the vector embedded memory (VEM) and further based on determined chunks of text that correspond to the activated nodes in the graph based memory (GBM).

2. Augmenting the generated response with reference to the chunk of text used as the basis for generating the response to allow for full traceability.

2. The computer-implemented method for generating a response to a query from a user or agent of claim 1, further comprising:

3. obtaining and processing natural language instructions from the user to generate chunks of text with a natural language processing module (NLP); generating vectors based on said chunks of text by a text embedding module (TE); storing, by said vector embedding memory (VEM), said vectors generated by said text embedding module (TE) in association with said generated chunks of text; searching the vector embedding memory (VEM) by the processor (21) to determine matching vectors based on similarity with the vector of the retrieved query from the user or agent, and retrieving the stored vectors from the vector embedding memory (VEM) by a vector retrieval module (VR) based on semantic closeness between the vectors stored in the vector embedding memory (VEM) and vectors generated by the text embedding module (TEM) based on chunks of natural language text of the query retrieved and processed by the natural language processing module (NLP) to generate a first ranked list of the determined matching vectors; determining nodes stored in the graph-based memory (GBM) based on a correspondence between the determined vector and the first ranked list of matching vectors to generate a second ranked list of corresponding nodes; wherein the graph-based memory (GBM) stores information in the form of nodes interconnected by links, each node containing a particular chunk of text representing a concept, each link being disposed between a source node and a target node among the nodes, each link representing a relationship between a concept of the source node and a concept of the target node, and the method further comprises: activating, by said processor (21), said corresponding node stored in said graph-based memory (GBM) and activating connected nodes based on links between said activated node and other nodes of said graph-based memory (GBM) by applying a graph traversal algorithm, in particular a random walk or a personalized page rank; generating, by the processor (21), a third ranked list comprising the activated nodes stored in the graph-based memory (GBM), selecting the activated nodes of the third ranked list for further processing, and determining chunks of text corresponding to the activated nodes of the third ranked list; receiving, by the natural language processing module (NLP), the determined chunk of text corresponding to the activated node of the third ranked list, and generating a response to the query based on a semantic closeness of the chunk of text between the stored vectors of the vector embedding memory (VEM) and the vector generated by the text embedding module (TEM) based on the chunk of natural language text of the query, and the determined chunk of text corresponding to the activated node of the third ranked list for further processing; 3. A computer-implemented method for generating a response to a query from a user or agent according to claim 1 or 2, comprising:

4. retrieving said vectors from said vector embedding memory (VEM) based on semantic closeness of said stored vectors in said vector embedding memory (VEM) in combination with keyword-based searching by said vector retrieval module (VR); 4. The computer-implemented method for generating a response to a query from a user or agent of claim 3, further comprising:

5. 5. A computer-implemented method for generating responses to queries from a user or agent according to any one of claims 1 to 4, comprising pre-populating the Vector Embedded Memory (VEM) and the Graph Based Memory (GBM) with data during a training phase of the system.

6. 6. A computer implemented method for generating responses to queries from a user or agent according to any one of claims 1 to 5, comprising dynamically populating said Vector Embedded Memory (VEM) and said Graph Based Memory (GBM) with data during an operational phase of the system.

7. 7. A computer-implemented method for generating responses to queries from a user or agent as claimed in any one of claims 1 to 6, wherein each node comprises a chunk of natural language text representing an abstracted concept.

8. 8. A computer-implemented method for generating responses to queries from a user or agent as claimed in any one of claims 1 to 7, wherein each concept is represented by a node stored in the graph-based memory (GBM), a corresponding chunk of text, and a corresponding vector stored in the vector-embedded memory (VEM).

9. 9. A computer-implemented method for generating responses to queries from a user or agent according to any one of claims 1 to 8, comprising: the processor (21) continually maintaining a correspondence between the vectors stored in the vector-embedded memory (VEM) and the corresponding nodes stored in the graph-based memory (GBM).

10. the nodes stored in the graph-based memory (GBM) correspond to concepts that are a superset of concepts corresponding to the vectors stored in the vector-embedded memory (VEM); The graph-based memory (GBM) is configured to store additional information about a set of concepts in the form of relationships between concepts of the set of concepts and additional concepts for which there is no corresponding vector in the vector-embedded memory (VEM). A computer implemented method for generating a response to a query from a user or agent according to any one of claims 1 to 9.

11. 11. A computer-implemented method for generating responses to queries from a user or agent as claimed in any one of claims 1 to 10, wherein the graph-based memory (GBM) is further configured to store additional information relating to additional concepts beyond the concepts corresponding to the vectors stored in the vector-embedded memory (VEM).

12. 12. The computer-implemented method for generating responses to queries from a user or agent according to any one of claims 1 to 11, comprising: analyzing, by the natural language processing module (NLP), a question, query or instruction obtained from the user or agent, and determining based on the analysis whether there is an intent of the user to access memory contents of a knowledge base and store new memory content in the knowledge base; and wherein the text embedding module (TE) is further configured to convert generated chunks of text describing the new memory content into vectors describing the new memory content.

13. analyzing a question, query, or command obtained from the user or agent by the natural language processing module (NLP); an activation step by said processor (21) activating corresponding nodes stored in said graph-based memory (GBM) based on said analysis, and activating connected nodes based on links between said activated nodes and other nodes in said graph-based memory (GBM), in particular constraining the activation of said connected nodes to a selected subset of link types and node types based on said analysis.

13. A computer implemented method for generating a response to a query from a user or agent according to any one of claims 1 to 12, comprising:

14. 14. A computer-implemented method for generating a response to a query from a user or agent according to any one of claims 1 to 13, comprising generating, by the processor (21), a first ranked list of matching vectors from the vectors stored in the vector embedded memory (VEM) for responding to the retrieved query from the user or agent, the first ranked list including, for each matching vector, a measure of match or a measure of appropriateness or a similarity rating.

15. the vector embedded memory (VEM) is configured to store content of at least one further modality; The method includes generating the response to the query based on the matching vectors retrieved from the vector embedded memory (VEM) and further based on stored content of the at least one further modality.

15. A computer implemented method for generating a response to a query from a user or agent according to any one of claims 1 to 14.

16. and dynamically constructing a knowledge database stored in the vector-embedded memory (VEM) and the graph-based memory (GBM) by a perception module (), the knowledge database including a plurality of chunks of text stored in the vector-embedded memory (VEM) and a plurality of nodes and links stored in the graph-based memory (GBM).

16. A computer implemented method for generating a response to a query from a user or agent according to any one of claims 1 to 15.

17. performing a synchronization process by the processor (21), the synchronization process including at least one of: comparing representations stored in the vector embedded memory (VEM) with corresponding representations stored in the graph based memory (GBM); and deriving, from one predetermined data source, the representations stored in the vector embedded memory (VEM) and the corresponding representations stored in the graph based memory (GBM).

17. A computer implemented method for generating a response to a query from a user or agent according to any one of claims 1 to 16.

18. determining, by the processor (21), a type of the query based on an analysis of the chunk of text; and activating, by the processor (21), the nodes stored in the graph-based memory (GBM) further based on the determined type of query; In particular, activating, by the processor (21), selected link types or node categories stored in the graph-based memory (GBM) further based on the determined type of query.

18. A computer implemented method for generating a response to a query from a user or agent according to any one of claims 1 to 17, comprising:

19. dynamically adjusting, by said processor (21), the number of chunks of text to be searched upon activation of corresponding nodes stored in said graph-based memory (GBM); Dynamically adjusting the number of chunks of text retrieved from the representation stored in said graph-based memory (GBM), in particular to avoid reaching a token limit; or determining, by said processor (21), whether information retrieved from said graph-based memory (GBM) and said vector-embedded memory (VEM) is sufficient to respond to said query; and dynamically adjusting the number of chunks of text retrieved from the graph-based memory (GBM) when the processor (21) determines that the information retrieved from the graph-based memory (GBM) and the vector-embedded memory (VEM) is insufficient to respond to the query.

20. A computer implemented method for generating a response to a query from a user or agent according to any one of claims 1 to 18, comprising:

20. determining, by said processor (21), the size of the retrieved chunk of text from the representations stored in said graph-based memory (GBM) and said vector-embedded memory (VEM) and, if said determined size exceeds a predefined threshold, applying a process for summarizing said retrieved chunk of text based on a large-scale language model, in particular based on the MapReduce algorithm; 20. A computer implemented method for generating a response to a query from a user or agent according to any one of claims 1 to 19, comprising:

21. The processor (21) is configured to determine a measure representative of a mapping of the chunks of text to a representation stored in the graph-based memory (GBM) and to adapt the process of generating chunks of text and determining the measure of mapping until the determined measure of mapping satisfies a predefined termination criterion.

21. A computer implemented method for generating a response to a query from a user or agent according to any one of claims 1 to 20.

22. 1. A system for generating a response to a query from a user or agent, comprising: a vector embedded memory (VEM) configured to store the vectors and the generated chunks of natural language text; a graph-based memory (GBM) configured to store information in the form of nodes interconnected by links; A processor (21), searching for matching vectors from the vector embedding memory (VEM) based on semantic closeness between the vectors stored in the vector embedding memory (VEM) and vectors generated based on the chunks of the natural language text of the retrieved query; determining and activating nodes stored in the graph-based memory (GBM) based on their correspondence with the matching vectors, and determining and activating connected nodes based on links between the activated nodes; generating a response to the query based on chunks of text determined based on the matching vectors from the vector embedded memory (VEM) and further based on chunks of text determined that correspond to the activated nodes of the graph based memory (GBM); and a processor (21) configured to perform the steps.

23. 23. A system comprising an agent and a system according to claim 22 for generating a response to a query from said agent, said agent being an autonomous agent.

24. 24. The system of claim 23, wherein the autonomous agent is configured to generate and output queries based on a large language model, and to perform behavior planning of the autonomous agent based on the responses received from the system, in particular with regard to relevant stakeholder preferences and causal chains of possible actions, to generate responses to the queries of the autonomous agent.

Images