Graph-based memory expansion for large-scale language models

Abstract

To provide assistance systems and methods with memory engines that store and retrieve information described and accessed as natural language text.SOLUTION: This method comprises: obtaining and processing, by a natural language processing module, instructions in natural language; generating chunks of text based on the obtained instructions; generating, by a text embedding module (TE), vectors based on the chunks of text generated from the obtained instructions; storing, by a vector embedding memory (VEM), the generated vectors in association with the generated chunks of text; searching the vector embedding memory to determine nodes corresponding to matching vectors; activating corresponding nodes stored in a memory (GBM); selecting the activated nodes for further processing; determining chunks of text corresponding to the activated nodes; generating a response to a query based on the determined chunks of text; and outputting the response.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] This disclosure is in the field of assistance systems and assistance methods using a memory engine that stores and retrieves information described and accessed in natural language. Specifically, techniques for providing the ability to generate responses to queries in dynamic conversations are disclosed.

Background Art

[0002] In systems that store, retrieve, and use information described and accessed in natural language, such as assistance systems, general-purpose memory engines are used. Current implementations of such systems often use deep neural networks (DNNs), particularly large language models (LLMs), i.e., DNNs trained based on a large corpus of natural language (NL) text. LLMs have a large amount of knowledge about general facts and common-sense statements that become available from the initial training using sources of natural language text. The strength of LLMs lies in their natural language user interface and their ability to incorporate human common sense to resolve language-based uncertainties and bridge the gaps that can arise with queries with insufficient constraints from the user.

[0003] However, there are several problems with current LLMs. Current LLMs exhibit deficiencies in terms of the accuracy, reliability, transparency, privacy, and flexibility of the information used within the LLMs and the information output by the LLMs for queries provided by the user. LLMs are particularly weak with respect to the management of additional data, such as personal data or proprietary data. Since the processing of natural language-based text by LLMs uses a type of statistically inferred likelihood-based completion, LLMs do not always function for the processing of extremely important applications.

[0004] For example, in use cases that involve dynamically generating additional data, such as when increasing the memory of an agent by, for example, measurements in the real world, special extensions of LLMs are currently used.

[0005] One available extension is to use LLM as an interface to connect LLM to a backend database that contains additional data, such as sensitive information, in a traditional memory structure. By using a traditional backend database in combination with LLM, it is possible to combine the strengths of LLM, including a natural language user interface, language-based uncertainty resolution, and the incorporation of human common sense to fill in inherent gaps in queries, with the strengths of a more canonical, schema-based database. The backend database has the advantages of high reliability, traceability, and unique access to sensitive information. For example, using LLM as an interface to a backend database to generate queries in a language specific to the backend database containing additional data can lead to bottlenecks due to limited pre-built queries or limitations in query generation.

[0006] An alternative approach to extend LLM to address at least some of its current shortcomings involves a less canonical, database-based hybrid structure that utilizes a different natural language-oriented memory storage system, including language embeddings of text chunks. This approach uses language embeddings to find a pre-selection of relevant text chunks (fragments) based on the determined similarity between the query and the text chunks in the embedding space. The pre-selection provides identifiable text chunks that contain text that is semantically similar to the query and therefore considered valid for answering the query. Together with the initial query, the (pre)selected text chunks are passed to LLM to generate an answer to the query. A known problem with this hybrid structure approach is that it relies solely on embedding similarity for the pre-selection of valid text chunks. Other memory items that may be relevant to the query but do not contain semantically similar words, for example, to contain supplementary information, are not considered for generating the response.

[0007] It is desirable to continuously and incrementally improve the increasing proprietary memory without requiring the costly retraining and fine-tuning typical of LLMs. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] Considering the above considerations, we need to address the challenge of improving natural language-based information retrieval and inference from memory containing unique relational information. [Means for solving the problem]

[0009] This computer implementation method and corresponding system provide a favorable solution to this problem and similar problems. Dependent claims define further advantageous embodiments.

[0010] A computer implementation method generates responses to queries from a user or agent within a system, with the agent generating the response. The system comprises a vector-embedded memory configured to store vectors (embedded vectors) and chunks of natural language text, a graph-based memory configured to store information in the form of nodes interconnected by links, and a processor. The method includes the steps of: searching for a matching vector in the vector-embedded memory based on the semantic proximity between the vectors stored in the vector-embedded memory and the vectors generated based on the chunks of natural language text of the retrieved query; determining and activating nodes stored in the graph-based memory based on the correspondence with the matching vectors, and determining and activating connected nodes based on the links between the activated nodes. The method further includes the step of generating a response to a query based on the chunks of text determined based on the semantic proximity between the matching vectors from the vector-embedded memory and the vectors generated based on the chunks of natural language text of the query. The step of generating the response generates the response to the query based on the determined chunks of text corresponding to the activated nodes from the graph-based memory.

[0011] The computer implementation method according to the first embodiment has several advantages. In detail, the method provides the ability to make additional information available to the question answering system and gives the question answering system full access to its own relational information without requiring extensive retraining of the question answering system.

[0012] Embedding systems known in this art consider each memory item to be independent of other memory items, thus constituting conventional indexed memory. This computer implementation overcomes this by providing memory related to real-world memory items using a graph structure that provides a web of interconnected concepts. In an example inspired by human memory, a particular event, such as a concert, may be associated with a date via the date of the concert, a location via the venue of the concert, friends who went with the person to attend the concert, mood, experience, musical taste, and bad weather. All these elements, and other similar elements, are related to one another in memory, but each element represents a semantically distinct and seemingly unrelated memory item. Similarly, notes in an individual's knowledge graph may be linked to various related nodes, and those related notes provide background information related to the note in question. As a result, this computer implementation offers a unique advantage over existing indexed memory in that it improves the quality of the retrieved memory items (which later play a role in generating responses to queries) in the form of chunks of text, by taking into account the relationships between memory items and between chunks of text. Instead of indexed memory as in known methods, this computer implementation uses an extensible knowledge graph over sensitive memory to generate responses to queries, retrieving further relevant information by following related relationships within the graph.

[0013] Current techniques for extending LLMs through additional training and fine-tuning essentially involve incorporating implicit information about the knowledge network that is static (fixed) and predetermined only by the text and language statistics used during LLM training. Currently, because retraining or fine-tuning an LLM with proprietary information for specific purposes is extremely costly, integration using an interface with external memory is now a feasible approach.

[0014] The computer implementation method according to the first aspect provides a solution that overcomes the limitations of known methods. Unique external information is not simply composed of isolated fragments of information and chunks of text. The relationships connecting these fragments of information capture a large proportion of unique external information. Returning to the example of a person's personal memory of attending a concert with friends, the entire description of this event, the people who went, their relationships, etc., may be contained in a single chunk of text. If this chunk is converted into an embedding vector, this information is likely to be found for queries such as "Who did X go to event Y with?". In real-world applications, relevant information can be expected to be distributed across chunks of text that include, to name a few, (i) friends and their relationships, (ii) concerts, performers, locations, dates, etc., (iii) plans to attend the concert on that particular day, and (iv) plans to spend time with that friend on that particular day. By connecting all these fragments of information using graph-based memory in addition to vector-based embedding memory, if we have information that someone is going to a concert on a given day, plans to spend time with someone on that day, and that someone is a friend, then it becomes possible to infer that the person going to the concert with must be that friend. The web of links and nodes in graph-based memory that covers the relationships between individual pieces of information makes it possible to infer this conclusion, which is a benefit from the additional information contained in the graph-based representation that is stored in graph-based memory and retrieved from there, as it is not contained in a single piece of information or chunk of text.

[0015] The current index-based proprietary memory may be sufficient if the granularity of the text chunks matches the granularity of the query. This effectively means that most of the information needed to answer the query is contained within a single chunk of text. However, if the query requires gathering information scattered across different chunks of text, and not all of it has an embedding vector for the query's embedding, the graph-based representation of this computer implementation method is advantageous in providing a more satisfactory response by performing an information retrieval process based on that representation.

[0016] A computer implementation method according to one embodiment includes augmenting the generated response by referencing chunks of text used as the basis for generating the response, in order to enable complete traceability.

[0017] Embodiments defined by a dependent claim include a computer implementation method for generating a response to a query from a user or agent according to a first aspect, the method comprising: a natural language processing module obtaining and processing a natural language instruction from a user to generate chunks of text; a text embedding module generating vectors based on these chunks of text; a vector embedding memory storing the vectors generated by the text embedding module in association with the generated chunks of text; a processor searching the vector embedding memory to determine matching vectors based on their similarity to the vectors of a query obtained from a user or agent; and a vector search module generating a first ranking list of the determined matching vectors, using the vectors stored in the vector embedding memory and the vectors generated by the text embedding module based on the chunks of natural language text of the query obtained and processed by the natural language processing module. The method further includes the steps of: searching for a stored vector from a vector embedding memory based on the semantic proximity of the vectors; determining a node stored in a graph-based memory based on the correspondence between the determined vector and a first ranked list of matching vectors, and generating a second ranked list of the corresponding nodes, wherein the graph-based memory stores information in the form of nodes interconnected by links, each node containing a specific chunk of text representing a certain concept, each link being located between a source node and a target node, each link representing a relationship between the concept of the source node and the concept of the target node, and the method further includes the steps of: activating the corresponding node stored in the graph-based memory by the processor, and activating the connected node based on the link between the activated node and other nodes in the graph-based memory by applying a graph-crossing method, such as a random walk or a graph-crossing algorithm such as a personalized PageRank, and the processorThe process includes the steps of: generating a third ranking list containing activated nodes stored in graph-based memory; selecting activated nodes from the third ranking list for further processing; determining chunks of text corresponding to the activated nodes in the third ranking list; and having a natural language processing module receive the determined chunks of text corresponding to the activated nodes in the third ranking list, and generating a response to the query based on the determined chunks of text determined based on the semantic proximity between a vector stored in vector embedding memory and a vector generated by the text embedding module based on chunks of natural language text of the query, and the determined chunks of text corresponding to the activated nodes in the third ranking list for further processing.

[0018] The agent may be an agent that constitutes the system. Alternatively, the agent may be a different agent from the agents that constitute the system.

[0019] Therefore, the computer implementation method provides a response to a query by a user or agent, based on determined chunks of text that correspond to activated nodes in a third ranking list for further processing, which are additional background information.

[0020] A computer implementation method for generating a response to a query from a user or agent according to one embodiment includes, by a vector search module, searching for a vector from vector embedding memory based on the semantic proximity of the vectors stored in the vector embedding memory, in combination with keyword-based searching.

[0021] According to one embodiment of a computer implementation method for generating responses to queries from a user or agent, the method includes pre-loading vector-embedded memory and graph-based memory with data in parallel during the training phase of the system.

[0022] Instead, a computer-implemented method for generating a response to a query from a user or an agent includes dynamically loading data into a vector embedding memory and a graph-based memory during an operating phase of the system.

[0023] In a computer-implemented method according to one embodiment, each node includes a chunk of natural language text representing an abstracted concept.

[0024] A computer-implemented method according to one embodiment includes continuously maintaining, by a processor, a correspondence between vectors stored in a vector embedding memory and corresponding nodes stored in a graph-based memory.

[0025] According to one embodiment of the computer-implemented method, the nodes stored in the graph-based memory correspond to concepts that are supersets of the concepts corresponding to the vectors stored in the vector embedding memory, and the graph-based memory stores additional information regarding a set of concepts (a set of general concepts) in the form of relationships between the concepts of the set of concepts and additional concepts for which there are no corresponding vectors in the vector embedding memory.

[0026] In mathematics, a set A is a subset of a set B if every element of A is also an element of B, and then B is a superset of A. A and B may be equal, and if not, A is a proper subset of B. The relationship that one set is a subset of another is called inclusion or sometimes containment. That A is a subset of B can also be expressed as B includes A or A is included in B. A k-subset is a subset having k elements.

[0027] A computer-implemented method according to one embodiment includes storing, in the graph-based memory, additional information regarding additional concepts that go beyond the concepts corresponding to the vectors stored in the vector embedding memory.

[0028] A computer-implemented method according to one embodiment includes analyzing, by a natural language processing module, a question, query, or instruction obtained from a user or an agent, and based on the analysis, determining whether there is an intention of the user to access the stored memory content of the knowledge base and store new memory content in the knowledge base, that is, separating the new memory content of the user. The text embedding module is further configured to convert a generated text chunk that describes the new memory content into a vector that describes the new memory content.

[0029] Thus, this embodiment can determine whether a user or an agent may want to search for information from the memory or store additional information in the memory. The system can make the determination of intention based on, for example, the analysis of a query or instruction by a natural language processing module. The method may be used for both adding information to the knowledge base and searching for information from the knowledge base.

[0030] A computer-implemented method according to one embodiment includes analyzing, by a natural language processing module, a question, query, or instruction obtained from a user or an agent. In an activation step by a processor, based on the analysis, a corresponding node stored in a graph-based memory is activated, and based on links between the activated node and other nodes of the graph-based memory, connected nodes are activated. In particular, activating a corresponding node stored in a graph-based memory based on the analysis and activating connected nodes based on links between the activated node and other nodes of the graph-based memory includes restricting the activation of the connected nodes to a selected subset of link types and node types based on the analysis.

[0031] In this embodiment, if the analysis of a query yields results indicating a specific type of question, such as one relating to location, time, or person, the system determines and activates nodes in graph-based memory, for example, by focusing on the spatial, temporal, or personal type of links and nodes, respectively. Thus, the retrieval of information from graph-based memory becomes more targeted, speeding up the search, reducing the processing resources required, and improving the results.

[0032] Analyzing a question, query, or instruction obtained from a user or agent may involve a natural language processing module searching for keywords containing words such as “where,” “when,” “who,” and “how” to determine whether the question is of a particular type relating to a place, time, or person. According to one embodiment of a computer implementation, the method includes generating a first ranked list of matching vectors for responding to a query obtained from a user or agent from vectors stored in vector embedding memory, the first ranked list including, for each matching vector, a measure of similarity or a measure of relevance or an evaluation of similarity.

[0033] A computer implementation method according to one embodiment uses a vector-embedded memory that stores the contents of at least one further modality. The method may include generating a response to a query based on a matching vector retrieved from the vector-embedded memory, and further based on the stored contents of at least one further modality.

[0034] Therefore, by optionally storing the contents of further modalities in memory and creating embedding vectors in the same embedding space, and then searching for the results of various modalities from the vector embedding memory, multi-mode search results can be provided.

[0035] Therefore, by using an appropriate embedding model, the method can also embed input from modalities other than text content, such as image content or audio recordings. This makes it possible to find and extract not only chunks of text, but also chunks of information from various modalities.

[0036] One embodiment of a computer implementation method includes a perceptual module dynamically constructing a knowledge database stored in vector-embedded memory and graph-based memory, the knowledge database comprising multiple chunks of text stored in vector-embedded memory and multiple nodes and links stored in graph-based memory.

[0037] A computer implementation method according to one embodiment includes a processor performing a synchronization process, the synchronization process including at least one of comparing a representation stored in vector-embedded memory with a corresponding representation stored in graph-based memory, and deriving the representation stored in vector-embedded memory and the corresponding representation stored in graph-based memory from a predetermined data source.

[0038] A computer implementation method according to one embodiment includes a processor determining the type of query based on an analysis of text chunks, and the processor activating nodes stored in graph-based memory based on the determined query type. In particular, the method may include the processor activating selected link types or node categories stored in graph-based memory based on the determined query type.

[0039] A computer implementation method according to one embodiment includes a processor determining constraints for refining a query based on an analysis of text chunks, and the processor activating nodes stored in graph-based memory based on the determined query type. In particular, the method may include the processor activating nodes stored in graph-based memory that are connected via selected link types or node categories, based on the refined query.

[0040] Therefore, by performing preprocessing, including intent analysis of the retrieved queries, and then appropriately adjusting the graph-based augmentation using the knowledge representation stored in graph-based memory, the search for appropriate information fragments based on the graph can become more targeted. Thus, processing becomes more efficient, and at the same time, the quality of information retrieved from the graph improves. For example, if a query asks a question that includes a "where" clause, the type of query being determined may be a location query. Furthermore, the search in memory based on the graph may be constrained by the spatial relationships within the links.

[0041] According to one embodiment, a computer implementation method includes a processor dynamically adjusting the number of chunks of text to be retrieved when activating corresponding nodes stored in graph-based memory. In particular, the method may dynamically adjust the number of chunks of text to be retrieved from representations stored in graph-based memory in order to avoid reaching a token limit. The method may include a processor determining whether the information retrieved from graph-based memory and vector-embedded memory is sufficient to respond to a query. If the method then determines that the information retrieved from graph-based memory and vector-embedded memory is insufficient to respond to a query, it may dynamically adjust the number of chunks of text to be retrieved from graph-based memory.

[0042] Therefore, post-processing LLM inference is optimized, resulting in reduced processing complexity and a response adapted to the response requirements. For example, the method can avoid reaching token limits, or by separately analyzing whether the retrieved information is sufficient to answer the query, or whether it is recommended to retrieve more information to provide a sufficient answer to the query.

[0043] A computer implementation method according to one embodiment includes a processor determining the size of chunks of retrieved text from representations stored in graph-based memory and vector-embedded memory, and if the determined size exceeds a predetermined threshold, applying a process to summarize the chunks of retrieved text based on a large-scale language model, particularly based on the MapReduce algorithm.

[0044] Therefore, the amount of data collected and used to generate responses is limited, thereby limiting the computational complexity of subsequent processing steps.

[0045] MapReduce is a programming model and associated implementation for processing and generating large datasets using a group of processing units with independent, and possibly parallel, processing algorithms. MapReduce involves mapping the same function to several inputs, enabling independent, and possibly parallel processing, followed by a summarization step. This summarization step may be performed in parallel on several graphics processing units (GPUs). Alternatively, a variant of MapReduce may be used to incrementally add information by retrieving further information from a knowledge base until the generated response to the query is deemed accurate.

[0046] According to one embodiment, a computer implementation method includes a processor determining a scale representing the mapping of chunks of text to representations stored in graph-based memory, and adapting the process of generating chunks of text and determining the mapping scale until the determined mapping scale satisfies a predetermined termination criterion.

[0047] The system according to the second embodiment provides the corresponding advantages described with respect to the computer implementation method according to the first embodiment.

[0048] In a second embodiment, an agent system for generating responses to queries from a user or agent comprises a vector-embedded memory configured to store vectors and chunks of generated natural language text; a graph-based memory configured to store information in the form of nodes interconnected by links; and a processor. The processor is configured to search the vector-embedded memory for matching vectors based on the semantic proximity between vectors stored in the vector-embedded memory and vectors generated based on chunks of natural language text of the retrieved query; to determine and activate nodes stored in the graph-based memory based on the correspondence with the matching vectors; to determine and activate connected nodes based on the links between the activated nodes; and to generate a response to the query based on chunks of text determined based on the semantic proximity between matching vectors from the vector-embedded memory and vectors generated based on chunks of natural language text of the query. The processor further generates the response based on the determined chunks of text corresponding to the activated nodes in the graph-based memory.

[0049] A system comprising an agent and a second-mode system for generating responses to queries from the agent is also advantageous, wherein the agent is an autonomous agent.

[0050] In detail, the autonomous agent is configured to generate and output queries based on a large-scale language model, and to plan the autonomous agent's behavior based on the responses received from the system in order to generate responses to the autonomous agent's queries.

[0051] Therefore, the responses to queries received from the system may include, for example, information about the preferences of relevant stakeholders and the causal chain of possible actions planned for execution by autonomous agents.

[0052] In this embodiment, the autonomous agent is configured to generate queries based on a large-scale language model and process them internally, and to plan its behavior based on responses received from the system (which serves as the autonomous agent's memory) in order to generate responses to the autonomous agent's queries. The memory can store information related to decision-making and robot planning, such as memory for actions that capture causal chains, not limited to graph-based memory, and information about potential outcomes, capabilities, means, or agents involved in a given action, not limited to text-based descriptions.

[0053] In this case, the agent is not using the proposed system to generate a response for the user who presented the query or for another agent different from the agent in question. The autonomous agent queries its own internal memory as a kind of internal monologue to retrieve specific information for reasoning, planning, or executing an action.

[0054] Therefore, the system and corresponding methods are particularly useful for planning the behavior of autonomous agents, such as robots. This technique can help retrieve relevant information for a robot's decision-making process. For example, a human user's preferences may be stored in specific text (e.g., "Paul likes to cuddle with cats"). If a robot presents a query to the system that corresponds to "How can we cheer Paul up?", the generated response output to the system may be based on the preferences in the stored text mentioned above.

[0055] Many interconnected memories relate to activities or actions. These interconnected memories may store information that represents causal chains. Examples of such causal chains may include reasoning similar to "something was observed, then something else happened, and as a result...". Causal chains stored in interconnected memories may also include direct information about consequences, abilities, means, or agents that may be involved in a particular action. This computer implementation method is suitable for storing and retrieving information about such causal chains, which is essential for performing predictive reasoning, including planning the actions of autonomous agents.

[0056] The following description of the embodiments will be shown in the following figure. [Brief explanation of the drawing]

[0057] [Figure 1] This is a schematic flowchart of a computer implementation method according to one embodiment. [Figure 2] This figure shows an overview of the graph-vector-text memory architecture of the proposed system in one embodiment, as well as its data input and usage. [Figure 3] This is a schematic diagram of the architecture of a search process that utilizes relational information in one embodiment. [Figure 4] This is a schematic diagram illustrating the highly abstracted architecture of the proposed system. [Modes for carrying out the invention]

[0058] In the diagrams, corresponding elements share the same reference numeral. Where possible, the diagram descriptions should avoid explaining the same reference numerals in different diagrams without negatively impacting comprehension, and should also avoid unnecessary repetition for the sake of brevity.

[0059] This disclosure refers to vectors, specifically dense vectors. A dense vector is a high-dimensional vector containing, for example, three-digit numbers in its dimensions. Each dimension contains relevant information, which can be determined by a neural network. Due to their high dimensionality, these vectors are more difficult to compress and therefore typically require more memory than sparse vectors.

[0060] Vectors define an embedded vector space that is fitted to capture the semantic proximity between vectors.

[0061] This application uses word tokens to represent words in natural language utterances that are mapped to vectors of embedding values ​​for nodes in a knowledge graph. Tokens may be part of a word, a word, or a sequence of words. Typically, one token corresponds to four characters.

[0062] This invention uses known word token representations as embedding vectors for natural language text. Tokens may be parts of words, words, or sequences of words. More specifically, tokens may refer to parts of words. Before processing prompts, the input is broken down into tokens. These tokens are not cut precisely where words begin or end. Tokens may include trailing spaces and even subwords. One token often refers to four letters in English, and consequently, one token corresponds to approximately three-quarters of a word, or 100 tokens correspond to approximately 75 words.

[0063] The term "token limit" refers to the maximum number of tokens that an LLM can process. The token limit is the sum of the input and output tokens.

[0064] In knowledge representation and reasoning, a knowledge graph is a knowledge base that integrates data using a graph structure data model or topology. Generally, a knowledge graph can store interconnected descriptions of entities—i.e., objects, events, situations, or abstract concepts—while encoding the meanings underlying the terms used. More specifically, a knowledge graph can represent a digital structure that represents knowledge as concepts and the relationships (facts) between those concepts. A knowledge graph can include explicit designations of concepts, enabling both humans and machines to understand and discuss its contents.

[0065] This computer implementation method may be particularly advantageous for agents, including virtual agents, which are computer-generated agents capable of interacting with human users. Task-oriented virtual agents can communicate with human users in natural language, collaborate with users, and assist users in performing various tasks.

[0066] A natural language understanding (NLP) system interprets sequences of words in a user's utterance. This NLP system is used by a task-oriented virtual agent.

[0067] This computer implementation method and system extends the advantages of graph databases, which store information in the form of heterogeneous graphs, where nodes (conceptual nodes) represent entities or concepts, and edges represent relationships between entities or concepts. In graph databases, relationships, such as heterogeneous connections between entities, are first-class citizens. Using graph structures and flexible schemas, graph databases enable a more efficient and expressive way to handle higher-order relationships between distant entities than relational databases, particularly navigating data structures that include multi-hop hierarchies. While traditional databases require costly join operations to retrieve information, graph databases can use the adjacency matrix of the graph database to directly traverse the graph and navigate links more efficiently.

[0068] An increasing amount of personal information is available in digital format. Access to this information source has been greatly facilitated by recent advancements in LLM and augmentation techniques such as retrieval augmented generation (RAG), which enables the development of chatbots for personal or private data. However, even when combined with keyword search, methods built around semantic similarity using embeddings are inherently limited by those search techniques that do not consider relational information. In contrast, the computer implementation methods disclosed provide such relational information and leverage it to identify additional relevant context, enabling deeper answers to questions that go far beyond the content of individual documents or isolated chunks of text, revealing a greater understanding of the subject of the query.

[0069] Specific examples of knowledge management systems, though not limited to those listed above, include systems that use graph-based knowledge bases such as Notion to respond to natural language queries, and chatbots for searching information about research papers that can also utilize information not explicitly included in the papers themselves. For example, relational information such as authors' affiliations may provide further insights. The following diagrams describe advantageous implementations of methods for responding to queries. Further advantageous application areas are summarized at the end.

[0070] Figure 1 shows a schematic flowchart of a computer implementation method according to one embodiment for generating a response to a query.

[0071] The method first involves step S1, in which a natural language processing module (NLP) obtains and processes natural language commands from the user and generates text chunks based on them. The natural language commands may correspond to one or more queries from the user or an agent. The agent may be a virtual agent or a real-world agent.

[0072] The Natural Language Processing (NLP) module may be a combination of multiple modules. The NLP module is configured to analyze and process natural language text to generate input for other modules.

[0073] The Natural Language Processing module (NLP) may be based on an LLM such as ChatGPT, which may be suitable for performing tasks involving natural language processing for intent classification. For intent classification, the NLP identifies the intent associated with the natural language (NL) query. In detail, the NLP primarily isolates the questions within the query that should receive a response. Furthermore, the NLP may identify the statements that should be stored in the system's memory 23.

[0074] In addition, the natural language processing module (NLP) may extract information by processing the NL statement, for example, through named entity recognition and named entity linking, in order to store the statement in memory 23.

[0075] The Natural Language Processing (NLP) module can also perform query generation (NLPQP). To perform NLPQP, the NLP module takes NL text as input and then converts it into a query, such as a Cypher.

[0076] System 20 may include a chunking module (CH). The chunking module CH divides the text input into manageable chunks of text (text chunks), each of which is smaller in size and optimized to meet the token limits of the text embedding module TE and the response generation module RGM, as well as the respective capabilities of the text embedding module TE and the response generation module RGM to capture the meaning of the text and generate response text.

[0077] The chunking module CH further concatenates the text chunks generated from the query received from the user or agent with selected natural language chunks, and passes the concatenated text chunks to the response generation module RGM.

[0078] In step S2, the method then generates a vector based on chunks of text generated based on the acquired instructions, using the text embedding module TE.

[0079] The Text Embedding Module (TE) accepts natural language text as input and processes it based on statistical correlations. These statistical correlations may be implicitly included in the neural network (NN) representation. The Text Embedding Module (TE) can also generate vector embeddings (VEs) or vectors, which can be stored in memory and compared with other vectors for similarity.

[0080] In step S3, the method may store the vector generated by the text embedding module TE in the vector embedding memory VEM, associated with the generated chunks of text.

[0081] Although not explicitly shown in Figure 1, the method may involve storing the generated vectors in vector-embedded memory (VEM), and further storing the corresponding nodes and the links between those corresponding nodes in graph-based memory (GBB).

[0082] The structure of the vector-embedded memory (VEM) will be described in detail with reference to the graph-vector-text memory (GVT) in Figure 2.

[0083] In step S4, the method includes the processor 21 searching the vector embedding memory VEM to determine matching vectors based on their similarity to the vectors of the query obtained from the user or agent, and the vector search module VR searching the vectors stored in the vector embedding memory VEM based on the semantic closeness between the vectors stored in the vector embedding memory VEM and the vectors generated by the text embedding module TEM based on the chunks of natural language text of the query obtained and processed by the natural language processing module NLP, in order to generate a first ranked list of the determined matching vectors.

[0084] Step S5, following step S4, includes the processor 21 determining the nodes stored in the graph-based memory GBM based on the correspondence between the determined vectors in a first ranking list of matching vectors, and generating a second ranking list of the corresponding nodes. The graph-based memory GBM stores information in the form of nodes interconnected by links, each node containing a specific chunk of text representing a certain concept, each link being located between a source node and a target node, and each link representing the relationship between the concept of the source node and the concept of the target node.

[0085] In step S6, the method then has the processor 21 activate the corresponding node stored in the graph-based memory GBM and activate further connected nodes based on the links between the activated node and other nodes in the graph-based memory GBM by applying a graph-crossing algorithm. The graph-crossing algorithm may include a random walk or a individually configured PageRank.

[0086] In step S7, the processor 21 further generates a third ranked list containing the activated nodes stored in the graph-based memory GBM, selects the activated nodes from the third ranked list for further processing, and determines the chunks of text corresponding to the activated nodes in the third ranked list.

[0087] In step S8, the method then receives determined text chunks corresponding to the activated nodes in a third ranking list via the Natural Language Processing module (NLP). The Natural Language Processing module (NLP) generates a response to the query based on the determined text chunks, which are determined based on the semantic proximity between the stored vectors in the vector embedding memory (VEM) and the vectors generated by the text embedding module (TEM) based on the natural language text chunks of the query, and the determined text chunks corresponding to the activated nodes in the third ranking list for further processing.

[0088] In detail, a question-answering module (QA), implemented using a natural language processing module (NLP), generates a response to a query based on the given context of chunks of text corresponding to activated nodes in a third ranked list. The question-answering module (QA) may be implemented using a specially trained language model or even a general-purpose language model (LLM).

[0089] Since the original entry point of the search and the search propagation process are known, the response can be easily extended with source citations, enabling complete traceability.

[0090] The generated response is then output to the user who submitted the original query or the agent providing the query. The response is in natural text, particularly in audible or visual form.

[0091] Figure 2 shows an overview of the architecture of the graph-vector-text memory (GVT) of the proposed system in one embodiment, as well as its data input and usage.

[0092] In the center of Figure 2, the graph-vector-text memory GVT is depicted.

[0093] Graph-vector-text memory (GVT) includes text chunk memory (TCM), vector-embedded memory (VEM), and graph-based memory (GBM).

[0094] The GVT may be implemented as part of the memory 23 of the system 20. Alternatively or additionally, at least a portion of the graph-vector-text memory (GVT), text chunk memory (TCM), vector-embedded memory (VEM), and graph-based memory (GBM) may be implemented on one or more servers accessible via the network 27 of the system 20.

[0095] Text chunk memory (TCM) is memory used to store and retrieve text chunks.

[0096] A graph-based memory (GBM) is a database that stores a knowledge graph and retrieves concepts from it. A graph-based memory (GBM) stores a graph that may have labeled nodes and optionally labeled links. Labeled links may include, for example, the properties of the relationship on which the link is based. Nodes in the graph represent memory items, and links between nodes in the graph represent associations between nodes, with each link existing between a start node and a target node.

[0097] More specifically, the graph-based memory (GBM) stores information in the form of nodes interconnected by links, each node containing a specific chunk of text representing a certain concept, each link being located between a source node and a target node, and each link representing the relationship between the concept of the source node and the concept of the target node.

[0098] A graph-based memory database (GBM) can be implemented using a knowledge graph (KG) within a graph database such as Neo4j.

[0099] Vector-embedded memory (VEM) is a vector storage for storing and retrieving vectors (embedded vectors). Each vector stored in the vector-embedded memory (VEM) maintains a reference to the original text chunk from which it was generated, as well as a reference to the corresponding node in the graph-based memory (GBM).

[0100] Therefore, vectors stored in vector-embedded memory (VEM) can, for example, trigger nodes stored in graph-based memory (GBM).

[0101] An implementation of vector-embedded memory (VEM) may be based on an existing vector store or database, including ChromaDB or Pinecone.

[0102] In a proof-of-concept for storing small chunks of text, graph-vector-text memory (GVT) is implemented, including text chunk memory (TCM), vector embedding memory (VEM), and graph-based memory (GBM), as well as the TCM and VEM modules, via Neo4j. Neo4j includes a graph database that supports the storage of vector embeddings, as disclosed in Hasbe, S.: Neo4j's Vector Search: Unlocking Deeper Insights for AI-Powered Applications, 2023 (https: / / neo4j.com / blog / vector-search-deeper-insights), and enables searching by embedding similarities using an implementation of the HNSW algorithm.

[0103] The queries retrieved by system 20 include natural language text (plain text) representing the input to system 20.

[0104] The text embedding module TE generates a vector (embedding vector) based on the retrieved query.

[0105] The generated vectors provide the basis for similarity searches within a vector representation that includes multiple vectors stored in the vector embedding memory (VEM).

[0106] The results of a similarity search within a vector representation include vectors that satisfy the similarity criteria of the similarity search. On the one hand, the resulting similar vectors directly correspond to (are associated with) text chunks stored in text chunk memory (TCM). On the other hand, the resulting similar vectors correspond to (are associated with) nodes in a graph representation stored in graph-based memory (GBM).

[0107] The nodes of the graph-based representation associated with the obtained similar vectors form the basis for association propagation searches for the graph representations stored in the graph-based memory (GBM).

[0108] The association propagation search for the graph representation may be performed iteratively until the termination criteria are met. The termination criteria may be based on determining that sufficient information to respond to the query has been retrieved.

[0109] The association propagation search returns an additional node. With the returned additional node, system 20 can retrieve from the text chunk memory (TCM) any further related chunks of text stored in the text chunk memory associated with this additional node.

[0110] Subsequently, all retrieved text chunks are provided to the collection and response generation module RG. The entire retrieved text chunk includes text chunks directly retrieved by similarity search performed on the embedded memory VEM (hereinafter referred to as (retrieved) text chunks) and additionally retrieved related text chunks determined by graph propagation using the graph-based memory GBM (hereinafter referred to as further (retrieved) text chunks).

[0111] The Collection and Response Generation Module RG collects all evidence, including, in particular, chunks of the searched text and any further chunks of the searched text identified as relevant by the correlation propagation search process. The Collection and Response Generation Module RG (Response Generation Module RG) then generates a response to the retrieved query and outputs the generated response to the user or agent presenting the query.

[0112] The generated response is in natural language text.

[0113] Figure 3 shows an overview of the architecture of a search process that utilizes relational information in one embodiment of a computer implementation method.

[0114] When a query to system 20 is constructed by a user or agent and received by system 20, the received query is processed by the natural language processing module (NLP).

[0115] The Natural Language Processing (NLP) module passes the resulting text, which contains one or more chunks of text, to the Text Embedding Module (TE), which generates embedding vectors (vectors) based on those text chunks. The geometric shape of the embedding vector space is suitable for capturing the semantic closeness between the vectors that make up the vector space. The generated vectors are then used to query the Vector Embedding Memory (VEM) to find their similarity to vectors stored in the VEM. The Vector Embedding Memory (VEM) returns a ranked list of the most similar vectors stored in the VEM. The returned most similar vector represents the vector or concept that is semantically closest to the vector generated based on the text chunks of the original query. Each vector returned by the Vector Embedding Memory (VEM) contains a reference to the corresponding node in the graph-based memory (GBM). More specifically, each vector returned by the Vector Embedding Memory (VEM) contains a reference to the corresponding node in the graph-based memory (GBM), thereby triggering a specific concept corresponding to a certain node. Therefore, the node corresponding to the concept with the highest relation to the original query can be retrieved from the graph-based memory GBM based on the results of a similarity search performed on the vectors stored in the vector embedding memory. Then, starting from the node corresponding to the concept with the highest relation, further related nodes and links are retrieved from the graph-based memory GBM using the link structure stored in the graph-based memory GBM.

[0116] The most relevant nodes, including those selected directly based on similarity searches performed on vectors stored in vector embedding memory (VEM) and further nodes found through searches along links in graph-based memory (GBM), are then used to retrieve the relevant text chunks from text chunk memory (TCM) as further retrieved text chunks.

[0117] The text of the retrieved text chunks is then compiled as background evidence in natural text, combined with metadata from the original query text and graph-based memory GBM for traceability of further retrieved text chunks, and passed to the response generation module RG to generate a response based on this. More specifically, the text of the retrieved text chunks forms the basis for constructing the answer to the original query provided to system 20.

[0118] In addition, system 20 may augment the generated response by referencing text chunks used as the basis for generating the response to enable complete traceability of the text chunks, and further text chunks used as the basis for generating the response. This reference may form augmentation information that may include metadata of further retrieved text chunks, such as identifiers associated with each text chunk and the source of each text chunk. This reference is advantageous for traceability of further retrieved text chunks, and metadata such as text chunk identifiers and sources may be output to the response generation module RG to augment the generated response. The bottom of Figure 3 provides further details of the search process for finding relevant concepts in the graph-based memory GBM.

[0119] The search process first determines the nodes stored in the graph-based memory GBM based on the correspondence between the determined vectors and the first ranked list of matching vectors, and generates a second ranked list of the corresponding nodes.

[0120] The search process then continues by activating corresponding nodes stored in the graph-based memory GBM, and also by activating connected nodes based on links between the activated nodes and other nodes in the graph-based memory GBM, for example by applying a random walk algorithm or a personalized PageRank algorithm, although other types of graph algorithms such as graph crossing algorithms and activation diffusion algorithms can also be applied.

[0121] The search process then continues by generating a third ranked list containing the activated nodes stored in GB of memory based on the graph, selecting the activated nodes from the third ranked list for further processing, and determining the chunks of text corresponding to the activated nodes in the third ranked list.

[0122] The ranking of the determined vectors of the activated nodes in the first ranking list within the second and third ranking lists may be done by calculating their respective importance scores and rearranging the first, second, and third ranking lists based on the calculated importance scores.

[0123] The importance score may include a numerical value that describes the relevance of a vector or node to respond to the query currently being processed.

[0124] Then, the third ranking list represents the input to be used to determine which chunks of text correspond to the activated nodes in the third ranking list, in order to further process them, in particular, to gather information for creating a response to the query.

[0125] Figure 4 shows an overview of the advanced abstraction of the proposed system 20 architecture.

[0126] The system 20 in Figure 4 includes a processor 21, data storage 23 (memory 23), input / output interface 24, and network interface 25, which are connected by a data bus 22.

[0127] The input / output interface 24 may, in particular, provide the ability to receive queries from other agents or human users via voice or text. Thus, the input / output interface 24 may represent an interface for connecting any combination of input / output devices 26, including, but not limited to, keyboards, mice, pointing devices, displays, microphones, and loudspeakers.

[0128] The input / output interface 24 may be implemented, at least partially, within a software module executed on the processor 21.

[0129] The processor 21 may be any type of controller or processor, and may be embodied as one or more processors 21 adapted to perform the functions described herein. Where the term processor is used herein, the processor 21 may include the use of a single integrated circuit (IC), or the use of multiple integrated circuits or other components connected, arranged or grouped together, such as controllers, microprocessors, digital signal processors (DSPs), parallel processors, multicore processors, custom ICs, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), and may also include adaptive computing ICs and associated memories, such as RAM, DRAM, and ROM, as well as other ICs and components. Therefore, the term processor 21 should be understood to mean and include, equally, a single IC, or a configuration of a custom IC, an ASIC, a processor, a microprocessor, a controller, an FPGA, an adaptive computing IC, or any other group of integrated circuits that perform the functions described with respect to the computer implementation method, along with any associated memory such as microprocessor memory or additional RAM, DRAM, SDRAM, SRAM, MRAM, ROM, FLASH, EPROM, or E2 PROM. The processor 21, along with its associated memory, may be adapted or configured via programming, FPGA interconnection, or wiring to perform the methodology of the computer implementation method. For example, the method may be programmed and stored in the processor 21, along with associated memory or memory 23 and other equivalent components, as a set of program instructions or other code to be executed when the processor 21 is operating, for example, when powered on and functioning.

[0130] Memory 23 may include a data repository or database, which may be embodied in any number of forms, including in any computer or other machine-readable data storage medium, memory device, or other storage or communication device for storing or communicating information, which may include, but are not limited to, memory integrated circuits (ICs) or memory portions of integrated circuits (such as resident memory in a processor 21), which may be volatile or non-volatile, removable or non-removable, and may include RAM, FLASH, DRAM, SDRAM, SRAM, MRAM, FeRAM, ROM, EPROM or E2 PROM, or any other form of memory device, such as a magnetic hard drive, optical drive, magnetic disk or tape drive, hard disk drive, other machine-readable storage or memory medium, such as a floppy disk, CD-ROM, CD-RW, digital multipurpose disk (DVD) or other optical memory, or any other type of memory, storage medium, or data storage device or circuit. In addition, such computer-readable media include any form of communication media that embodies computer-readable instructions, data structures, program modules or other data in data signals or modulated signals such as electromagnetic or optical carriers or other transport mechanisms, and any information transmission media that can encode data or other information in wired or wireless signals, including electromagnetic, optical, acoustic, RF, or infrared signals. Memory 23 may be adapted to store various lookup tables, parameters, coefficients, other information and data, programs or instructions, and other types of tables such as database tables of the software of this disclosure.

[0131] Memory 23 may implement at least some of the graph-vector-text memory (GVT), text chunk memory (TCM), vector embedding memory (VEM), and graph-based memory (GBM). However, in alternative embodiments of the present disclosure, the text chunk memory (TCM), vector embedding memory (VEM), and graph-based memory (GBM) are implemented as a single graph-vector-text memory (GVT) that stores graph, text chunk, and embedding vector information together, rather than being separate modules of a computer program.

[0132] The processor 21 is programmed using the software and data structures of the disclosed computer implementation, for example, to perform the methodology of the present disclosure. Thus, the system and computer implementation of the present invention may be embodied as software that provides programming or other instructions as described above, such as a set of instructions and / or metadata embodied in a computer-readable medium. In addition, metadata may be used to define various data structures of lookup tables or databases. Such software may, without limitation, be in the form of source code or object code, for example. Source code may further be compiled into some form of instructions or object code (including assembly language instructions or configuration information). The software, source code or metadata of the present invention may be embodied as any kind of code, such as C, C++, Python, SystemC, LISA, XML, Java, Brew, SQL and its variants (e.g., SQL99 or a proprietary version of SQL), DB2, Oracle, or other kinds of programming languages ​​that perform the functions described herein, including various hardware definition or hardware modeling languages ​​(e.g., Verilog, VHDL, RTL) and the resulting database files (e.g., GDSII). As a result, as used herein by equivalent, “structure,” “program structure,” “software structure,” or “software” means and refers to any programming language of any kind having any syntax or signature that provides or is interpreted to provide any associated functionality or methodology (for example, when instantiated or loaded into a processor or computer including processor 21 and executed).

[0133] The software, metadata, or other source code and the resulting bit files (object code, database, or lookup tables) of the present invention may be embodied as computer-readable instructions, data structures, program modules, or other data in any tangible storage medium, such as any computer or other machine-readable data storage medium as described above with respect to memory 175, including, for example, floppy disks, CD-ROMs, CD-RWs, DVDs, magnetic hard drives, optical drives, or any other type of data storage device or medium.

[0134] The processor 21 may execute software, which controls the processor 21 to perform functions such as a natural language processing module (NLP), a text embedding module (TE), a chunking module (CH), and a question answering module (QAM).

[0135] The network interface 25 provides the system 20 with the ability to connect to external databases and servers via the communication network 27. In particular, the network interface 25 enables the spatially distributed implementation of the system 20 by performing at least some of the individual method steps at least partially remotely from the system 20.

[0136] In particular, the description of Figures 1 to 3 highlights the features of this disclosure by referring to a specific embodiment of the system 20 that generates natural language responses to queries presented in natural language by a user or agent. The embodiments described represent one specific application example.

[0137] Agents operating in the real world may have a digital form (virtual agent) or a materialized form (physical agent). Both virtual and physical agents require the ability to store knowledge about their environment, common sense, and personal memories and beliefs. System 20 of this disclosure can store information specific to each individual agent. This information may originate from the hardware, but it may also be found in the personal preferences of the agent's human owner (user).

[0138] In addition, virtual and physical agents benefit from the ability to continuously update their knowledge during the operational phase after the initial training phase. Information for updating an agent's knowledge base is often available in text, or at least in text form, often in natural language text. Information for updating an agent's knowledge base may include at least some of the following: descriptions of objects useful or at least relevant to addressing tasks and their purposes, human descriptions including characteristics and preferences, and a history of actions that have occurred in the agent's environment.

[0139] The computer implementation method and system 20 may be used to incrementally update the knowledge base of an agent, whether a physical or virtual agent. The proposed memory structure architecture and the associated management of the memory architecture assist in incrementally updating the pre-trained knowledge base during the agent's operational phase.

[0140] Specific examples of such agents include, but are not limited to, digital agents such as Apple's Siri, and agents involving robotic embodiments such as personal assistance robots.

[0141] The computer implementation method and system 20 assist in the process of automating workflows that humans repeatedly perform using their own computers, and enhance the ability of embodied agents to assist persons with mental or physical disabilities by anticipating their intentions and proactively performing useful actions after the planning process of assistive actions has been executed. These intelligent agents, which are both virtual and physical agents, can benefit from the incremental updating of their respective knowledge bases. By incorporating system 20 and the computer implementation method into embodiments of such agents and assistive systems, a favorable field of application of this disclosure is derived.

[0142] All steps performed by the various entities described in this disclosure, and functions described as being performed by the various entities, are intended to mean that each entity is adapted or configured to perform its respective step and function.

[0143] In the claims and detailed description, the phrase "equipped with" does not preclude the presence of other elements or steps. The indefinite article "a" or "an" does not preclude the plural.

[0144] A single element or other unit may perform the functions of several entities or items described in the claims. The mere fact that different dependent claims describe specific means and functions of a control circuit does not preclude the fact that combinations of those means and functions cannot be combined as a favorable implementation.

[0145] Terms, abbreviations and definitions ChatGPT Chatbot Generative Pre-trained Transformer CH chunking DNN (Deep Neural Network) EM Embedding Model: A neural network specifically trained to create vector embeddings for a model, such as text. EV Embedding Vector GBM graph-based memory GNN (Graph Neural Network) GPT Generative Pre-trained Transformer: A type of LLM GVT Graph Vector Text (Memory) KG Knowledge Graph Large-Scale Language Model (LLM) LTM Long-Term Memory NEL named entity linking NER named entity recognition NL natural language NLP (Natural Language Processing) Natural language processing for NLPRP response generation Natural language processing for NLPQP query creation NN (Neural Network) MTM Mid-Term Memory QA questions and answers RAG Search Augmentation Generation RGM Response Generation Module STM Short-Term Memory TCM text chunk memory TE text embedding VE Vector Embedding VEM Vector Embedded Memory VS Vector Store

Claims

1. A computer implementation method for generating responses to queries from users or agents within a system by an agent, wherein the system The method comprises a vector-embedded memory (VEM) configured to store vectors and chunks of generated natural language text, a graph-based memory (GBM) configured to store information in the form of nodes interconnected by links, and a processor (21), each of which includes a specific chunk of natural language text representing a concept. Based on the semantic similarity between the vectors stored in the vector embedding memory (VEM) and the vectors generated based on the chunks of natural language text of the query generated and obtained by the natural language processing module (NLP), a matching vector is searched from the vector embedding memory (VEM). Based on the correspondence between the matching vectors and the nodes, the nodes stored in the graph-based memory (GBM) are determined and activated, and based on the links between the activated nodes, the connected nodes are further activated. A computer implementation method comprising generating a response to the query based on chunks of text determined based on the matching vectors from the vector-embedded memory (VEM), and further based on the determined chunks of text corresponding to the activated nodes, including the connected nodes, from the graph-based memory (GBM).

2. To enable traceability of the chunks of text used to generate the response, the generated response is augmented by referencing the chunks of text used as the basis for generating the response. A computer implementation method for generating a response to a query from a user or agent as described in claim 1, further comprising:

3. The natural language processing module (NLP) acquires and processes natural language commands from the user and generates text chunks from the natural language commands, The Text Embedding Module (TE) generates vectors based on chunks of the text, The vector embedding memory (VEM) stores the vectors generated by the text embedding module (TE) in association with the generated chunks of text. The processor (21) searches the vector embedding memory (VEM) to determine matching vectors based on the similarity between the vectors of the queries obtained from the user or agent and the vectors of the determined matching vectors, and the vector search module (VR) searches the stored vectors from the vector embedding memory (VEM) based on the semantic similarity between the vectors stored in the vector embedding memory (VEM) and the vectors generated by the text embedding module (TEM) based on the chunks of natural language text of the queries obtained and processed by the natural language processing module (NLP), in order to generate a first ranked list of the determined matching vectors, The method further includes determining the nodes stored in the graph-based memory (GBM) based on the correspondence between the first ranking list of matching vectors and the determined vectors, and generating a second ranking list of the corresponding nodes. Here, the graph-based memory (GBM) stores information in the form of nodes interconnected by links, each node containing a specific chunk of text representing a certain concept, each link being located between a source node and a target node, each link representing the relationship between the concept of the source node and the concept of the target node, and the method further, The processor (21) activates the corresponding node stored in the graph-based memory (GBM) and activates connected nodes based on the links between the activated node and other nodes in the graph-based memory (GBM) by applying a cross-graph algorithm, particularly a random walk or a individually configured PageRank. The processor (21) generates a third ranking list containing the activated nodes stored in the graph-based memory (GBM) which includes the activated corresponding nodes and the activated connected nodes, selects the activated nodes from the third ranking list for use in generating a response to the query, and determines the chunk of text corresponding to the activated nodes in the third ranking list. The natural language processing module (NLP) receives the determined text chunks corresponding to the activated nodes in the third ranking list, and generates a response to the query based on the semantic proximity between the stored vectors in the vector embedding memory (VEM) and the vectors generated by the text embedding module (TEM) based on the natural language text chunks of the query, and the determined text chunks corresponding to the activated nodes in the third ranking list for generating the response. A computer implementation method for generating a response to a query from a user or agent as described in claim 1 or 2.

4. The vector search module (VR) searches for the vector from the vector embedding memory (VEM) based on the semantic proximity of the stored vectors in the VEM, in combination with keyword-based searching using queries. A computer implementation method for generating a response to a query from a user or agent, as described in claim 3, further comprising:

5. A computer implementation method for generating responses to queries from a user or agent according to Claim 1, comprising pre-loading data into the vector-embedded memory (VEM) and the graph-based memory (GBM) during the training phase of the system before the operation phase of the system.

6. A computer implementation method for generating a response to a query from a user or agent according to claim 1, comprising dynamically populating the vector-embedded memory (VEM) and the graph-based memory (GBM) with data during the operation phase of the system.

7. A computer implementation method for generating a response to a query from a user or agent according to claim 1, wherein each node includes a chunk of natural language text representing an abstracted concept.

8. A computer implementation method for generating a response to a query from a user or agent according to claim 1, wherein each concept is represented by a node stored in graph-based memory (GBM), a corresponding chunk of text, and a corresponding vector stored in vector-embedded memory (VEM).

9. A computer implementation method for generating a response to a query from a user or agent according to claim 1, comprising, during the operation phase of the system, the processor (21) continuously maintaining a correspondence between vectors stored in the vector-embedded memory (VEM) and corresponding nodes stored in the graph-based memory (GBM).

10. The nodes stored in the graph-based memory (GBM) correspond to a concept that is a superset of concepts corresponding to the vectors stored in the vector-embedded memory (VEM), The graph-based memory (GBM) is configured to store additional information relating to a set of concepts corresponding to vectors stored in the vector-embedded memory (VEM) in the form of a relationship between the concepts in the set of concepts and additional concepts for which there are no corresponding vectors in the vector-embedded memory (VEM). A computer implementation method for generating a response to a query from a user or agent as described in claim 1.

11. A computer implementation method for generating a response to a query from a user or agent according to claim 1, wherein the graph-based memory (GBM) is configured to further store additional information relating to additional concepts not included in the concepts corresponding to the vectors stored in the vector-embedded memory (VEM).

12. A computer implementation method for generating a response to a query from a user or agent according to claim 3, comprising: analyzing a question, query, or instruction obtained from the user or agent using the natural language processing module (NLP); determining, based on the analysis thereof, whether the user intends to access the memory contents of a knowledge base comprising the vector embedding memory (VEM) and the graph-based memory (GBM) and store new memory contents in the knowledge base; wherein the text embedding module (TE) is configured to convert the generated chunks of text describing the new memory contents into vectors describing the new memory contents, store them in the vector embedding memory (VEM), and associate them with nodes in the graph-based memory (GBM).

13. The aforementioned natural language processing module (NLP) analyzes the questions, queries, or instructions obtained from the user or agent, The processor (21) activates the corresponding node stored in the graph-based memory (GBM) based on the analysis stored in the graph-based memory (GBM), and activates connected nodes based on the links between the activated node and other nodes in the graph-based memory (GBM), in particular restricting the activation of the connected nodes to a selected subset of link types and node types based on the analysis. A computer implementation method for generating a response to a query from a user or agent as described in claim 3, including the method described in claim 3.

14. A computer implementation method for generating a response to a query from a user or agent according to claim 1, comprising the processor (21) generating a first ranking list of matching vectors for responding to the query obtained from the user or agent from the vector embedding memory (VEM), wherein the first ranking list includes, for each matching vector, a measure of matching, a measure of appropriateness, or an evaluation of similarity.

15. The vector embedding memory (VEM) is configured to store the contents of at least one further modality, The method includes generating the response to the query based on the matching vector retrieved from the vector embedding memory (VEM) and further based on the contents of the at least one additional modality stored. A computer implementation method for generating a response to a query from a user or agent as described in claim 1.

16. The perceptual module includes dynamically constructing a knowledge database to be stored in the vector-embedded memory (VEM) and the graph-based memory (GBM), wherein the knowledge database includes a plurality of text chunks stored in the vector-embedded memory (VEM) and a plurality of nodes and links stored in the graph-based memory (GBM). A computer implementation method for generating a response to a query from a user or agent as described in claim 1.

17. The processor (21) performs a synchronization process, which includes at least one of comparing a representation stored in the vector-embedded memory (VEM) with a corresponding representation stored in the graph-based memory (GBM), and deriving the representation stored in the vector-embedded memory (VEM) and the corresponding representation stored in the graph-based memory (GBM) from a predetermined data source. A computer implementation method for generating a response to a query from a user or agent as described in claim 1.

18. The processor (21) determines the type of the query based on an analysis of the chunks of natural language text of the query, and The processor (21) further activates the nodes stored in the graph-based memory (GBM) based on the type of query determined, In particular, the processor (21) activates selected link types or node categories stored in the graph-based memory (GBM) based on the type of query determined. A computer implementation method for generating a response to a query from a user or agent as described in claim 1, including the method described in claim 1.

19. The processor (21) dynamically adjusts the number of chunks of text to be searched when activating the corresponding node stored in the memory (GBM) based on the graph. In particular, to avoid reaching the token limit of the large language model configured to generate the response, the number of chunks of text retrieved from the representations stored in the graph-based memory (GBM) is dynamically adjusted, or The processor (21) determines whether the information retrieved from the graph-based memory (GBM) and the vector-embedded memory (VEM) is sufficient to respond to the query, and If the processor (21) determines that the information retrieved from the graph-based memory (GBM) and the vector-embedded memory (VEM) is insufficient to respond to the query, it dynamically adjusts the number of chunks of text retrieved from the graph-based memory (GBM). A computer implementation method for generating a response to a query from a user or agent as described in claim 1, including the method described in claim 1.

20. The processor (21) determines the size of the chunk of text retrieved from the representations stored in the graph-based memory (GBM) and the vector-embedded memory (VEM), and if the determined size exceeds a predetermined threshold, it applies a process to summarize the chunk of text retrieved based on a large-scale language model, particularly based on the MapReduce algorithm. A computer implementation method for generating a response to a query from a user or agent as described in claim 1, including the method described in claim 1.

21. The processor (21) is configured to determine a scale representing the mapping of chunks of text to representations stored in the graph-based memory (GBM), and to generate chunks of text and adapt the process of determining the scale of the mapping until the determined scale of the mapping satisfies a predetermined termination criterion. A computer implementation method for generating a response to a query from a user or agent as described in claim 1.

22. A system for generating responses to queries from users or agents, A vector-embedded memory (VEM) configured to store vectors and chunks of generated natural language text, A graph-based memory (GBM) configured to store information in the form of nodes interconnected by links, wherein each of the nodes contains a memory (GBM) containing a specific chunk of natural language text representing a concept, A processor (21), Based on the semantic similarity between the vectors stored in the vector embedding memory (VEM) and the vectors generated based on the chunks of natural language text of the query generated and obtained by the natural language processing module (NLP), a matching vector is searched from the vector embedding memory (VEM). Based on the correspondence between the matching vectors and the nodes, the nodes stored in the graph-based memory (GBM) are determined and activated, and based on the links between the activated nodes, the connected nodes are further activated. To generate a response to the query based on chunks of text determined based on the matching vectors from the vector-embedded memory (VEM), and further based on the determined chunks of text corresponding to the activated nodes, including the connected nodes, from the graph-based memory (GBM). A system comprising a processor (21) configured to perform the following.

23. A system comprising an agent and a system according to claim 22 for generating a response to a query from the agent, wherein the agent is an autonomous agent.

24. The system according to claim 23, wherein the autonomous agent is configured to generate and output queries based on a large-scale language model and to plan the autonomous agent's behavior based on the response received from the system, the response includes information on the preferences and causal chain of possible actions of relevant stakeholders in order to generate the autonomous agent's response to the queries.

Images