Modifying vectors representations for a retrieval augmented decoder

Abstract

In some examples, there is provided a method for modifying vector representations for a retrieval augmented decoder configured to generate an output on the basis of a natural language input query, wherein the vector representations comprise multiple interdependent text embeddings configured to provide contextual information for the query from a pair of jointly observable heterogeneous information sources for the decoder.

Description

[0001] MODIFYING VECTORS REPRESENTATIONS FOR A RETRIEVAL AUGMENTED DECODER

[0002] TECHNICAL FIELD

[0003] The present disclosure relates, in general, to a retrieval augmented decoder-large language model (LLM) system. Aspects of the disclosure relate to modifying vector representations for a retrieval augmented decoder.

[0004] BACKGROUND

[0005] Decoder-based language models (LLMs) have become integral to advancements in natural language processing (NLP) and artificial intelligence (Al). These models, which primarily focus on generating human-like text, have seen widespread adoption across various applications, including chatbots, content creation, translation, and summarisation. The underlying architecture of decoder-LLMs, often built on transformers, relies on self-attention mechanisms to predict the next word in a sequence, making them highly effective in tasks that require coherent and contextually accurate text generation. However, despite their success, there are notable challenges related to the performance of these models, particularly when scaling up to handle more complex tasks or larger datasets.

[0006] One of the key issues with decoder-LLMs is their computational inefficiency. The process of generating text involves significant computational overhead due to the sequential nature of prediction. Each token generated requires the model to consider all previous tokens, which increases the time complexity quadratically as the sequence length grows. This issue is exacerbated in scenarios requiring the generation of long texts or when real-time responses are needed, such as in interactive systems. Consequently, there is a trade-off between the quality of the generated text and the speed of generation, limiting the practical applicability of decoder-LLMs in many real- world scenarios.

[0007] Another challenge pertains to the model’s capacity to maintain contextual relevance over long sequences. Decoder-LLMs are prone to losing track of long-range dependencies, leading to text that may become repetitive, incoherent, or irrelevant as the sequence progresses. This problem is particularly evident in tasks such as story generation or summarisation, where maintaining a coherent narrative or accurately reflecting the source material is essential. The limitations in managing context over extended sequences often necessitate the use of additional mechanisms, such as memory augmentation or attention windowing, which can introduce further complexity and computational burden.

[0008] Furthermore, as decoder-LLMs are scaled to accommodate more parameters in an attempt to improve their language understanding and generation capabilities, they require exponentially more computational resources for training and inference. This scaling issue presents significant barriers in terms of both cost and environmental impact, as the energy demands of training large-scale models are substantial. The current trajectory of increasing model size to enhance performance is unsustainable and necessitates the development of more efficient algorithms or architectures that can achieve similar results with fewer resources.

[0009] Given these challenges, there is a clear need for advancements in the performance of decoder-LLMs.

[0010] SUMMARY

[0011] An objective of the present disclosure is to provide a retrieval augmented decoder-LLM system having an improved performance. The foregoing and other objectives are achieved by the features of the independent claims.

[0012] Further implementation forms are apparent from the dependent claims, the description and the Figures.

[0013] A first aspect of the present disclosure provides a method for modifying vector representations for a retrieval augmented decoder configured to generate an output on the basis of a natural language input query, wherein the vector representations comprise multiple interdependent text embeddings configured to provide contextual information for the query from a pair of jointly observable heterogeneous information sources for the decoder, wherein the vector representations comprise a first set of embeddings logically defining a first set of clusters for a first information source of the pair of heterogeneous information sources for the decoder, the first set of clusters defining a cluster dictionary of the first set of clusters and comprising indexed subsets of the first set of embeddings, wherein each vector representation of the first set of embeddings comprises an embedding for respective portions of text, each portion of text defining a chunk for input text data stored in a low tier storage apparatus, a second set of embeddings logically defining a second set of clusters for a second information source of the pair of heterogeneous information sources for the decoder, the second set of clusters defining a cluster dictionary of the second set of clusters and comprising indexed subsets of the second set of embeddings, wherein each vector representation of the second set of embeddings comprises an embedding for respective entities and for respective relationships between the entities, wherein the entities and relationships define a knowledge graph, the method comprising receiving a natural language input query at a proxy of the retrieval augmented decoder, wherein the proxy of the retrieval augmented decoder comprises a large language model configured to generate respective attention scores for tokens of the chunks and for tokens of groups of words forming natural language sentences derived using the knowledge graph, calculating a set of similarity values to quantify: a relationship between the natural language input query and a chunk, and a relationship between the natural language input query and a sentence derived using the knowledge graph, calculating, on the basis of the attention scores using an encoder comprising a large language model, a set of loss functions, and, using the set of loss functions, modifying the vector representations, whereby to minimize a distance between the attention scores and the similarity values.

[0014] Accordingly, a retrieval augmented decoder-LLM system with enhanced performance is presented. The approach integrates information from both the knowledge graph (KG) and text chunks, optimizing the context for the decoder-LLM through interdependent text embeddings. This leads to better context quality, notably reducing interference latency by utilizing shorter graph sentences. Additionally, by adjusting the proportion of information from the KG and chunks in the decoder-LLM context, the relevance of an information source is identified using language model signals rather than heuristics. The need for a reranking algorithm is eliminated, reducing resource usage, computational complexity, and latency, thanks to the optimized text embeddings of KG entities and relationships. Moreover, storing only active data on high-tier storage allows for a low-power, cost-effective storage solution with enhanced data security. Since the movement of active data to and from high- tier storage is directed by user and LLM policies, the need for frequent low-tier storage access (and thus high latency) is minimized.

[0015] The first set of embeddings and the second set of embeddings may be stored in the low tier storage apparatus or a high tier storage apparatus.

[0016] The method may further comprise processing each portion of text defining respective chunks for the input text data to construct the knowledge graph, and augmenting the entities and relationships of the knowledge graph using an external information source, wherein the external information source comprises at least metadata relating to the input text data.

[0017] Text defining respective chunks for the input text data including their embeddings may be stored in a chunk vector database, relationship data representing relationships between entities in the knowledge graph including their embeddings may be stored in a relationship vector database, entities of the knowledge graph including their embeddings and relationship instances for the knowledge graph, jointly representing a graph structure of the knowledge graph, may be stored in a graph database. The method may further comprise inferencing respective ones of the chunks for input text data, information representing the entities, and information representing the relationships through a pre-trained language encoder model to generate chunk embeddings, the entity embeddings, and the relationship embeddings, respectively.

[0018] The method may further comprise clustering the first set of embeddings, whereby to form the first set of clusters, generating the cluster dictionary of the first set of clusters, wherein a cluster index is mapped to a list of chunk indices belonging to the cluster, mapping each cluster index to a centroid of the cluster index, and storing the cluster dictionary of the first set of clusters in a high tier storage apparatus.

[0019] The method may further comprise clustering the entity embeddings belonging to the second set of embeddings, whereby to form the second set of clusters, forming a subgraph for the knowledge graph for each cluster of the second set of embeddings by maintaining relationship instances connecting entities within the same cluster, generating the cluster dictionary of the second set of clusters, wherein a subgraph index is mapped to a list of entity indices belonging to the subgraph, mapping each subgraph index to a centroid belonging to the subgraph, and storing cluster dictionary of the second set of clusters in the high tier storage apparatus.

[0020] Entity information may comprise information relating to semantic entities identified in the input text data, and relationship information may comprise information relating to relationships between the semantic entities identified in the input text data.

[0021] The method may further comprise pre-processing each data chunk, wherein the pre-processing comprises at least one of text cleaning, filtering, and deduplication.

[0022] The method may further comprise generating each portion of text defining a chunk for input text data using a chunking algorithm comprising a fixed and non-overlapping sliding window.

[0023] The method may further comprise monitoring accesses to each of the clusters of the first set of clusters and the subgraphs of the second set of clusters using egress variables, wherein the egress variables indicate a number of time units since a last access.

[0024] The method may further comprise periodically checking the egress variables, and, in response to determining that an egress variable of a cluster and / or a subgraph exceeds a system parameter, offloading the cluster and / or the subgraph to the low tier storage apparatus.

[0025] The method may further comprise promoting the cluster and / or the subgraph to the high tier storage apparatus in the event that the cluster and / or the subgraph is requested and / or to be used as the result of an input query.

[0026] Calculating a set of similarity values may comprise calculating a similarity score between an embedding relating to the natural language input query and vector representations of chunks, wherein the embedding relating to the natural language input query is generated using a pretrained model, and calculating a similarity score between the embedding relating to the natural language input query and vector representations of the sentences derived using the knowledge graph by calculating a similarity score between the embeddings relating to the natural language input query and embeddings relating to entities and embeddings relating to relationships.

[0027] The method may further comprise using the retrieval augmented decoder, generating an output on the basis of the natural language input query.

[0028] A size of the contextual information for the query may be reduced over time as embeddings are optimised. A second aspect of the present disclosure provides an apparatus comprising a processor, and a memory coupled to the processor, the memory configured to store program code executable by the processor, the program code comprising one or more instructions for modifying vector representations for a retrieval augmented decoder configured to generate an output on the basis of a natural language input query, wherein the vector representations comprise multiple interdependent text embeddings configured to provide contextual information for the query from a pair of jointly observable heterogeneous information sources for the decoder, wherein the vector representations comprise a first set of embeddings logically defining a first set of clusters for a first information source of the pair of heterogeneous information sources for the decoder, the first set of clusters defining a cluster dictionary of the first set of clusters and comprising indexed subsets of the first set of embeddings, wherein each vector representation of the first set of embeddings comprises an embedding for respective portions of text, each portion of text defining a chunk for input text data stored in a low tier storage apparatus, a second set of embeddings logically defining a second set of clusters for a second information source of the pair of heterogeneous information sources for the decoder, the second set of clusters defining a cluster dictionary of the second set of clusters and comprising indexed subsets of the second set of embeddings, wherein each vector representation of the second set of embeddings comprises an embedding for respective entities and for respective relationships between the entities, wherein the entities and relationships define a knowledge graph, wherein the one or more instructions are configured to cause the apparatus to receive a natural language input query at a proxy of the retrieval augmented decoder, wherein the proxy of the retrieval augmented decoder comprises a large language model configured to generate respective attention scores for tokens of the chunks and for tokens of groups of words forming natural language sentences derived using the knowledge graph, calculate a set of similarity values to quantify: a relationship between the natural language input query and a chunk, and a relationship between the natural language input query and a sentence derived using the knowledge graph, calculate, on the basis of the attention scores using an encoder comprising a large language model, a set of loss functions, and, using the set of loss functions, modify the vector representations, whereby to minimize a distance between the attention scores and the similarity values.

[0029] The first set of embeddings and the second set of embeddings may be stored in the low tier storage apparatus or a high tier storage apparatus.

[0030] The one or more instructions may be configured to cause the apparatus to process each portion of text defining respective chunks for the input text data to construct the knowledge graph, and augment the entities and relationships of the knowledge graph using an external information source, wherein the external information source comprises at least metadata relating to the input text data.

[0031] The one or more instructions may be configured to cause the apparatus to store text defining respective chunks for the input text data including their embeddings in a chunk vector database, store relationship data representing relationships between entities in the knowledge graph including their embeddings in a relationship vector database, store entities of the knowledge graph and relationship instances for the knowledge graph including their embeddings, jointly representing a graph structure of the knowledge graph, in a graph database.

[0032] The one or more instructions may be configured to cause the apparatus to inference respective ones of the chunks for input text data, information representing the entities, and information representing the relationships through a pre-trained language encoder model to generate chunk embeddings, the entity embeddings, and the relationship embeddings, respectively.

[0033] The one or more instructions may be configured to cause the apparatus to cluster the first set of embeddings, whereby to form the first set of clusters, generate the cluster dictionary of the first set of clusters, wherein a cluster index is mapped to a list of chunk indices belonging to the cluster, map each cluster index to a centroid of the cluster index, and store the cluster dictionary of the first set of clusters in a high tier storage apparatus. The one or more instructions may be configured to cause the apparatus to cluster the entity embeddings belonging to the second set of embeddings, whereby to form the second set of clusters, form a subgraph for the knowledge graph for each cluster of the second set of embeddings by maintaining relationship instances connecting entities within the same cluster, generate the cluster dictionary of the second set of clusters, wherein a subgraph index is mapped to a list of entity indices belonging to the subgraph, map each subgraph index to a centroid belonging to the subgraph, and store cluster dictionary of the second set of clusters in the high tier storage apparatus.

[0034] The one or more instructions may be configured to cause the apparatus to generate each portion of text defining a chunk for input text data using a chunking algorithm comprising a fixed and non-overlapping sliding window.

[0035] The one or more instructions may be configured to cause the apparatus to monitor access to each of the clusters of the first set of clusters and the subgraphs of the second set of clusters using egress variables, wherein the egress variables indicate a number of time units since a last access.

[0036] The one or more instructions may be configured to cause the apparatus to periodically check the egress variables, and, in response to determining that an egress variable of a cluster and / or a subgraph exceeds a system parameter, offloading the cluster and / or the subgraph to the low tier storage apparatus.

[0037] The one or more instructions may be configured to cause the apparatus to promote the cluster and / or the subgraph to the high tier storage apparatus in the event that the cluster and / or the subgraph is requested and / or to be used as the result of an input query.

[0038] The one or more instructions may be configured to cause the apparatus to calculate a similarity score between an embedding relating to the natural language input query and vector representations of chunks, wherein the embedding relating to the natural language input query is generated using a pretrained model, and calculate a similarity score between the embedding relating to the natural language input query and vector representations of the sentences derived using the knowledge graph by calculating a similarity score between the embeddings relating to the natural language input query and embeddings relating to entities and embeddings relating to relationships.

[0039] The one or more instructions may be configured to cause the apparatus to using the retrieval augmented decoder, generate an output on the basis of the natural language input query.

[0040] These and other aspects of the disclosure will be apparent from the embodiments) described below.

[0041] BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order that the present disclosure may be more readily understood, embodiments of the disclosure will now be described, by way of example, with reference to the accompanying drawings, in which:

[0043] Fig. 1 is a flow chart of a method for modifying vector representations for a retrieval augmented decoder according to an example;

[0044] Fig. 2 is a flow chart of a method according to an example;

[0045] Fig. 3 is a flow chart of a process according to an example; Fig. 4 is a schematic representation of an encoder-decoder architecture with a fusion- in-decoder method for a teacher- LLM;

[0046] Fig. 5 is a schematic representation of a retrieval-augmented Decoder-LLM system according to an example;

[0047] Fig. 6 is a schematic representation of an apparatus according to an example; and

[0048] Fig. 7 is a schematic representation of an apparatus according to an example.

[0049] DETAILED DESCRIPTION

[0050] Example embodiments are described below in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein.

[0051] Accordingly, while embodiments can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate.

[0052] The terminology used herein to describe embodiments is not intended to limit the scope. The articles “a,” “an,” and “the” are singular in that they have a single referent, however the use of the singular form in the present document should not preclude the presence of more than one referent. In other words, elements referred to in the singular can number one or more, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, items, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or groups thereof.

[0053] Unless otherwise defined, all terms (including technical and scientific terms) used herein are to be interpreted as is customary in the art. It will be further understood that terms in common usage should also be interpreted as is customary in the relevant art and not in an idealized or overly formal sense unless expressly so defined herein.

[0054] Multi-tier storage systems address establish a hierarchy of storage devices, such as SSDs, HDDs, DRAM, and magnetic tapes, based on their characteristics, with the aim of optimising overall system performance and reducing costs. For instance, higher- tier SSDs are increasingly becoming the standard for active large data storage due to their efficiency in fast random data access. However, within the constraints of energy and cost, SSDs may offer limited storage capacity. Conversely, magnetic tape, a low-tier storage medium, offers significant advantages such as low power consumption, cost efficiency, high capacity, and data security. Although data retrieval from magnetic tape can involve higher latency, the rapidly growing volume of data makes magnetic tape an attractive option due to its green, cost-effective, and reliable storage capabilities. Read and write operations on magnetic tape are conducted via a tape drive, with effective data access typically achieved through Linear Tape File Systems (LTFS) and the accompanying software stack tailored for various user applications.

[0055] Large language models (LLMs) are extensive artificial intelligence models, typically based on the transformer architecture, designed to process natural language text. The input text is first processed by a tokenizer, such as byte -pair encoding (BPE), before being fed into the LLM. The three primary types of LLMs are encoder-only, encoder-decoder, and decoder-only transformer architectures, each suited to different types of language processing tasks. Encoder and decoder models primarily employ self-attention mechanisms, where the model focuses on the token values from the previous layer of the input sequence (and in the case of decoders, also on the generated output). In the encoder-decoder architecture, for example, the Flan-T5 model, input sequences are processed within the encoder, where each layer attends to the token values of the previous layer’s input sequence. The decoder layer utilises both cross-attention (to tokens from the encoder) and self-attention (to tokens of the previous layer and the generated output). The Fusion-in-Decoder technique can be applied to scale the LLM to process very large input sequences, improving efficiency by linearly scaling the complexity of encoder processing with the input size, while the decoder considers the entire input before generating a response.

[0056] The latest base models of LLMs are decoder-only models containing hundreds of billions of parameters. These models are pretrained on massive text corpora, which have undergone extensive pre-processing, including data cleaning, filtering, text deduplication, and quality evaluation. Assistant decoder-LLMs, such as Llama-2-7b-chat-hf, are derived by fine-tuning these base models on smaller question-answer datasets and applying techniques to align the model's behaviour with human feedback. During the online phase, a natural language query is tokenized and input into the decoder-LLM, which then generates an answer. Despite storing vast amounts of knowledge from their training data, decoder-LLMs can still produce incorrect responses (often referred to as hallucinations) or may lack sufficient information to generate accurate answers due to outdated or missing knowledge. Techniques such as retrieval augmented generation (RAG) have recently been introduced to enhance the model by adding query-related text information to the input sequence.

[0057] It has been demonstrated that augmenting the decoder-LLM’ s input with additional context — by creating a prompt that combines context with the query — significantly mitigates issues related to hallucinations and outdated or missing knowledge. For the LLM to produce coherent responses, it is essential that the retrieval algorithm selects the most relevant pieces of information from the vast available data to provide the necessary context for the query. In practice, this involves pre-processing large amounts of raw text, dividing it into small chunks, and storing these chunks on media with relatively fast access times. The retrieval algorithm then determines which chunks are most relevant to the query, ensuring that the retrieved passages enhance the model's ability to generate accurate and contextually appropriate responses.

[0058] The retrieval augmented generation approach has gained popularity due to its ability to provide a decoder-based language model with the necessary query -related information instantly, by simply reading chunks from a database without the need for any additional training. The most common RAG pipeline involves several key steps. During the offline phase:

[0059] (i) raw text from a large corpus is pre-processed and segmented into chunks;

[0060] (ii) text embeddings for each chunk are generated by inferencing a pre-trained encoder-based language model (Encoder-LLM);

[0061] (iii) these embeddings are clustered using the K-means algorithm, with the centroid of each cluster being stored; and

[0062] (iv) the chunks, along with their corresponding embedding vectors, are indexed and stored in a vector database on a medium with low access latency.

[0063] In the online phase:

[0064] (i) the text embedding of a user query is generated via inference of a pre-trained Encoder-LLM;

[0065] (ii) clusters whose centroid embeddings are most similar to the query embedding are identified;

[0066] (iii) the similarity between the query embedding and all vectors within the selected clusters is calculated, and the TOP-N embeddings are extracted;

[0067] (iv) a sophisticated re-ranking algorithm determines which K of the N chosen vectors are most relevant to the query and the Decoder-LLM; and (v) the vector indices are used to retrieve the K chunks, which are then fed into the Decoder-LLM.

[0068] Despite the advantages of the RAG approach, several performance issues have been identified. One of the primary concerns is the efficiency of the similarity search process, which has become a major focus of recent optimisations. Although the re-ranking step is designed to improve the relevance of the retrieved text passages, it introduces additional computational complexity and latency, particularly when dealing with large datasets. Furthermore, while some recent research has focused on optimising the embedding vector generation to ensure that the TOP-K target text passages obtained based on attention scores are the ones with the highest semantic similarity after the similarity search, these improvements still fall short of addressing the fundamental inefficiencies of the process. The reliance on clustering and re-ranking adds layers of processing that can slow down the overall response time, particularly in real-time applications.

[0069] Another approach involves using a knowledge graph (KG) to instantly provide the necessary query-related information to a Decoder-LLM, again without requiring any training. In this method, selected parts of the KG are read from a database and sent to the Decoder-LLM in a textual form, by concatenating the chosen entities and relationship instances. The main steps of KG retrieval in the offline phase include:

[0070] (i) constructing a knowledge graph from the given textual data and other external sources;

[0071] (ii) generating text embeddings for each entity and relationship via inference of a pre-trained Encoder-LLM; and

[0072] (iii) storing the KG in a graph database, with the entities and relationships indexed alongside their corresponding embedding vectors in a vector database on a medium with low access latency.

[0073] During the online phase:

[0074] (i) the entities and relationships present in the user query are identified;

[0075] (ii) text embeddings for these entities and relationships are generated by a pre-trained Encoder-LLM;

[0076] (iii) the similarity between the query’s entity / relationship embeddings and the vectors in the vector database is calculated, and the TOP-M most similar entities and relationships are extracted;

[0077] (iv) the vector indices are used to retrieve the relevant entities and relationships from the vector database; and

[0078] (v) a sophisticated algorithm determines a subgraph, generated from the selected entities and relationship instances, which is then textualized and sent to the Decoder-LLM input.

[0079] While this KG-based approach offers a more structured method of retrieving query-relevant information, it also encounters significant performance challenges. The process of constructing and maintaining the knowledge graph, as well as generating and indexing embeddings for every entity and relationship, is resource-intensive. Additionally, the algorithm responsible for selecting and textualizing the relevant subgraph must balance accuracy with efficiency, often leading to trade-offs that impact the system's overall speed and responsiveness. Moreover, similar to the RAG approach, the KG-based retrieval method involves multiple stages of processing, each introducing potential delays, particularly when handling complex queries or extensive datasets.

[0080] According to an example, there is provided a retrieval augmented decoder-LLM system having an improved performance. By combining information from the KG and the chunks to build decoder-LLM context through interdependent text embeddings optimisation, the context quality can be improved. In particular, interference latency is reduced due to shorter graph sentences. Furthermore, by determining the ratio of information from the KG and the chunks for the decoder-LLM context, the relevance of an information source can be detected based on LM signals, and not heuristics. The usage of resources, computational complexity and latency can be reduced by omitting a reranking algorithm. This is possible due to the fact that the text embeddings of KG entities and relationships are optimised. Additionally, by storing only active data on a high-tier storage, a low-power, low-cost storage with high data security can be provided. As the policies for the active data movement to and from the high tier storage are user and LLM led, the need for low tier storage access (and therefore high latency) can be avoided.

[0081] Examples in the present disclosure can be provided as methods, systems or machine-readable instructions, such as any combination of software, hardware, firmware or the like. Such machine-readable instructions may be included on a computer readable storage medium (including but not limited to disc storage, CD-ROM, optical storage, etc.) having computer readable program codes therein or thereon.

[0082] The present disclosure is described with reference to flow charts and / or block diagrams of the method, devices and systems according to examples of the present disclosure. Although the flow diagrams described above show a specific order of execution, the order of execution may differ from that which is depicted. Blocks described in relation to one flow chart may be combined with those of another flow chart. In some examples, some blocks of the flow diagrams may not be necessary and / or additional blocks may be added. It shall be understood that each flow and / or block in the flow charts and / or block diagrams, as well as combinations of the flows and / or diagrams in the flow charts and / or block diagrams can be realized by machine readable instructions.

[0083] The machine-readable instructions may, for example, be executed by a machine such as a general-purpose computer, user equipment such as a smart device, e.g., a smart phone, a special purpose computer, an embedded processor or processors of other programmable data processing devices to realize the functions described in the description and diagrams. In particular, a processor or processing apparatus may execute the machine-readable instructions. Thus, modules of apparatus (for example, a module implementing a comparator unit, or a firewall structure and so on) may be implemented by a processor executing machine readable instructions stored in a memory, or a processor operating in accordance with instructions embedded in logic circuitry. The term 'processor' is to be interpreted broadly to include a CPU, processing unit, ASIC, logic unit, or programmable gate set etc. The methods and modules may all be performed by a single processor or divided amongst several processors.

[0084] Such machine-readable instructions may also be stored in a computer readable storage that can guide the computer or other programmable data processing devices to operate in a specific mode. For example, the instructions may be provided on a non- transitory computer readable storage medium encoded with instructions, executable by a processor.

[0085] Figure 1 is a flow chart of a method for modifying vector representations for a retrieval augmented decoder according to an example. The retrieval augmented decoder is configured to generate an output on the basis of a natural language input query.

[0086] The vector representations comprise multiple interdependent text embeddings configured to provide contextual information for the query from a pair of jointly observable heterogenous information sources for the decoder. The vector representations comprise a first set of embeddings logically defining a first set of clusters for a first information source of the pair of heterogeneous information sources for the decoder. The first set of clusters defines a cluster dictionary of the first set of clusters and comprises indexed subsets of the first set of embeddings. Each vector representation of the first set of embeddings comprises an embedding for respective portions of text. Each portion of text defines a chunk for input text data stored in a low tier storage apparatus.

[0087] The vector representations also comprise a second set of embeddings logically defining a second set of clusters for a second information source of the pair of heterogeneous information sources for the decoder. The second set of clusters defines a cluster dictionary of the second set of clusters and comprises indexed subsets of the second set of embeddings. Each vector representation of the second set of embeddings comprises an embedding for respective entities and for respective relationships between the entities, wherein the entities and relationships define a knowledge graph. The method of Figure 1 comprises, in block 101, receiving a natural language input query at a proxy of the retrieval augmented decoder, wherein the proxy of the retrieval augmented decoder comprises a large language model configured to generate respective attention scores for tokens of the chunks and for tokens of groups of words forming natural language sentences derived using the knowledge graph.

[0088] The chunks may be obtained, for example, during system initialisation. Figure 2 is a flow chart of a method according to an example. The method 200 may be divided into three processes - a first process 201, a second process 210, and a third process 220. As part of the first process 201 , system initialisation 202 may take place.

[0089] To aid understanding of the first process 201, reference will now be made to Figure 3. Figure 3 is a flow chart of a process according to an example. During a first phase of system initialization 202, each raw textual file of the existing data may be stored on tape 301. A copy of each file may then be divided into chunks using a chunking algorithm with a fixed, nonoverlapping sliding window. Each chunk may be further pre-processed by employing state-of-the-art methods of text cleaning, filtering, and text deduplication.

[0090] To construct the knowledge graph (KG), network-based language models may be applied to each chunk to identify real- world entities and relationship instances within the text. If certain entities and relationships are not found in the current KG, an internet search may be conducted to fetch additional text that provides more details about these entities and relationships. The downloaded text may then be chunked and pre-processed, after which the external real- world entities and relationship instances may be extracted. All (i) chunks, (ii) relationships, and (iii) entities may be indexed. Chunks and relationships may be stored in the chunk vector database 303 and relationship vector database 302, respectively, while entities and relationship instances may be stored in a graph database 302 on SSD.

[0091] In a second phase of system initialization 202, the chunks, entities, and relationships may be inferenced through pretrained contriever encoder-LLMs 304-1, 304-2, 304-3. Although three encoder-LLMs 304-n are shown in Figure 3, the disclosure is not limited thereto. The resulting (i) chunk, (ii) relationship, and (iii) entity embeddings may then be appended to their corresponding textual entries and indices on SSD. The chunk embeddings may subsequently be clustered using the K-means algorithm, and a dictionary may be created where each cluster index is mapped to the list of chunk indices that belong to it. Additionally, each cluster index may be mapped to its centroid, with the resulting dictionary stored on SSD.

[0092] Similarly, the entity embeddings may be clustered using the K-means algorithm. Each entity cluster may form a subgraph for the existing KG, maintaining the relationship instances that connect entities within the same cluster. As with the chunks, a dictionary may be created where each subgraph index is mapped to the list of entity indices that belong to it, with each subgraph index also potentially mapped to the centroid obtained by the entity clustering algorithm. The resulting dictionary may then be stored on SSD.

[0093] The chunk embeddings may be clustered using a suitable algorithm into individual chunk embeddings clusters. The knowledge graph may be divided into subgraphs by clustering the entity embeddings and by considering the relationship instances between the entities of a subgraph. The terms “clusters” and “subgraphs” used throughout the specification refer to the abstract sets resulting from these two operations. Each cluster / subgraph may be assigned a cluster / subgraph index.

[0094] Cluster or subgraph indices mapping to their centroids and embedding indices may result in dictionaries, which might always be stored on a high-tier storage device. Data in the chunk vector database related to clusters accessed within the last G time units, such as months, could be stored on a high-tier storage device for fast access, while the remaining data may be stored on a low-tier storage device. For instance, X% of the data in the chunk vector database might be stored on the high-tier storage device, with the remaining 100%-X% stored on the low-tier storage device. Similarly, data in the graph database associated with subgraphs accessed within the last G time units might be stored on a high-tier storage device for faster access, while the rest could be stored on a low-tier storage device. This could mean that Y% of the data in the graph database is stored on the high-tier storage device, with the remaining 100%- Y% stored on the low-tier storage device. Data in the relationship vector database may always be stored entirely on the high-tier storage device. The system parameter G could be adjusted to optimise storage efficiency and improve Decoder-LLM+RAG+KG inference. The layout of data on the low-tier storage device may be specifically designed to organise semantic information associated with clusters and subgraphs, thereby reducing access time to data that exhibits high semantic similarity.

[0095] Referring back to Figure 2, at the end of system initialization 202, two background processes may be initiated. These processes may need to be activated to serve input queries (second process 210) or to add new data (third process 220). To ensure optimal use of the available storage resources, a background process 203 that performs tape reclaiming and SSD garbage collection may be executed periodically. Another background process 204 may monitor accesses. This may be achieved through egress variables, which could be initialized at 0 and assigned to clusters or subgraphs to indicate the number of time units, such as months, since the last access. The egress variables may be checked periodically at the end of each time unit and incremented by 1 if no access occurred. Clusters or subgraphs for which the egress variables have exceeded the value of the system parameter G may be offloaded and sequentially written to magnetic tape. Specifically, the embedding vectors, along with their indices and textual data, may be offloaded. This approach to monitoring clusters or subgraphs access may facilitate the sequential writing and reading of large amounts of data, which could be important for limiting overall latency.

[0096] Referring back to Figure 1, in block 102, the method comprises calculating a set of similarity values to quantify: a relationship between the natural language input query and a chunk, and a relationship between the natural language input query and a sentence derived using the knowledge graph.

[0097] The second process 210 may be triggered each time a new query is input into the system, thereby activating the online phase 211 responsible for generating an answer. Similar to the KG construction, network-based language models may be applied to the query to identify real- world entities and relationships. The query, along with its identified entities and relationships, may then be inferenced through pretrained Contriever Encoder-LLMs. Using L2 distance as a metric, the similarity of the query embedding to all cluster centroids, as well as the similarity of all query entities’ embeddings to all subgraph centroids, may be computed. It is worth noting that the dictionaries mapping cluster or subgraph indices to their respective centroids and embedding vector indices may always be kept on SSD for fast access.

[0098] Using L2 distance as a metric, the similarity between the query embedding and all chunk embedding vectors belonging to the closest clusters may be computed, and the TOP-Lnclosest vectors may be selected. Additionally, the closest subgraphs may be traversed to compute the similarity between query entities and relationships and those present in the closest subgraphs. The similarity of paths consisting of two entities and a relationship instance connecting them — termed graph sentences — may be evaluated by calculating the mean similarity score of the three components. It is important to note that when multiple entities and relationships are present in the query, multiple similarity scores may be assigned to the same graph sentence. Among all the closest subgraphs, the TOP-Jngraph sentences with the highest similarity scores may be selected. The choice of appropriate values for Lnand Jnat the n-th iteration of the offline phase is a significant aspect of this disclosure and will be discussed further in the offline phase.

[0099] The indices of the selected TOP-Lnchunk embeddings, as well as the indices of the entity and relationship embeddings corresponding to the extracted TOP-Jngraph sentences, may be used to retrieve the corresponding textual chunks from the vector and graph databases. These chunks and graph sentences may then be concatenated to the query, tokenized using the BPE algorithm, and sent to the input of the Llama-2-7b-chat-hf model. The same BPE algorithm may be used to decode the answer to the given query, which is then output to the user. At the end of the online phase 211, the query, its embedding vector, the identified query entities and relationships, their embedding vectors, and the indices of the detected closest clusters or subgraphs may be stored in a buffer implemented as a first-in-first-out (FIFO) queue. If the buffer 212 does not reach a certain occupancy size B, measured in the number of episodes (i.e., queries), the process may be terminated; otherwise, the offline phase 213 may be initiated. It is crucial to note that once the processing of the current query is completed in the online phase 211, the system is ready to receive the next query. If a new query arrives while the offline phase 213 is running, the online processing of the query takes precedence.

[0100] The first step of the offline phase 213 involves removing all content from the FIFO queue and temporarily storing it in memory. Since the buffer contains only a few textual elements, embeddings, and indices, this operation may not create significant memory overhead. The iteration counter, initially set to 0, is incremented by 1 , and the n-th iteration is then started.

[0101] For each episode in memory, the similarity between the query embedding and all chunk embedding vectors belonging to the clusters whose indices are stored may be calculated. The closest TOP-N chunk embedding vectors are retained along with their corresponding indices, using L2 distance as the chosen similarity metric. Furthermore, for each episode, the identified query entities and relationships may be indexed. The subgraphs whose indices are stored are traversed, and the similarities between query entities and relationships and those in the subgraphs may be computed. Similar to the online phase 211, the similarity of graph sentences may be evaluated by calculating the mean of the similarity scores of the two entities and the relationship instance connecting them. Multiple entities and relationships in the query may lead to multiple similarity scores for the same graph sentence. Therefore, both the indices of entity and relationship vectors and the indices of the query entities and relationships used for a particular similarity score may be retained. Across all observed subgraphs, the TOP-M graph sentences with the highest similarities may be selected. The indices of the extracted TOP-N chunk embeddings may be used to retrieve the corresponding textual chunks from the chunk vector database. Similarly, the indices of the entity and relationship embeddings from the extracted TOP-M graph sentences may be used to retrieve the corresponding entities and relationships from the graph and relationship vector databases. Finally, the indices of query entities and relationships may be used to retrieve the specific entities and relationships used to obtain each graph sentence similarity score.

[0102] If the content of clusters or subgraphs detected to be closest to the query and its entities is occasionally not found on SSD, a recall signal may be sent to the tape, and the data may be restored to the SSD. The chunks, entities, and relationships recalled from the tape may then be inferenced through the Encoder-LLMs, and the updated embeddings may be modified in the corresponding databases on SSD. The clustering of chunk or entity embeddings may be performed, with the current cluster or subgraph centroids serving as the algorithm's initialization. The altered cluster or subgraph centroids, along with the embedding indices associated with each cluster or subgraph index, may then be updated in the SSDs. Importantly, to avoid long tape latencies, the online phase 211 may ignore missing clusters or subgraphs and continue without waiting for the recall operation to be completed. In any case, the egress variable may be reset to 0 for all clusters or subgraphs identified as closest to the query and its entities.

[0103] Referring back to Figure 1, in block 103, the method comprises calculating, on the basis of the attention scores using an encoder comprising a large language model, a set of loss functions.

[0104] To aid understanding of the process, reference will now be made to Figure 4. Figure 4 is a schematic representation of an encoder-decoder architecture with a fusion-in-decoder method for a teacher-LLM. In Figure 4, a plus sign (‘+’) denotes the operation of concatenation. The dashed arrows inside the encoders 401-1, 401-2 and the decoder 402 indicate self-attention, whereas the regular arrows refer to cross-attention. To assess the significance of each chunk and graph sentence in answering a given query, an Encoder-Decoder Flan-T5 Teacher-LLM may be employed. An Encoder-Decoder Flan-T5 Teacher-LLM is a language model that uses both an encoder and a decoder to process and generate text, with the encoder handling input sequences and the decoder generating output based on both the input and previously generated tokens. During this process, the Fusion-in-Decoder technique may be applied, where each obtained chunk and graph sentence is concatenated with the query, tokenized using the Flan-T5 method, and then processed individually through the encoder of the Flan-T5 Teacher-LLM. The output token values from the encoder 401-1 and 401-2 for each text passage are concatenated, and the decoder 402 of the Flan-T5 Teacher-LLM processes them collectively. The cross-attention scores within the decoder 402 at the time of generating the first output token are used as an indicator of the importance of each passage for effectively answering the query.

[0105] Specifically, the attention score Aq passigned to text passage p for query q may be calculated as the average cross-attention score across all layers, attention heads, and tokens associated with p. This may be expressed as: where j iterates over J tokens obtained from the encoder output as part of the Fusion-in-Decoder procedure, / iterates over L transformer layers in the decoder and h iterates over / / cross-attention multi-attention heads.

[0106] This procedure may be applied to each query q in the memory, using chunks and graph sentences as text passages. The top- n passages based on their Aq pscores may then be selected. Additionally, the number of chunks Lnand graph sentences Jnamong the top-Kn may be recorded. The mean number of chunks Lnand graph sentences Jnmay be calculated by averaging across all queries in the memory for the current iteration n. This information may be used in the online processing phase to determine how many chunks and graph sentences to retrieve. In later iterations (n > 1), as the retrieved information adapts to user workload, the context size Kn may be reduced using the formula: where l is a positive constant and y is the smallest allowed context size.

[0107] The cross-attention scores may then be distilled into the Encoder-LLM so that the similarity score between the query and each chunk / graph sentence reflects the Aq pattention value. For a given model, the similaritys(qModet,pModet) between query q and text passage p may be defined as: where 11 • 11 denotes L2 norm. Three Encoder-LLMs may be initialized with a pretrained Contriever model, resulting in three new models: Contriever_chunk, Contriever_entity, and Contriever_relationship. The loss function for the Contriever_chunk model may be defined to minimize the mean square error (MSE) as: where q iterates over all B queries found in the buffer at the iteration start, c iterates over the top-N retrieved chunks, and contr c denotes the Contriever_chunk model.

[0108] For updating the Contriever_entity and Contriever_relationship models, the graph sentence may be denoted as gs= / ei(gs), r(gs), 62(gs)J, where ei(gs) and e2(gs) are the entities and r(gs) is a relationship class. Therefore, the attention score may need to be scaled by considering the three components. The loss functions for Contriever_entity and Contriever_relationship models may be defined as: where q iterates over all B queries found in the buffer at the iteration start, and gs iterates over the top-M retrieved graph sentences. The subscript (q) denotes the query entity and query relationship used to obtain the specific similarity score for a graph sentence. sq gsdenotes the similarity of a graph sentence, computed as the mean similarity of its three components:

[0109] The method comprises, in block 104, using the set of loss functions, modifying the vector representations, whereby to minimize a distance between the attention scores and the similarity values.

[0110] The defined loss functions may be used to fine-tune the Contriever_chunk, Contriever_entity, and Contriever_relationship models using the backpropagation algorithm. Importantly, the fine-tuning may be performed with a small learning rate to ensure that changes in system behaviour between consecutive iterations are minimal. Once the models converge, the chunks, entities, and relationships may be accessed and inferenced through their respective Contriever models, and the updated embeddings may be stored on SSD, in the vector and graph databases. The clustering of chunk / entity embeddings may be performed with the current cluster / subgraph centroids serving as initialization. The updated cluster / subgraph centroids and embedding indices associated with each cluster / subgraph index may be refreshed on SSDs. Lastly, the fine-tuned Encoder-LLMs may be updated on SSDs.

[0111] After the training is complete, the iteration checkpoint may be stored on tape for access to previous checkpoints for historization / versioning purposes. Finally, the buffer content may be cleared from memory, marking the end of the offline phase 213.

[0112] Referring back to Figure 2, the third process 220 may be triggered when new data 221 is added to the system. Similar to the data initialization phase, each raw textual file may be stored on tape, with a copy of each file potentially divided into chunks using a chunking algorithm that employs a fixed, non-overlapping sliding window. Each chunk may then undergo preprocessing involving standard text cleaning, filtering, and text deduplication methods. Network-based language models might be applied to each chunk to identify real- world entities and relationship instances within the text. An internet search may be conducted to retrieve additional details about entities and relationships that are not present in the current knowledge graph. The downloaded text may be chunked and pre-processed, after which external real- world entities and relationship instances could be extracted. These (i) chunks, (ii) relationships, and (iii) entities might be indexed accordingly. Chunks and relationships may be stored in a chunk and relationship vector database, respectively, while entities and relationship instances could be stored in a graph database on SSD.

[0113] The obtained chunks, entities, and relationships may be processed through a pretrained Contriever Encoder-LLM. Using L2 distance as a metric, the similarity between chunk embeddings and all cluster centroids, as well as between entity embeddings and all subgraph centroids, might be computed. If clusters or subgraphs that are identified as closest are not found on SSD, they may be recalled from tape. Importantly, the recalled chunks, entities, and relationships could be processed through the Encoder-LLMs, and the updated embeddings might be modified in their corresponding databases on SSD. The recall operation may need to be completed before proceeding to the next step.

[0114] The egress variable may be reset to 0 for all clusters and subgraphs identified as closest. The chunks, entities, and relationships might be processed through the Contriever_chunk, Contriever_entity, and Contriever_relationship models, respectively. All (i) chunk, (ii) relationship, and (iii) entity embeddings may be added to their corresponding textual entries and indices on SSD. The two dictionaries on SSD could be updated to map the cluster / subgraph indices to an extended list of chunk / entity indices that includes the new data. Clustering of chunk / entity embeddings might be performed with the current cluster / subgraph centroids serving as initialization, and the altered cluster / subgraph centroids and embedding indices associated with each cluster / subgraph index may be updated on SSDs.

[0115] In a second embodiment, which may closely resemble the first in most components, there could be differences in data access monitoring. To potentially reduce the number of recalls that introduce high latency due to tape access, a more sophisticated offloading policy might be applied. During system initialization, a cluster distance matrix might be generated, possibly containing the L2 distance between cluster centroids, and stored on SSD. Similarly, a subgraph distance matrix may be created, containing the L2 distance between subgraph centroids, and stored on SSD.

[0116] A background process could monitor data access, characterized by egress variables, the cluster distance matrix, and the subgraph distance matrix. As in the first embodiment, egress variables initialized at zero might be assigned to clusters and subgraphs, representing the number of time units, such as months, since the last access. These egress variables might be checked periodically at the end of each time unit and incremented by one if no access occurred. If the egress variable for any cluster or subgraph exceeds a value G, the neighbouring clusters or subgraphs within a distance D might be identified from the distance matrix. If any neighbouring clusters or subgraphs have an egress variable smaller than or equal to G, the cluster or subgraph that has not been accessed for more than G months may remain on SSD. Otherwise, it could be offloaded and written sequentially on magnetic tape, including the embedding vectors, their indices, and associated textual data.

[0117] During the online phase, for clusters or subgraphs closest to a query and its entities, as well as for all neighbouring clusters or subgraphs within distance D, a recall signal might be sent to the tape if their content is not already on SSD. Optionally, a flag could be provided to inform the user that additional information might be available to refine the answer. Similarly, when adding new data, for clusters or subgraphs closest to the extracted chunks or entities, and for all neighbouring clusters or subgraphs within distance D, a recall signal might be sent to the tape if their content isn’t on SSD. In both cases, the chunks, entities, and relationships recalled from the tape could be processed by the Encoder-LLMs, and the updated embeddings might be modified in the corresponding SSD databases. The two dictionaries on SSD could be updated so that cluster or subgraph indices are mapped to an extended list of chunk or entity indices, including the new data. Clustering of chunk or entity embeddings might be performed, with current cluster or subgraph centroids serving as algorithm initialization. The altered centroids and embedding indices associated with each cluster or subgraph index could be updated in the SSDs. Finally, during the offline phase following model convergence, the updated embeddings may be updated on SSD, and the clustering of chunk or entity embeddings might be performed. The current centroids could serve as the algorithm's starting point, and any altered centroids and embedding indices could be updated in the SSDs. In all three scenarios, whenever the centroids of clusters or subgraphs are modified, the distance between them might be recalculated, and the corresponding distance matrices could then be updated.

[0118] Figure 5 is a schematic representation of a retrieval-augmented Decoder-LLM system according to an example. As discussed above, the system 500 may comprise an offline phase 501 and an online phase 502. In the online phase 502, the answer to a user query 503 may be generated. In the offline phase 501, the text embeddings may be jointly modified for the multiple different information sources. A scoring function 503 may utilise the attention scores of a large language model (LLM) to jointly observe the TOP-N text passages from multiple information sources, rather than relying on heuristic methods.

[0119] In this approach, N>K text passages might be retrieved from each information source using a similarity metric, ensuring that passages with smaller similarity scores are also considered. This retrieval process could be facilitated by the encoder-LLM 504-1, 504-2, 504-3 (504-n), which calculates the similarity between text passages and the query. To enhance the evaluation process, a mechanism may be employed to determine the importance of each information source based on the LLM attention scores, rather than heuristic methods, with the analysis potentially refined by the decoder 505. Additionally, loss functions might be applied to distil attention scores into similarity scores for text embeddings of passages originating from various information sources. To support this framework, a multi-tier storage system 506 comprising multiple information sources (IS) could be used, containing raw data, pre-processed data, databases, and other related information essential for the data processing and retrieval tasks.

[0120] For the offline phase 501, to ensure that the top-Lnmost similar chunks and top-Jnmost similar graph sentences represent the best context for a given query and the Decoder-LLM during the online phase, the embedding vectors of chunks, entities, and relationships output by Encoder-LLMs are iteratively fine-tuned. The O-th iteration corresponds to the pretrained Encoder- LLMs, and a continual refinement process is envisioned for a dynamic online workload. Each time a sufficient number of user queries are gathered into a buffer, the iteration number increases by 1 , the Encoder-LLMs are optimised, and the queries are removed from the buffer.

[0121] Initially, the sum of Lo and Jo equals a given Ko. At the n-th iteration, for each query, the system retrieves TOP-N chunks, where N is greater than Ko and Ln, and TOP-M graph sentences, where M is greater than Ko and Jn, in the same manner as during the online phase. A pretrained Teacher-LLM, significantly smaller than the Decoder-LLM, is used as a proxy for the Decoder-LLM’ s attention scores for a given query and context. The Teacher-LLM receives as input (i) the TOP-N chunks, (ii) the TOP-M graph sentences, and (iii) the query. To handle long contexts efficiently, a Teacher-LLM with an Encoder-Decoder architecture may be employed, leveraging the Fusion-in-Decoder technique, as described above in relation to Figure 4.

[0122] The amount of attention the Teacher-LLM allocates to the tokens of a particular chunk or graph sentence determines the importance of that chunk or graph sentence for answer generation. The joint attention of the Teacher-LLM to both chunks and graph sentences allows for a comprehensive observation of the information provided by both sources, considering their interdependence. The higher the attention the Teacher-LLM gives to the tokens within a chunk or graph sentence, the higher the similarity score between the embedding and the corresponding chunk or graph sentence should be. Loss functions are defined at the output of the Encoder-LLMs with the aim of optimising the embeddings to provide optimal similarity scores. The resulting training checkpoints are stored on a low-tier storage device for archival purposes. To update the embeddings affected by fine-tuning, the Encoder-LLMs infer embeddings for chunks, entities, and relationships at each iteration. Clustering of the modified chunk or entity embeddings is performed, using the current cluster or subgraph centroids as algorithm initialization. The altered embeddings, cluster or subgraph centroids, and the embedding indices associated with each cluster or subgraph index are updated in the high-tier storage device.

[0123] Over many iterations, this refinement process yields a significantly improved context for input queries. As the refinement progresses, the length of the input context may be reduced, so that Ln+ Jn = Knis less than Ko. For a given query at the n-th iteration, the Teacher-LLM provides an attention score for each input chunk or graph sentence. At each iteration, the ratio of the number of chunks (Ln) to the number of graph sentences (Jn) within the top-Knretrieval can be determined by averaging across all queries, expressed as where Eqdenotes the expectation of the ratio across all queries, and L’ and

[0124] / ’represent the number of chunks and graph sentences for a particular query q. From these equations, the appropriate values of Ln and Jn for use in the online phase can be determined.

[0125] New queries that arrive while the offline stage is active will be executed immediately to minimise online stage latency. Fine- tuning is carried out with a small learning rate, resulting in minor embedding variations per iteration, ensuring that the retrieved context remains relevant for the queries.

[0126] Figure 6 is a schematic representation of an apparatus according to an example. The apparatus 600 comprises a processor 601, and a memory 602 coupled to the processor 601 and configured to store instructions or program code 603, executable by the processor 601. The apparatus 600 comprises the program code 603 arranged to cause the apparatus to perform the method described herein.

[0127] Figure 7 is a schematic representation of an apparatus according to an example. The system 700 may incorporate multiple tape drives as low-tier storage, each featuring a length of magnetic tape stored on two reels, managed by a microcontroller 704. The tape microcontroller 704 may control data access to and from the host microcontroller 705 via a fibre channel or SAS, and it may send control signals to the tape transport and track following servomechanisms for read and write operations. The magnetic tape may be formatted according to the LTFS open standard specification. Additionally, the system 700 may employ multiple SSDs 706 as high-tier storage, each connected via a CXL / PCIe interface 703 to other system components. Although only one SSD 706 is shown in Figure 7, the skilled person would readily understand that the disclosure is not limited to this number of SSDs, and that more SSDs may be employed as needed.

[0128] The host microcontroller 705 may serve as the system’s entry point, being the sole component connected to all major system parts, and it may provide control signals necessary to execute user workloads. The host microcontroller 705 may implement LTFS, potentially allowing easy navigation through tape-stored data using a standard hierarchical file and folder structure, similar to HDDs. It may also manage a global flash translation layer for all SSDs 706, performing garbage collection and reclaiming unreferenced storage space on both tapes and SSDs 706 at specified intervals.

[0129] A specific application may run in the background to monitor data access for each cluster or subgraph stored on SSDs 706. This application may initiate data offloading to tape after the data has not been accessed for a period exceeding a system parameter G, identifying it as "cold" data. When new data is added within the corresponding cluster or subgraph stored on tape, the application may recall the relevant cluster or subgraph and store it back on SSDs 76. During the online phase, if a user requests data stored on tape, the application may recall the necessary data and store it back on SSDs 706. However, the system 700 may not delay the generation of an appropriate user response while waiting for this operation to complete.

[0130] During system initialization, the microcontroller application may pre-process data, generating chunks, entities, relationship instances, and relationship classes. The application may also manage the control flow by: (a) sending signals to tape drives to store raw data files,

[0131] (b) initiating the addition of external information to new entities and relationships,

[0132] (c) indexing and storing the extracted chunks and relationships in vector databases, and storing entities and relationship instances in graph databases on SSDs 706,

[0133] (d) initiating the inference of Encoder-LLMs on the accelerator 701 (i.e., an Al processor), and

[0134] (e) evaluating clusters and subgraphs based on the obtained embeddings.

[0135] In the online phase, when a new query arrives, the microcontroller application may pre-process it, identifying relevant entities and relationships. It may then initiate the following control flow:

[0136] (a) trigger the Encoder-LLMs on the Al processor (e.g., accelerator 701) to infer the embeddings for the query, entities, and relationships, and identify the most similar clusters and subgraphs;

[0137] (b) using FPGAs (e.g., accelerator 701), retrieve the TOP-Lnmost similar chunk embeddings and TOP-Jngraph sentences with the highest similarity;

[0138] (c) use the obtained vector indices to access the vector and graph databases on SSDs 706, extracting the closest chunks and graph sentences;

[0139] (d) merge the query with the retrieved context, tokenize the input, and send the sequence to the Decoder-LLM on Al processors for inference, which may then generate the answer for the user;

[0140] (e) store the query’s textual content, embeddings, identified entities / relationships, and indices of the closest clusters / subgraphs as one episode in a buffer.

[0141] Once the buffer reaches a certain capacity, the offline phase may be initiated. During the offline phase, the microcontroller application may execute the following steps:

[0142] (a) move all episodes from the buffer to temporary memory storage (DRAM module), initiating the n-th iteration;

[0143] (b) trigger the FPGAs to find the most similar TOP-N chunk embeddings and TOP-M graph sentences for each query using the clusters and subgraphs stored in the buffer;

[0144] (c) retrieve the most similar chunks and graph sentences from SSDs 706 using the obtained vector indices;

[0145] (d) merge each query with its acquired context, tokenize the input, and send the sequence to the Teacher-LLM on the accelerator for inference;

[0146] (e) train the Encoder-LLMs on the accelerator 701, minimising the distance between the Teacher-LLM’s attention scores and the similarity scores;

[0147] (f) store the training checkpoints on tape drives;

[0148] (g) initiate inference of the Encoder-LLMs on the accelerator 701 for all chunks, entities, and relationships, followed by clustering with updated embeddings;

[0149] (h) calculate Lnand Jnby determining the average ratio of chunks to graph sentences among passages with the highest attention scores and integrate this ratio into the Ln+ Jn = Knformula;

[0150] (i) clear the buffer content from the memory.

[0151] The system 700 may include accelerators 701, which may perform processing for computationally intensive workloads. There may be two types of accelerators: Al processors, which handle Al workloads, and FPGAs, which are flexibly programmed for specific computations and may be equipped with additional DRAM memory 702 if needed. During system initialization, the pretrained Decoder-LLM, Encoder-LLMs, and Teacher-LLM may be stored on SSDs 706. The Encoder-LLMs may be loaded onto the Al processor and used for inference on chunks, entities, and relationships extracted from existing data. The resulting embedding vectors may be stored in corresponding vector databases, with entity embeddings directly stored in graph databases on SSDs 706. After this process, the Al processors may cluster all chunk embeddings and entity embeddings, dividing the knowledge graph into subgraphs accordingly. The dictionaries mapping cluster / subgraph indices to their embedding indices and cluster centroids may be directly stored on SSDs 706.

[0152] During the online phase, the Decoder-LLM, Encoder-LLMs, and cluster / subgraph centroids may be preloaded onto the Al processors. For each new query, the Encoder-LLMs may be used to infer embeddings for the entire query as well as the identified entities and relationships. Similarity scores between the query embedding and cluster centroids may be calculated, identifying the most relevant clusters for answer generation. Similarly, similarity scores between the query entities' embeddings and subgraph centroids may be calculated, determining the most relevant KG subgraphs. The FPGA retrievers (may directly access the vector and graph databases on SSDs 706 (and potentially DRAM) to process the embedding vectors of chunks, entities, and relationships from the selected clusters / subgraphs. These vectors may be processed in batches, retaining only the TOP-Ln most similar chunk embeddings and TOP-Jngraph sentences. The Decoder-LLM may process the input query along with the retrieved context and output the answer.

[0153] In the offline phase, at the beginning of each iteration, the Teacher-LLM and Encoder-LLMs may be loaded onto the Al processors. The FPGA retrievers may directly access the vector and graph databases on SSDs 706 (potentially also DRAM) to process the embedding vectors of chunks, entities, and relationships from the clusters / subgraphs stored in memory. The vectors may be processed in batches, retaining the TOP-N most similar chunk embeddings and TOP-M graph sentences. After this process, the Teacher-LLM may be used to infer each query with its context, calculating the cross-attention needed to generate the first token at the decoder's output. The attention scores for each text passage may be obtained by averaging the attention vectors across tokens, attention heads, and layers of the decoder. The Encoder-LLMs may be trained to minimise attentionbased loss functions, and after fine-tuning, the modified Encoder-LLMs may be updated on SSDs 706 and used to infer all chunks, entities, and relationships. The updated embedding vectors may be altered in vector and graph databases on SSDs 706, and clustering of chunk / entity embeddings may be performed using the Al processors, with current cluster / subgraph centroids serving as algorithm initialization. The updated cluster / subgraph centroids and embedding indices associated with each cluster / subgraph index may then be stored on SSDs 706.

[0154] According to an example, machine-readable instructions can be loaded onto a computer or other programmable data processing devices, so that the computer or other programmable data processing devices perform a series of operations to produce computer-implemented processing, thus the instructions executed on the computer or other programmable devices provide an operation for realizing functions specified by flow(s) in the flow charts and / or block(s) in the block diagrams.

[0155] Further, the teachings herein may be implemented in the form of a computer or software product, such as a non-transitory machine-readable storage medium, the computer software or product being stored in a storage medium and comprising a plurality of instructions, e.g., machine readable instructions, for making a computer device implement the methods recited in the examples of the present disclosure.

[0156] In some examples, some methods can be performed in a cloud-computing or network-based environment. Cloud-computing environments may provide various services and applications via the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessible through a web browser or other remote interface of the user equipment for example. Various functions described herein may be provided through a re2mote desktop environment or any other cloud-based computing environment.

[0157] While various embodiments have been described and / or illustrated herein in the context of fully functional computing systems, one or more of these exemplary embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable-storage media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the exemplary embodiments disclosed herein. In addition, one or more of the modules described herein may transform data, physical devices, and / or representations of physical devices from one form to another. The preceding description has been provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the instant disclosure. The embodiments disclosed herein should be considered in all respects illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of the instant disclosure.

Claims

CLAIMS1. A method for modifying vector representations for a retrieval augmented decoder configured to generate an output on the basis of a natural language input query, wherein the vector representations comprise multiple interdependent text embeddings configured to provide contextual information for the query from a pair of jointly observable heterogeneous information sources for the decoder, wherein the vector representations comprise: a first set of embeddings logically defining a first set of clusters for a first information source of the pair of heterogeneous information sources for the decoder, the first set of clusters defining a cluster dictionary of the first set of clusters and comprising indexed subsets of the first set of embeddings, wherein each vector representation of the first set of embeddings comprises an embedding for respective portions of text, each portion of text defining a chunk for input text data stored in a low tier storage apparatus; a second set of embeddings logically defining a second set of clusters for a second information source of the pair of heterogeneous information sources for the decoder, the second set of clusters defining a cluster dictionary of the second set of clusters and comprising indexed subsets of the second set of embeddings, wherein each vector representation of the second set of embeddings comprises an embedding for respective entities and for respective relationships between the entities, wherein the entities and relationships define a knowledge graph, the method comprising: receiving a natural language input query at a proxy of the retrieval augmented decoder, wherein the proxy of the retrieval augmented decoder comprises a large language model configured to generate respective attention scores for tokens of the chunks and for tokens of groups of words forming natural language sentences derived using the knowledge graph (101); calculating a set of similarity values to quantify: a relationship between the natural language input query and a chunk, and a relationship between the natural language input query and a sentence derived using the knowledge graph (102); calculating, on the basis of the attention scores using an encoder comprising a large language model, a set of loss functions (103); and using the set of loss functions, modifying the vector representations, whereby to minimize a distance between the attention scores and the similarity values (104).

2. The method of claim 1 , wherein the first set of embeddings and the second set of embeddings are stored in the low tier storage apparatus or a high tier storage apparatus.

3. The method of claim 1 or 2, further comprising: processing each portion of text defining respective chunks for the input text data to construct the knowledge graph; and augmenting the entities and relationships of the knowledge graph using an external information source, wherein the external information source comprises at least metadata relating to the input text data.

4. The method of any preceding claim, wherein text defining respective chunks for the input text data including their embeddings are stored in a chunk vector database, relationship data representing relationships between entities in the knowledge graph including their embeddings are stored in a relationship vector database, entities of the knowledge graph including their embeddings and relationship instances for the knowledge graph, jointly representing a graph structure of the knowledge graph, are stored in a graph database.

5. The method of any preceding claim, further comprising: inferencing respective ones of the chunks for input text data, information representing the entities, and information representing the relationships through a pre-trained language encoder model to generate chunk embeddings, the entity embeddings, and the relationship embeddings, respectively.

6. The method of any preceding claim, further comprising: clustering the first set of embeddings, whereby to form the first set of clusters; generating the cluster dictionary of the first set of clusters, wherein a cluster index is mapped to a list of chunk indices belonging to the cluster; mapping each cluster index to a centroid of the cluster index; and storing the cluster dictionary of the first set of clusters in a high tier storage apparatus.

7. The method of any preceding claim, further comprising: clustering the entity embeddings belonging to the second set of embeddings, whereby to form the second set of clusters; forming a subgraph for the knowledge graph for each cluster of the second set of embeddings by maintaining relationship instances connecting entities within the same cluster; generating the cluster dictionary of the second set of clusters, wherein a subgraph index is mapped to a list of entity indices belonging to the subgraph; mapping each subgraph index to a centroid belonging to the subgraph; and storing cluster dictionary of the second set of clusters in the high tier storage apparatus.

8. The method of any preceding claim, wherein entity information comprises information relating to semantic entities identified in the input text data, and relationship information comprises information relating to relationships between the semantic entities identified in the input text data.

9. The method of any preceding claim, further comprising: pre-processing each data chunk, wherein the pre-processing comprises at least one of text cleaning, filtering, and deduplication.

10. The method of any preceding claim, further comprising: generating each portion of text defining a chunk for input text data using a chunking algorithm comprising a fixed and non-overlapping sliding window.

11. The method of any preceding claim, further comprising: monitoring accesses to each of the clusters of the first set of clusters and the subgraphs of the second set of clusters using egress variables, wherein the egress variables indicate a number of time units since a last access.

12. The method of claim 11 , further comprising : periodically checking the egress variables, and, in response to determining that an egress variable of a cluster and / or a subgraph exceeds a system parameter, offloading the cluster and / or the subgraph to the low tier storage apparatus.

13. The method as claimed in claim 12, further comprising: promoting the cluster and / or the subgraph to the high tier storage apparatus in the event that the cluster and / or the subgraph is requested and / or to be used as the result of an input query.

14. The method of any preceding claim, wherein calculating a set of similarity values comprises: calculating a similarity score between an embedding relating to the natural language input query and vector representations of chunks, wherein the embedding relating to the natural language input query is generated using a pretrained model; and calculating a similarity score between the embedding relating to the natural language input query and vector representations of the sentences derived using the knowledge graph by calculating a similarity score between the embeddings relating to the natural language input query and embeddings relating to entities and embeddings relating to relationships.

15. The method of any preceding claim, further comprising : using the retrieval augmented decoder, generating an output on the basis of the natural language input query.

16. The method of any preceding claim, wherein a size of the contextual information for the query is reduced over time as embeddings are optimised.

17. Apparatus (600) comprising a processor (601), and a memory (602) coupled to the processor (601), the memory (602) configured to store program code (603) executable by the processor (601), the program code (603) comprising one or more instructions for modifying vector representations for a retrieval augmented decoder configured to generate an output on the basis of a natural language input query, wherein the vector representations comprise multiple interdependent text embeddings configured to provide contextual information for the query from a pair of jointly observable heterogeneous information sources for the decoder, wherein the vector representations comprise: a first set of embeddings logically defining a first set of clusters for a first information source of the pair of heterogeneous information sources for the decoder, the first set of clusters defining a cluster dictionary of the first set of clusters and comprising indexed subsets of the first set of embeddings, wherein each vector representation of the first set of embeddings comprises an embedding for respective portions of text, each portion of text defining a chunk for input text data stored in a low tier storage apparatus; a second set of embeddings logically defining a second set of clusters for a second information source of the pair of heterogeneous information sources for the decoder, the second set of clusters defining a cluster dictionary of the second set of clusters and comprising indexed subsets of the second set of embeddings, wherein each vector representation of the second set of embeddings comprises an embedding for respective entities and for respective relationships between the entities, wherein the entities and relationships define a knowledge graph, wherein the one or more instructions are configured to cause the apparatus to: receive a natural language input query at a proxy of the retrieval augmented decoder, wherein the proxy of the retrieval augmented decoder comprises a large language model configured to generate respective attention scores for tokens of the chunks and for tokens of groups of words forming natural language sentences derived using the knowledge graph; calculate a set of similarity values to quantify: a relationship between the natural language input query and a chunk, and a relationship between the natural language input query and a sentence derived using the knowledge graph; calculate, on the basis of the attention scores using an encoder comprising a large language model, a set of loss functions; andusing the set of loss functions, modify the vector representations, whereby to minimize a distance between the attention scores and the similarity values.

18. The apparatus of claim 17, wherein the first set of embeddings and the second set of embeddings are stored in the low tier storage apparatus or a high tier storage apparatus.

19. The apparatus of claim 17 or 18, wherein the one or more instructions are configured to cause the apparatus (600) to: process each portion of text defining respective chunks for the input text data to construct the knowledge graph; and augment the entities and relationships of the knowledge graph using an external information source, wherein the external information source comprises at least metadata relating to the input text data.

20. The apparatus of any of claims 17 to 19, wherein the one or more instructions are configured to cause the apparatus (600) to: store text defining respective chunks for the input text data including their embeddings in a chunk vector database, store relationship data representing relationships between entities in the knowledge graph including their embeddings in a relationship vector database, store entities of the knowledge graph and relationship instances for the knowledge graph including their embeddings, jointly representing a graph structure of the knowledge graph, in a graph database.

21. The apparatus of any of claims 17 to 20, wherein the one or more instructions are configured to cause the apparatus (600) to: inference respective ones of the chunks for input text data, information representing the entities, and information representing the relationships through a pre-trained language encoder model to generate chunk embeddings, the entity embeddings, and the relationship embeddings, respectively.

22. The apparatus of any of claims 17 to 21, wherein the one or more instructions are configured to cause the apparatus (600) to: cluster the first set of embeddings, whereby to form the first set of clusters; generate the cluster dictionary of the first set of clusters, wherein a cluster index is mapped to a list of chunk indices belonging to the cluster; map each cluster index to a centroid of the cluster index; and store the cluster dictionary of the first set of clusters in a high tier storage apparatus.

23. The apparatus of any of claims 17 to 23, wherein the one or more instructions are configured to cause the apparatus (600) to: cluster the entity embeddings belonging to the second set of embeddings, whereby to form the second set of clusters; form a subgraph for the knowledge graph for each cluster of the second set of embeddings by maintaining relationship instances connecting entities within the same cluster; generate the cluster dictionary of the second set of clusters, wherein a subgraph index is mapped to a list of entity indices belonging to the subgraph; map each subgraph index to a centroid belonging to the subgraph; and store cluster dictionary of the second set of clusters in the high tier storage apparatus.

24. The apparatus of any of claims 17 to 23, wherein the one or more instructions are configured to cause the apparatus to: pre-process each data chunk, wherein the pre-processing comprises at least one of text cleaning, filtering, and deduplication.

25. The apparatus of any of claims 17 to 24, wherein the one or more instructions are configured to cause the apparatus (600) to: generate each portion of text defining a chunk for input text data using a chunking algorithm comprising a fixed and non-overlapping sliding window.

26. The apparatus of any of claims 17 to 25, wherein the one or more instructions are configured to cause the apparatus (600) to: monitor access to each of the clusters of the first set of clusters and the subgraphs of the second set of clusters using egress variables, wherein the egress variables indicate a number of time units since a last access.

27. The apparatus of claim 26, wherein the one or more instructions are configured to cause the apparatus (600) to: periodically check the egress variables, and, in response to determining that an egress variable of a cluster and / or a subgraph exceeds a system parameter, offloading the cluster and / or the subgraph to the low tier storage apparatus.

28. The apparatus of any of claims 17 to 27, wherein the one or more instructions are configured to cause the apparatus (600) to: promote the cluster and / or the subgraph to the high tier storage apparatus in the event that the cluster and / or the subgraph is requested and / or to be used as the result of an input query.

29. The apparatus of any of claims 17 to 28, wherein the one or more instructions are configured to cause the apparatus (600) to: calculate a similarity score between an embedding relating to the natural language input query and vector representations of chunks, wherein the embedding relating to the natural language input query is generated using a pretrained model; and calculate a similarity score between the embedding relating to the natural language input query and vector representations of the sentences derived using the knowledge graph by calculating a similarity score between the embeddings relating to the natural language input query and embeddings relating to entities and embeddings relating to relationships.

30. The apparatus of any of claims 17 to 29, wherein the one or more instructions are configured to cause the apparatus (600) to: using the retrieval augmented decoder, generate an output on the basis of the natural language input query.

Images