Optimising retrieval augmented generation and large language model applications by iterated learning

Abstract

In some examples, a method comprises dividing each of multiple input text files into chunks using a chunking algorithm to acquire multiple data chunks, indexing each data chunk of the multiple data chunks to obtain multiple chunk indices, wherein each chunk index is associated with a unique data chunk, and storing each data chunk in a chunk vector database, generating multiple chunk embeddings using a pre-trained encoder language model, wherein each chunk embedding is associated with a unique data chunk, applying a clustering algorithm to the multiple chunk embeddings to create multiple clusters, wherein each cluster comprises an index, a centroid, and a list of chunk indices associated with the cluster, defining multiple subspaces using the clustering algorithm and associating each subspace with a low-rank adaptation (LoRA) module, wherein each LoRA module comprises an index, multiple low-rank matrices, a centroid of a corresponding subspace, and embedding indices.

Description

[0001] OPTI MISING RETRIEVAL AUGMENTED GENERATION AND LARGE LANGUAGE MODEL APPLICATIONS BY ITERATED LEARNING

[0002] TECHNICAL FIELD

[0003] The present disclosure relates, in general, to a system combining a decoder-large language model (LLM) with retrieval augmented generation (RAG). Aspects of the disclosure relate to optimising LLM and RAG applications using iterated learning with low-rank adaptation (LoRA).

[0004] BACKGROUND

[0005] Large language models (LLMs) have significantly advanced the field of artificial intelligence, enabling highly sophisticated natural language processing tasks such as text generation, summarisation, and translation. Among various LLM architectures, decoder-based models, such as GPT (generative pre-trained transformer), have gained prominence due to their ability to generate coherent and contextually accurate text. Decoder-LLMs focus on the task of generating outputs by predicting the next word in a sequence, leveraging vast amounts of pre-trained data to achieve remarkable performance. These models are widely used in applications that require the generation of human-like responses or content creation.

[0006] Retrieval-augmented generation (RAG) is an emerging approach that enhances the capabilities of LLMs by integrating an external knowledge retrieval component. RAG systems combine a language model with a retrieval mechanism that searches a database of relevant information to augment the generation process. By incorporating external knowledge, RAG systems can generate more accurate and contextually enriched outputs, particularly when dealing with specialised or less common topics. This approach effectively bridges the gap between the model’s training data and real-time, task-specific knowledge, enabling the generation of responses that are both more relevant and up-to-date.

[0007] The combination of decoder-LLMs with RAG systems brings significant advantages, allowing the language model to draw from external data sources dynamically, rather than relying solely on the information encoded during training. This dynamic retrieval of information enhances the system’s versatility and adaptability, providing the ability to generate outputs that are not only contextually rich but also grounded in current and specific data. This hybrid architecture is particularly beneficial in domains such as customer support, content creation, and data-driven decision-making, where timely and contextually accurate information is critical.

[0008] Despite the advantages of combining decoder-LLMs with RAG systems, existing implementations face several limitations. Current approaches typically update the entire embedding vector database during each iteration of optimising the embedding vector generation. This process is computationally intensive and can lead to inefficiencies, particularly when only a subset of elements in the database requires updating. Furthermore, existing systems do not address the challenge of reducing the optimisation costs associated with embedding vector generation.

[0009] Additionally, current systems do not consider a design that minimises computational load during the offline phase while maintaining low latency during the online phase. As a result, these inefficiencies not only increase the operational costs but also impact the overall performance of the system, particularly in applications that demand real-time responses. Addressing these challenges requires a novel approach that rethinks how decoder-LLM and RAG systems manage embedding vector updates, optimise fine-tuning processes, and leverage computational architecture to balance performance and costeffectiveness. SUMMARY

[0010] An objective of the present disclosure is to provide a mechanism for optimising a system based on a decoder-LLM and RAG by iterated learning with low-rank adaptation (LoRA).

[0011] The foregoing and other objectives are achieved by the features of the independent claims.

[0012] Further implementation forms are apparent from the dependent claims, the description and the Figures.

[0013] A first aspect of the present disclosure provides a method comprising dividing each of multiple input text files into chunks using a chunking algorithm to acquire multiple data chunks, indexing each data chunk of the multiple data chunks to obtain multiple chunk indices, wherein each chunk index is associated with a unique data chunk, and storing each data chunk in a chunk vector database, generating multiple chunk embeddings using a pre-trained encoder language model, wherein each chunk embedding is associated with a unique data chunk, applying a clustering algorithm to the multiple chunk embeddings to create multiple clusters, wherein each cluster comprises an index, a centroid, and a list of chunk indices associated with the cluster, defining multiple subspaces using the clustering algorithm and associating each subspace with a low-rank adaptation (LoRA) module, wherein each LoRA module comprises an index, multiple low-rank matrices, a centroid of a corresponding subspace, and embedding indices, creating a global mapping from each chunk index to the index of a corresponding LoRA module, and storing the global mapping in a memory.

[0014] Accordingly, the present disclosure provides several benefits, including optimising chunk embeddings with iterated learning for online workloads, which limits the update frequency of text embeddings in the database. This approach significantly decreases the amount of Encoder-LLM inference and reduces database access, thereby minimising resource usage, computational complexity, and latency during the offline phase. Additionally, the optimisation of chunk embeddings using parameter-efficient fine-tuning of a single Encoder-LLM with LoRA modules significantly decreases the number of updated parameters during fine-tuning. This reduces the fine-tuning cost while maintaining the inference latency and performance comparable to that of the original Encoder-LLM fine-tuning. Furthermore, stemming from the method, the system architecture based on near-storage computation leverages high-performance Al processors with direct access to DRAM for low-latency operations required for answering online queries, while utilising high-efficiency, low-power ARM processors with fast access to SSDs for offline iterated learning tasks that can tolerate higher latency. This design reduces computational cost and power consumption during the offline phase while achieving low latency in the online phase.

[0015] The method may further comprise, in response to receiving a new query, generating a query embedding using the pre-trained encoder language model, computing a similarity between the query embedding and a centroid of the cluster of the multiple clusters using a distance metric to determine at least one closest cluster of the multiple clusters, wherein the at least one closest cluster comprises a cluster having a high similarity to the query embedding, computing a similarity between the query embedding and all of the chunk embeddings of the at least one closest cluster based on the list of chunk indices associated with the cluster, selecting a first group of closest chunk embeddings based on the computed similarities, retrieving data chunks corresponding to the selected chunk embeddings from the chunk vector database using the chunk indices of the selected chunk embeddings, concatenating the retrieved data chunks to the query to acquire a combined text, tokenising the combined text and processing it using the pre-trained LLM to generate a response, decoding the response, and outputting the response to a user.

[0016] The method may further comprise, in response to receiving a further query from the user, decreasing a size of the first group.

[0017] The method may further comprise creating a cluster dictionary that maps each cluster index to the centroid of the cluster and the list of associated chunk indices, and storing the cluster dictionary in the memory. The method may further comprise, for each received query, storing query-related data in a buffer implemented as a first-in-first out (FIFO) queue, wherein the query-related data comprises the query, the embedding vector of the query and the indices of the closest clusters, wherein the buffer is arranged to store multiple instances of query-related data.

[0018] The method may further comprise, in response to determining that the buffer had reached a predetermined occupancy size, emptying the FIFO queue by removing all query-related data present in the FIFO queue and temporarily storing the removed query-related data in the memory, for each instance of the query-related data stored in the memory, calculating a similarity between the stored query embedding and all chunk embeddings stored in the at least one closest cluster of the multiple clusters, whereby to determine a second group of closest chunk embeddings, wherein the closest chunk embeddings comprise the chunk embeddings having a high similarity to the stored query embedding, retaining the second group of closest chunk embeddings and the chunk indices corresponding to the second group of closest chunk embeddings, retrieving the data chunks associated with the second group of closest chunk embeddings from the vector database and the LoRA indices associated with said data chunks from the global mapping, measuring a relevance of each data chunk of the retrieved data chunks to the query using an encoder-decoder model and a cross-attention mechanism, and modifying the chunk embeddings to match the measured relevance, clustering the chunk embeddings, and updating the chunk embeddings in the chunk vector database, the list of associated chunk indices and the cluster centroids in the cluster dictionary.

[0019] Calculating the similarity between the stored query embedding and the chunk embeddings may comprise defining the similarity between the query and the multiple chunk embeddings using a similarity score.

[0020] The method may further comprise training the matrices of LoRA modules using a loss function that minimises a mean square error between the cross-attention scores and the similarity scores.

[0021] The method may further comprise, in response to determining that the buffer had reached the predetermined occupancy size, grouping closest chunk embeddings and the chunk indices corresponding to all the queries found in the buffer based on the LoRA indices associated with the chunk embeddings, sequentially training the LoRA modules associated with the grouped chunk embeddings by plugging matrices of each LoRA module into the pretrained encoder model and using the corresponding loss function until convergence, and updating the chunk embeddings associated with the LoRA modules stored in the chunk vector database and the low-rank matrices of the multiple low-rank matrices of the LoRA module after the training.

[0022] The method may further comprise adding a new text file to the multiple input text files, wherein each new text file is stored in the low-tier storage, divided into chunks, embedded and indexed in the vector database, computing a similarity between each new chunk embedding and all centroids in the LoRA modules using a distance metric, assigning each new chunk embedding to the closest LoRA module based on the computed similarity, wherein the closest LoRA module comprises the LoRA module having the highest similarity to the new chunk embedding, adding the index of each assigned chunk embedding to an embedding indices list of the corresponding LoRA module, updating the global mapping of chunk indices to their corresponding LoRA indices, processing the chunk embeddings using the pretrained encoder model together with the matrices of the corresponding LoRA module and updating them in the chunk vector database, and clustering all chunk embeddings and updating the cluster centroids in the memory.

[0023] A second aspect of the present disclosure provides an apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform a method described herein.

[0024] The apparatus may comprise a host processor, a second processor; configured to provide hardware acceleration for the host processor, a third processor, configured to provide hardware acceleration for the host processor, wherein the host processor performs the indexing, defining, creating and storing steps described herein, and the second processor performs the generating and applying steps described herein.

[0025] The apparatus may be configured such that, under the control of the host processor and upon receiving an online query, the host processor performs the selecting, concatenating and decoding steps described herein, and the third processor performs the computing, retrieving and processing the combined text using the pre-trained LLM steps described herein.

[0026] The apparatus may be configured such that, under the control of the host processor and upon receiving an offline query the host processor performs the emptying and retaining steps described herein, and the second processor performs the calculating similarity, retrieving, measuring a relevance, modifying the chunk embeddings, clustering and updating steps described herein.

[0027] A third aspect of the present disclosure provides a computer program stored on a non-transitory medium and including code instructions, which, when executed on more or more processors, cause the one or more processors to execute the method described herein.

[0028] These and other aspects of the invention will be apparent from the embodiment(s) described below.

[0029] BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order that the present invention may be more readily understood, embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:

[0031] Fig. 1 is a flow chart of a method according to an example;

[0032] Fig. 2 is a flow chart of a method according to an example;

[0033] Fig. 3 is a flow chart of a process according to an example;

[0034] Fig. 4 is a schematic representation of an encoder-decoder architecture with a fusion-in-decoder method for a teacher-

[0035] LLM according to an example;

[0036] Fig. 5 is a schematic representation of an apparatus according to an example; and

[0037] Fig. 6 is a schematic representation of system architecture according to an example.

[0038] DETAILED DESCRIPTION

[0039] Example embodiments are described below in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein.

[0040] Accordingly, while embodiments can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate. The terminology used herein to describe embodiments is not intended to limit the scope. The articles “a,” “an,” and “the” are singular in that they have a single referent, however the use of the singular form in the present document should not preclude the presence of more than one referent. In other words, elements referred to in the singular can number one or more, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, items, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or groups thereof.

[0041] Unless otherwise defined, all terms (including technical and scientific terms) used herein are to be interpreted as is customary in the art. It will be further understood that terms in common usage should also be interpreted as is customary in the relevant art and not in an idealized or overly formal sense unless expressly so defined herein.

[0042] Large language models (LLMs) are extensive artificial intelligence models based on the transformer architecture, designed for processing natural language text. Input text is initially processed by a tokenizer, such as byte-pair encoding (BPE), before being fed into the LLM. The most common LLM architectures include encoder, decoder, and encoder-decoder transformers, each tailored for different language processing tasks. Encoder and decoder models rely solely on self-attention mechanisms, focusing on the token values from previous layers. In contrast, the encoder-decoder architecture, exemplified by models like Flan-T5, utilises both self-attention and cross-attention, allowing the decoder to generate output based on the entire input sequence processed by the encoder. Techniques like Fusion-in-Decoder can scale LLMs to handle very large input sequences by concatenating multiple encoded input sequences for joint processing by the decoder, improving computational efficiency.

[0043] Recent advancements in LLMs, particularly decoder-based models, have resulted in highly sophisticated models containing hundreds of billions of parameters, such as Llama-2-7b-chat-hf. These models are pretrained on vast text corpora that undergo extensive preprocessing, including data cleaning, filtering, deduplication, and quality evaluation. Decoder-LLMs are often finetuned on specific datasets, such as question-answer pairs, and adjusted to align model behaviour with human feedback. Despite their vast stored knowledge, Decoder-LLMs can generate incorrect responses (hallucinations) or provide outdated or insufficient information. To address this, techniques like retrieval-augmented generation (RAG) enhance model performance by adding query-related text information to the input sequence, thus improving the relevance and accuracy of the generated responses.

[0044] Fine-tuning large LLMs for specific tasks, such as question-answering, becomes increasingly challenging as model size grows. Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method that updates two low-rank matrices instead of the full pretrained attention weight matrix, maintaining computational efficiency, inference latency, and performance. This modular approach enables deploying a single pretrained LLM with LoRA modules selectively added based on task requirements.

[0045] Combining Decoder-LLMs with RAG has proven effective in mitigating issues like hallucinations and outdated knowledge by augmenting the LLM input with additional context. This approach relies on preprocessing large volumes of text into manageable chunks, which are then stored for fast access. When a query is received, a retrieval algorithm selects the most relevant chunks based on a similarity measure, appending them to the query to provide the LLM with enriched context for response generation.

[0046] In addition to chunk-based retrieval, structured retrieval using knowledge graphs (KGs) offers a more refined approach. KGs represent information as entities (nouns) and relationships (verbs), capturing the semantic structure of text in a way that prioritises meaningful elements. The construction of KGs involves processes like named entity recognition, relationship extraction, and coreference resolution. Compared to unstructured text chunks, KGs generally contain less noisy information and can be enhanced with external data, providing a reliable context for query responses.

[0047] For retrieval tasks, text embedding techniques play a crucial role, with methods such as those based on Encoder-LLMs like BERT offering high-quality semantic embeddings. These embeddings translate the semantic meaning of text into vector space, enabling similarity-based searches that are computationally efficient and scalable. Elowever, directly comparing query embeddings against the entire data set can be computationally prohibitive, particularly as data sizes grow.

[0048] To address this, clustering algorithms partition the embedding space into manageable subspaces or clusters, each represented by a centroid. Common algorithms like K-means create spherical clusters, while hierarchical clustering allows for more diverse cluster shapes. Elybrid approaches combining these algorithms can effectively balance performance and computational cost, optimising retrieval tasks in large-scale data environments.

[0049] The retrieval-augmented generation (RAG) approach has gained popularity due to its ability to enhance a Decoder-LLM with query-related information instantly, by reading chunks from a database without requiring additional training. A typical RAG pipeline operates in two phases: offline and online. During the offline phase, raw text from a large corpus is preprocessed, and chunks are extracted. Text embeddings for each chunk are generated using a pretrained Encoder-LLM, followed by clustering of these embeddings using the K-means algorithm, with the centroids stored. The chunks and their corresponding embedding vectors are indexed and stored in a vector database on low-latency storage media.

[0050] In the online phase, the system generates a text embedding for the user query using a pretrained Encoder-LLM. Clusters with centroid embeddings most similar to the query embedding are identified, and similarities between the query embedding and vectors in the selected clusters are computed to extract the top N embeddings. A reranking algorithm then determines which K of the N selected vectors are most relevant, and the indices of these vectors are used to retrieve the corresponding chunks, which are then fed into the Decoder-LLM. In recent RAG implementations, the vector database is stored in DRAM, while raw data is kept on SSDs or HDDs, with the similarity search often being the primary optimisation focus. Some recent advancements optimise the generation of embedding vectors to directly identify the most semantically similar text passages, potentially eliminating the need for reranking.

[0051] Another approach utilises knowledge graphs (KGs) to provide relevant query-related information to a Decoder-LLM without additional training. Here, portions of the KG are retrieved from a database and presented to the Decoder-LLM in textual form. The offline phase involves constructing the KG from textual data and external sources, generating text embeddings for each entity and relationship using a pretrained Encoder-LLM, and storing these embeddings in a vector database on low-latency media.

[0052] During the online phase, entities and relationships in the user query are identified, and their embeddings are generated using a pretrained Encoder-LLM. Similarities between these embeddings and vectors in the vector database are computed to extract the top M most similar entities and relationships. The indices are then used to retrieve the relevant entities and relationships, which are processed to form a subgraph that is textualized and sent to the Decoder-LLM. Typically, frequently accessed parts of the graph database are stored in DRAM, while other sections are kept on SSDs or HDDs. Recent applications have combined chunk-based information with KGs, allowing the top N chunks to be reranked and refined using KGs as a less noisy source of additional context, enhancing the overall retrieval quality and providing more precise responses.

[0053] According to an example, a Decoder-LLM+RAG system is provided, which iteratively optimizes the similarity of chunk embeddings with submitted queries. This optimization is guided by attention scores from a Teacher-LLM that acts as a proxy to the Decoder-LLM network. Advantageously, this mechanism updates only a small number of elements in the chunk embedding vector database at each iteration, thereby limiting the frequency of text embedding updates and significantly reducing the amount of Encoder-LLM inference and database access required during online workload processing. In an example, the system further optimizes chunk embedding vector generation by fine-tuning the Encoder-LLM using parameterefficient fine-tuning (PEFT) based on low-rank adaptation (LoRA). This approach enables the system to decrease the number of updated parameters during fine-tuning, thereby reducing fine-tuning costs while maintaining the inference latency and performance comparable to the original Encoder-LLM fine-tuning. The use of LoRA modules ensures that the efficiency and effectiveness of the Encoder-LLM are preserved without extensive computational overhead. Additionally, the Decoder- LLM+RAG system employs an architecture based on near-storage computation, specifically designed to optimize computational costs and power consumption during the offline phase while achieving low latency during the online phase. High-performance Al processors with direct access to DRAM are utilized for low-latency operations required for answering online queries, whereas offline iterated learning operations, which can tolerate higher latency, are executed by high-efficiency, low-power ARM processors with fast access to SSDs. This configuration reduces computational cost and power consumption during offline processing while maintaining rapid response times for online query handling.

[0054] Examples in the present disclosure can be provided as methods, systems or machine-readable instructions, such as any combination of software, hardware, firmware or the like. Such machine-readable instructions may be included on a computer readable storage medium (including but not limited to disc storage, CD-ROM, optical storage, etc.) having computer readable program codes therein or thereon.

[0055] The present disclosure is described with reference to flow charts and / or block diagrams of the method, devices and systems according to examples of the present disclosure. Although the flow diagrams described above show a specific order of execution, the order of execution may differ from that which is depicted. Blocks described in relation to one flow chart may be combined with those of another flow chart. In some examples, some blocks of the flow diagrams may not be necessary and / or additional blocks may be added. It shall be understood that each flow and / or block in the flow charts and / or block diagrams, as well as combinations of the flows and / or diagrams in the flow charts and / or block diagrams can be realized by machine readable instructions.

[0056] The machine-readable instructions may, for example, be executed by a machine such as a general-purpose computer, user equipment such as a smart device, e.g., a smart phone, a special purpose computer, an embedded processor or processors of other programmable data processing devices to realize the functions described in the description and diagrams. In particular, a processor or processing apparatus may execute the machine-readable instructions. Thus, modules of apparatus (for example, a module implementing a comparator unit, or a firewall structure and so on) may be implemented by a processor executing machine readable instructions stored in a memory, or a processor operating in accordance with instructions embedded in logic circuitry. The term 'processor' is to be interpreted broadly to include a CPU, processing unit, ASIC, logic unit, or programmable gate set etc. The methods and modules may all be performed by a single processor or divided amongst several processors.

[0057] Such machine-readable instructions may also be stored in a computer readable storage that can guide the computer or other programmable data processing devices to operate in a specific mode. For example, the instructions may be provided on a non- transitory computer readable storage medium encoded with instructions, executable by a processor.

[0058] Fig. 1 is a flow chart of a method according to an example. The method comprises, in block 101, dividing, using a chunking algorithm, each of multiple input text files into chunks, whereby to acquire multiple data chunks. Here, the term "data chunk" may refer to a discrete segment of text obtained by dividing larger input text files using a chunking algorithm. A data chunk may comprise a small portion of text, such as a sentence, paragraph, or other defined segment, depending on the chunking criteria used. Any suitable clustering algorithm may be used. In block 102, the method comprises indexing each data chunk of the multiple data chunks, whereby to obtain multiple chunk indices, wherein each chunk index of the multiple chunk indices is associated with a unique data chunk of the multiple data chunks, and storing each data chunk of the multiple data chunks in a chunk vector database. As used herein, the term "chunk index" may refer to a unique identifier assigned to each data chunk within a set of multiple data chunks. By associating each chunk with a specific index, organisation and retrieval of data chunks can be achieved. In other words, the chunk indices may serve as reference points, allowing the system to efficiently locate and manage the corresponding data chunks within the chunk vector database.

[0059] The method comprises, in block 103, generating, using a pre-trained encoder language model, multiple chunk embeddings, wherein each chunk embedding of the multiple chunk embeddings is associated with a unique data chunk of the multiple data chunks. The term "chunk embedding" may refer to a numerical representation of a data chunk generated using a pre-trained encoder language model. This representation may capture the semantic meaning of the text within the data chunk, transforming it into a fixed-size vector that can be used for various computational tasks. Each chunk embedding may uniquely correspond to a specific data chunk.

[0060] In block 104, the method comprises applying a clustering algorithm to the multiple chunk embeddings, whereby to create multiple clusters, wherein each cluster of the multiple clusters comprises an index, a centroid and a list of chunk indices associated with the cluster. In this context, each cluster may represent a set of chunk embeddings that share similar semantic characteristics. Each cluster may be defined by an index, a centroid, and a list of chunk indices associated with the cluster. The centroid may refer to a central point within the cluster, typically representing the average or most representative embedding of the cluster, which helps in determining the similarity of new embeddings to the cluster. The list of chunk indices may identify the specific data chunks that belong to each cluster.

[0061] In block 105, the method comprises defining multiple subspaces using the clustering algorithm and associating each subspace of the multiple subspaces with a low-rank adaptation (LoRA) module, wherein each LoRA module comprises an index, multiple low-rank matrices, a centroid of a corresponding subspace of the multiple subspaces and embedding indices. Here, the term "subspace" may refer to a defined region within the overall embedding space, which is formed by grouping chunk embeddings using the clustering algorithm. Each subspace may be associated with a specific LoRA module that is tailored to capture the unique characteristics of that subspace. In this context, the LoRA module may include an index, multiple low-rank matrices, a centroid of the corresponding subspace, and embedding indices. The centroid may represent the central point of the subspace, reflecting the average properties of the chunk embeddings within that subspace. The low-rank matrices within each LoRA module may be used to fine-tune the model's attention mechanisms. The embedding indices may help identify the specific embeddings that are associated with each subspace.

[0062] The method comprises, in block 106, creating a global mapping from each chunk index of the multiple chunk indices to the index of a corresponding LoRA module of the multiple LoRA modules. In block 107, the method comprises storing the global mapping in a memory. In the context of the present invention, the term "global mapping" may refer to a data structure that associates each chunk index from the set of multiple chunk indices with the index of a corresponding LoRA module from the set of multiple LoRA modules. This mapping may serve as a reference framework that links each data chunk, via its chunk index, to the specific LoRA module responsible for handling its associated subspace.

[0063] To aid understanding of the method, Fig. 2 is a flow chart of a method according to an example. The method 200 may be divided into three processes - a first process 201, a second process 210, and a third process 220. As part of the first process 201, system initialisation 202 may take place. To aid understanding of the first process 201, reference will now be made to Fig. 3. Fig. 3 is a flow chart of a process according to an example. As will be discussed in more detail later in the specification, generally speaking, the process according to the present invention may be divided into two distinct phases - online phase 310 and offline phase 320. In the offline phase 320, in Fig. 3, raw data 301, pre-trained encoder-LLM 302, chunk vector database 303, LoRA 304, teacher-LLM 305 and Retriever Top-N>Ko 306 are depicted. In the online phase 310, decoder-LLM 311, input 312, and retriever Top-Kn313 are depicted.

[0064] During the system initialisation 202, each raw textual file of the existing data 301 may be stored on a low tier storage. Here, the term “low-tier storage” refers to storage solutions that are cost-effective but offer lower performance and slower access speeds compared to high-tier options (such as solid-state drives). Examples of low tier storage may comprise magnetic tape. A copy of each file may then be divided into chunks using a chunking algorithm with a fixed, non-overlapping sliding window. Each chunk may be further preprocessed by employing state-of-the-art methods of text cleaning, filtering, and text deduplication. All of the chunks may be stored in the chunk vector database 303.

[0065] In the second phase of system initialisation 202, the chunks may be interference through a pre-trained contriever encoder-LLM 302. All chunk embeddings may be added to their corresponding textual entry and index on a higher-tier storage, e.g., SSD. The obtained chunk embeddings may be clustered using K-means. As discussed in relation to Fig. 1, a cluster dictionary may be created, mapping each cluster index to its corresponding cluster centroid and the list of chunk indices that belong to that cluster. This setup may facilitate fast access. The cluster dictionary may typically be stored on SSD. Additionally, LoRA modules 304 may be associated with different subspaces defined during the initial clustering process. As discussed earlier in relation to Fig. 1 , each LoRA module may include an index, low-rank matrices, the centroid of its corresponding subspace, and the associated embedding indices. A global mapping that links chunk indices to LoRA indices may also be established, and both the global mapping and the LoRA modules may be stored on SSDs.

[0066] The second process 210 may be triggered whenever a new query is received by the system, activating the online phase 211 responsible for generating a response. The query may be processed using a pretrained Contriever Encoder-LLM. The similarity between the query embedding and all cluster centroids may be computed using L2 distance as a metric. Similarly, L2 distance may be used to determine the similarity between the query embedding and all chunk embedding vectors within the closest clusters, with the closest TOP-Knbeing selected. The indices of these TOP-Kn chunk embeddings may then be used to retrieve the corresponding textual chunks from the vector database. In subsequent iterations (n>l), as the retrieved information is adapted to the user workload, the size of the context Knmay be reduced according to the expression:

[0067] The obtained chunks may be concatenated with the query, tokenised using the BPE algorithm, and input to the Llama-2-7b- chat-hf model. The same BPE algorithm may be used to decode the answer, which is then provided to the user. At the end of the online phase 211, the query, its embedding vector, and the indices of the detected closest clusters may be stored in a buffer implemented as a first-in-first-out (FIFO) queue. A buffer occupancy check 212 may be performed. If the buffer occupancy does not reach a certain size B, measured in the number of episodes (i.e., queries), the process may end 214. Otherwise, the offline phase 213 may commence. It is important to note that after processing the current query in the online phase 211, the system remains ready to handle the next query. If a new query arrives while the offline phase 213 is ongoing, online processing of the new query has priority.

[0068] The first step of the offline phase 213 may involve removing all content from the FIFO queue and temporarily storing it in memory. Given that the buffer contains only a few textual elements, embeddings, and indices, this operation may not impose significant memory overhead. The iteration counter, initially set to 0, may be incremented by 1, and the n-th iteration may then commence. For each episode from the memory, similarity between the query embedding and all chunk embedding vectors associated with the stored cluster indices may be calculated. The closest TOP-N chunk embedding vectors, along with their corresponding indices, may be retained. As in the online phase 211, L2 distance may be the chosen similarity metric. The indices of these TOP-N chunk embeddings may be used to retrieve the corresponding textual chunks from the vector database and their associated LoRA indices.

[0069] To assess the relevance of each chunk for answering a given query, an Encoder-Decoder Flan-T5 Teacher-LLM may be used. To aid understanding of the process, reference will now be made to Fig. 4. Fig. 4 is a schematic representation of an encoderdecoder architecture with a fusion-in-decoder method for a teacher-LLM according to an example. In Fig. 4, a plus sign (‘+’) denotes the operation of concatenation. The dashed arrows inside the encoders 401-1, 401-2 and the decoder 402 indicate selfattention, whereas the regular arrows refer to cross-attention. To assess the significance of each chunk and graph sentence in answering a given query, an Encoder-Decoder Flan-T5 Teacher-LLM may be employed.

[0070] The Fusion-in-Decoder technique may be employed, where each chunk is concatenated with the query, tokenised using the Flan-T5 method, and processed individually through the encoder of the Flan-T5 Teacher-LLM. The token values obtained from the encoder output for each text passage may be concatenated. The decoder of the Flan-T5 Teacher-LLM may then consider these concatenated token values together. The cross-attention scores within the decoder, generated during the first token output, may be used to measure the importance of each chunk in relation to answering the query.

[0071] The attention score Aq cgiven to the chunk c when answering query q may be calculated as the average cross-attention score across all layers, cross-attention heads and tokens associated with c, i.e. A pretrained Contriever Encoder-LLM fine-tuned with LoRA modules may yield a Contriever_chunk model. The loss function for the Contriever_chunk model may be defined to minimise the mean square error (MSE) as: where q iterates over B queries found in memory, c iterates over TOP-N retrieved chunks, and contr and contr c denote Contriever and Contriever_chunk model, respectively.

[0072] In the following, the defined loss function may be used to train the Contriever_chunk models through a backpropagation algorithm. Training may be conducted with LoRA modules, so that all B*N inputs in the Contriever_chunk training dataset may be grouped according to the LoRA indices associated with the retrieved chunks. Each LoRA module may be then sequentially integrated into the pretrained Contriever model and trained with the corresponding loss function, as specified in the equation above, until convergence is achieved. Notably, fine-tuning may be carried out with a small learning rate to ensure minimal changes in system behaviour between iterations. Once a LoRA module has converged, the embedding indices associated with that module may be used to access the corresponding chunks in the vector database, which are processed through the Contriever_chunk model. The updated chunk embeddings are then stored on SSDs within the vector database, and the modified low-rank matrices of the LoRA module are also updated on SSDs.

[0073] After all LoRA modules have converged, clustering of the chunk embeddings may be performed using the current cluster centroids as initialisation for the algorithm. The updated cluster centroids and the indices of embeddings associated with each cluster index are then updated in the cluster dictionary on SSDs. Finally, the buffer content is removed from memory, marking the end of the offline phase 213.

[0074] The third process 220 may be triggered when new data is added to the system, resulting in addition of the new data 221. Similar to the initial data setup, each raw textual file may be stored on a low storage tier, with a copy being divided into chunks using a chunking algorithm that employs a fixed and non-overlapping sliding window. Each chunk may then be preprocessed using standard text cleaning, filtering, and deduplication methods. The new chunks may be indexed and stored on SSDs within a vector database.

[0075] The obtained chunks may be processed through a pretrained Contriever Encoder-LLM. Using L2 distance as a metric, the similarity of each chunk embedding to all centroids within the LoRA modules may be computed, and the closest match may be selected. For each chunk, the chosen LoRA module may correspond to the semantically most similar subspace, as defined by the pretrained Contriever Encoder-LLM. The index of each chunk may be added to the embedding indices list of the selected LoRA module, and the mapping of all chunk indices to their corresponding LoRA indices may be incorporated into the global mapping.

[0076] The new chunks may be grouped by the index of the assigned LoRA module. Each LoRA module is then sequentially integrated into the pretrained Contriever model, and the Contriever_chunk model is processed for chunks assigned to the current LoRA module. The obtained chunk embeddings may be updated in their corresponding textual entries and indices on SSDs. After processing all LoRA modules, clustering of the chunk embeddings may be performed using the current cluster centroids for algorithm initialisation. The updated cluster centroids and the indices of embeddings associated with each cluster index may then be revised in the cluster dictionary on SSDs.

[0077] In an alternative embodiment, the size of the subspace associated with each LoRA module may differ from what has been previously described. It is possible that assigning LoRA modules to subspaces obtained through K-means clustering could result in high granularity. A large number of clusters, and thus LoRA modules, may lead to significant storage overhead and increased computational costs during fine-tuning. Therefore, in this embodiment, the LoRA modules may be assigned to subspaces derived from hierarchical clustering, which might merge some of the clusters obtained through the K-means algorithm.

[0078] In the offline phase, training may be conducted using LoRA modules. Consequently, all B*N inputs in the Contriever_chunk training dataset may be grouped by the LoRA indices with which the retrieved chunks are associated. LoRA modules may then be sequentially integrated into the pretrained Contriever model and trained with the corresponding loss function until convergence. Once a LoRA module has converged, the embedding indices associated with that module may be used to access the corresponding chunks in the vector database, which may be processed through the Contriever_chunk model. The obtained chunk embeddings may be updated on SSDs in the vector database, and the modified low-rank matrices of the LoRA module may be updated on SSDs. After all LoRA modules have converged, clustering of the chunk embeddings may be performed using the current cluster centroids for initialisation. The altered cluster centroids and the embedding indices associated with each cluster index may be updated in the cluster dictionary on SSDs.

[0079] When new data is added to the system, the obtained chunks may be processed through a pretrained Contriever Encoder-LLM. Using L2 distance as a metric, the similarity of each chunk embedding to all centroids contained in the LoRA modules may be computed, and the closest match may be selected. The index of each chunk may be added to the embedding indices list of its selected LoRA module, and the mapping of all chunk indices to their corresponding LoRA indices may be added to the global mapping. The chunks may be grouped by the index of the assigned LoRA module. Each LoRA module may then be sequentially integrated into the pretrained Contriever model, and the Contriever_chunk model may be processed for chunks assigned to the current LoRA module. The obtained chunk embeddings may be added to their corresponding textual entry and index on SSD. After processing all LoRA modules, clustering of the chunk embeddings may be performed using the current cluster centroids for initialisation. The updated cluster centroids and embedding indices associated with each cluster index may be revised in the cluster dictionary on SSDs.

[0080] Fig. 5 is a schematic representation of an apparatus according to an example. The apparatus 500 comprises a processor 501, and a memory 502 coupled to the processor 501 and configured to store instructions or program code 503, executable by the processor 501. The apparatus 500 comprises the program code 503 arranged to cause the apparatus to perform the method described herein.

[0081] Fig. 6 is a schematic representation of system architecture according to an example. The system 600 a host microcontroller 601 , multiple accelerators 602 (also comprising FPGA), an ARM processor 603 (also comprising FPGA) and a DRAM memory controller 604. Although only one accelerator 602 is shown in Fig. 6, the skilled person would understand that the invention is not limited thereto. The system 600 may also comprise a DRAM module 605-1 connected to at least one of the multiple accelerators 602, another DRAM module 605-2 connected to the memory controller 604, a flash translation layer (FTL) controller 606 and a NAND flash chip 607 connected thereto. The system 600 may also comprise a CXL / PCIe interface 608 connected to, inter alia, the host microcontroller 601 , the accelerator 602 and the ARM processor 603. The system 600 may comprise multiple solid-state drives (SSDs). Each SSD may be controlled via the flash translation layer (FTL), which may be connected through a high-speed PCIe interface and CXL protocol 608 to other components within the system 600. The functionalities of the Flash Translation Layer may typically be executed on the host microcontroller 601. To further enhance system implementation, this operation may be offloaded from the host microcontroller 601 and assigned to a dedicated controller located near the storage, such as the FTL controller 606. Consequently, most high-performance computational operations within the proposed system architecture may be performed in proximity to the storage devices, such as the NANE) flash chip 607, thereby accelerating processing and mitigating potential bottlenecks at the host microcontroller 601.

[0082] The host microcontroller 601 may serve as the entrance point to the system 600. It may be the only component connected to all the major parts of the architecture and may send main control signals to execute user workloads.

[0083] During the system initialisation 202, the microcontroller 601 application may preprocess the data and generate chunks. The microcontroller 601 application may then provide the following control flow: a) it may index and store the extracted chunks on SSDs and initiate the inference of the Encoder-LLM on the ARM processors 603; b) it may initiate the process of obtaining clusters for the generated embeddings; c) it may assign a new LoRA module with low-rank matrices to different parts of the high-dimensional space and associate the corresponding centroid and embedding vectors’ indices with it, as well as a LoRA index; it may also create a global mapping of embedding indices to their corresponding LoRA indices, with all data stored on SSDs.

[0084] In the online phase 211, each time a new query arrives at the system 600, the microcontroller 601 application may provide the following control flow: a) for the given query, it may initiate the inference of the Encoder-LLM on the accelerator 602 and find the most similar clusters for the obtained query embedding; b) for the closest clusters and the current value of Kn, it may initiate the finding of the most similar TOP-Kn chunk embeddings on FPGAs; c) it may use the indices of the obtained vectors to address the vector database on SSDs and extract the closest chunks; d) it may merge the query with the acquired context, tokenize the input, and send the obtained sequence to the input of the Decoder-LLM on the accelerator; after inference is complete, it may read the answer and output it to the user; e) for each query, its textual content and embedding, as well as the indices of the closest clusters, may be stored as one episode in a buffer; if the buffer reaches a certain occupancy, the offline phase 213 may be initiated.

[0085] In the offline phase 213, the microcontroller 601 application may provide the following control flow: a) it may remove all episodes from the buffer and temporarily store them in memory, after which the n-th iteration may be initiated; b) for each query, it may initiate the search for the most similar TOP-N chunk embeddings within the clusters stored in memory on FPGAs, where N is a system parameter such that N > K0, for a given K0; c) it may use the indices of the obtained vectors to address the vector database on SSDs and extract the closest chunks; d) it may merge each query with its acquired context, tokenize the input, and send the obtained sequence to the input of the Teacher-LLM on the ARM processors 603: e) for the obtained Teacher-LLM attention scores, it may calculate the loss function which minimises the distance between the obtained attention scores and the similarity scores; f) it may initiate the training of the LoRA modules associated with the indices of the most similar chunks across different queries on the ARM processors 603; g) it may initiate Encoder+LoRA-LLM inference on the ARM processors 603 for the chunks whose indices are associated with the fine-tuned LoRA module and may store the altered low-rank matrices of the LoRA module on SSDs; h) after the fine-tuning is fully finalized, it may initiate clustering with updated embeddings; i) it may remove the buffer content from the memory.

[0086] The accelerator 602 may perform near-memory processing for the computationally intensive workload. Two types of accelerators may be considered: (i) Al workload accelerators, such as Huawei’s Ascend 910s, and (ii) FPGAs, which may be flexibly programmed to accelerate specific computations and to which additional DRAM 605-1 may be assigned if needed.

[0087] During the system initialisation 202, the pretrained Decoder-LLM, the pretrained Encoder-LLM, as well as the pretrained Teacher-LLM may be stored on SSDs. The Encoder-LLM may be loaded onto ARM processors 603 and may be used for inference on the chunks extracted from the existing data. The obtained embedding vectors may be stored in the vector database on SSDs. Once this process is completed, the embedding vectors of all chunks may be clustered by utilising the ARM processors 603. The cluster dictionary mapping each cluster to the respective embedding indices, as well as the cluster centroid, may be directly stored on SSDs.

[0088] In the online phase, the Decoder-LLM, the Encoder-LLM, as well as the cluster centroids, may be preloaded into the accelerator 602. For each new query, the Encoder-LLM may be used for inference on the query. The similarity scores between the obtained query embedding and cluster centroids may be calculated, yielding the most relevant clusters for answer generation. The FPGA- Retrievers may directly access the vector database on SSDs (and potentially also DRAM 605-1) to process the embedding vectors of chunks that belong to the selected clusters. The vectors may be processed in batches, retaining the TOP-Knmost similar chunk embeddings. The Decoder-LLM may process the provided query and context at the input and output the answer.

[0089] In the offline phase 213, at the beginning of each iteration, the Teacher-LLM, the Encoder-LLM, as well as the LoRA modules associated with indices whose chunks belong to the clusters found in the buffer, may be loaded into the ARM processors 603. The FPGA-Retrievers may directly access the vector database on SSDs (and potentially also DRAM 605-1) to process the embedding vectors of chunks that belong to the clusters whose indices are contained in the buffer at the iteration start. For every query, the vectors may be processed in batches, retaining the TOP-N most similar chunk embeddings. After this process is finalised, the Teacher-LLM may be used for inference on each query with its created context, and the amount of crossattention needed to generate the first token at the output of the decoder may be calculated. The attention scores for each text chunk may be obtained by averaging the acquired attention vectors across tokens, attention heads, and layers of the decoder. The LoRA modules associated with the index of at least one chunk within the TOP-N across different queries may be trained to minimise the provided attention-based loss functions. After convergence of each LoRA module, the Encoder+LoRA-LLM may be used for inference on the chunks whose indices belong to that LoRA module. The obtained embedding vectors may be updated in the vector database on SSDs, and the modified low-rank matrices of the LoRA module may be updated on SSDs. After the fine-tuning is finalised, the embedding vectors may be clustered using the ARM processors 603, and the cluster dictionary may be updated in the vector database so that it corresponds to the new clusters. The “heavy” operations, such as online model inference, may typically be performed on an Al accelerator, such as accelerator 602. While accelerator 602 may be powerful, it may also be costly and energy-consuming. To make the system more economical and energy-efficient, it may be envisioned that some operations could be executed on an ARM processor. In the online phase 211, fast processing may be important to deliver the answer to the user with minimal latency . However, during system initialisation 202, the offline phase 213, as well as the process of adding new data 221, may tolerate higher latency. Therefore, the operations associated with these parts of the system may be executed on ARM processors 603.

[0090] An ARM processor 603 may require logic to manage the flow of data to and from the ARM main memory. The memory controller 604 may implement this important functionality, as well as the logic needed to read from and write to DRAM, and to handle the background memory refreshing.

[0091] According to an example, machine-readable instructions can be loaded onto a computer or other programmable data processing devices, so that the computer or other programmable data processing devices perform a series of operations to produce computer-implemented processing, thus the instructions executed on the computer or other programmable devices provide an operation for realizing functions specified by flow(s) in the flow charts and / or block(s) in the block diagrams.

[0092] Further, the teachings herein may be implemented in the form of a computer or software product, such as a non-transitory machine-readable storage medium, the computer software or product being stored in a storage medium and comprising a plurality of instructions, e.g., machine readable instructions, for making a computer device implement the methods recited in the examples of the present disclosure.

[0093] In some examples, some methods can be performed in a cloud-computing or network-based environment. Cloud-computing environments may provide various services and applications via the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessible through a web browser or other remote interface of the user equipment for example. Various functions described herein may be provided through a remote desktop environment or any other cloud-based computing environment.

[0094] While various embodiments have been described and / or illustrated herein in the context of fully functional computing systems, one or more of these exemplary embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable-storage media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the exemplary embodiments disclosed herein. In addition, one or more of the modules described herein may transform data, physical devices, and / or representations of physical devices from one form to another.

[0095] The preceding description has been provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the instant disclosure. The embodiments disclosed herein should be considered in all respects illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of the instant disclosure.

Claims

CLAIMS1. A method comprising : dividing, using a chunking algorithm, each of multiple input text files into chunks, whereby to acquire multiple data chunks (101); indexing each data chunk of the multiple data chunks, whereby to obtain multiple chunk indices, wherein each chunk index of the multiple chunk indices is associated with a unique data chunk of the multiple data chunks, and storing each data chunk of the multiple data chunks in a chunk vector database (102); generating, using a pre-trained encoder language model, multiple chunk embeddings, wherein each chunk embedding of the multiple chunk embeddings is associated with a unique data chunk of the multiple data chunks (103); applying a clustering algorithm to the multiple chunk embeddings, whereby to create multiple clusters, wherein each cluster of the multiple clusters comprises an index, a centroid and a list of chunk indices associated with the cluster (104); defining multiple subspaces using the clustering algorithm and associating each subspace of the multiple subspaces with a low-rank adaptation (LoRA) module, wherein each LoRA module comprises an index, multiple low-rank matrices, a centroid of a corresponding subspace of the multiple subspaces and embedding indices (105); creating a global mapping from each chunk index of the multiple chunk indices to the index of a corresponding LoRA module of the multiple LoRA modules (106); and storing the global mapping in a memory (107).

2. The method of claim 1 , further comprising: in response to receiving a new query, generating, using the pre-trained encoder language model a query embedding; computing a similarity between the query embedding and a centroid of the cluster of the multiple clusters using a distance metric, whereby to determine at least one closest cluster of the multiple clusters, wherein the at least one closest cluster of the multiple clusters comprises a cluster having a high similarity to the query embedding; computing a similarity between the query embedding and all of the chunk embeddings of the at least one closest cluster based on the list of chunk indices associated with the cluster; selecting a first group of closest chunk embeddings based on the computed similarities; retrieving data chunks corresponding to the selected chunk embeddings from the chunk vector database using the chunk indices of the selected chunk embeddings; concatenating the retrieved data chunks to the query, whereby to acquire a combined text, tokenising the combined text and processing the combined text using the pre-trained LLM whereby to generate a response; decoding the response and outputting the response to a user.

3. The method of claim 2, further comprising: in response to receiving a further query from the user, decreasing a size of the first group.

4. The method of claim 2 or 3, further comprising: creating a cluster dictionary that maps each cluster index to the centroid of the cluster and the list of associated chunk indices; and storing the cluster dictionary in the memory.

5. The method of any one of claims 2 to 4, further comprising: for each received query, storing query-related data in a buffer implemented as a first-in-first out (FIFO) queue,wherein the query-related data comprises the query, the embedding vector of the query and the indices of the closest clusters, wherein the buffer is arranged to store multiple instances of query-related data.

6. The method of claim 5, further comprising, in response to determining that the buffer had reached a predetermined occupancy size: emptying the FIFO queue by removing all query-related data present in the FIFO queue and temporarily storing the removed query-related data in the memory; for each instance of the query-related data stored in the memory, calculating a similarity between the stored query embedding and all chunk embeddings stored in the at least one closest cluster of the multiple clusters, whereby to determine a second group of closest chunk embeddings, wherein the closest chunk embeddings comprise the chunk embeddings having a high similarity to the stored query embedding; retaining the second group of closest chunk embeddings and the chunk indices corresponding to the second group of closest chunk embeddings; retrieving the data chunks associated with the second group of closest chunk embeddings from the vector database and the LoRA indices associated with said data chunks from the global mapping; measuring a relevance of each data chunk of the retrieved data chunks to the query using an encoder-decoder model and a cross-attention mechanism, and modifying the chunk embeddings to match the measured relevance; clustering the chunk embeddings; and updating the chunk embeddings in the chunk vector database, the list of associated chunk indices and the cluster centroids in the cluster dictionary.

7. The method of claim 6, wherein calculating the similarity between the stored query embedding and the chunk embeddings comprises defining the similarity between the query and the multiple chunk embeddings using a similarity score.

8. The method of claim 6 or 7, further comprising: training the matrices of LoRA modules using a loss function that minimises a mean square error between the crossattention scores and the similarity scores.

9. The method of any of claims 5 to 8 further comprising, in response to determining that the buffer had reached the predetermined occupancy size: grouping closest chunk embeddings and the chunk indices corresponding to all the queries found in the buffer based on the LoRA indices associated with the chunk embeddings; sequentially training the LoRA modules associated with the grouped chunk embeddings by plugging matrices of each LoRA module into the pretrained encoder model and using the corresponding loss function until convergence; and updating the chunk embeddings associated with the LoRA modules stored in the chunk vector database and the low- rank matrices of the multiple low-rank matrices of the LoRA module after the training.

10. The method of any preceding claim, further comprising: adding a new text file to the multiple input text files, wherein each new text file is stored in a low-tier storage, divided into chunks, embedded and indexed in the vector database; computing a similarity between each new chunk embedding and all centroids in the LoRA modules using a distance metric; assigning each new chunk embedding to the closest LoRA module based on the computed similarity, wherein the closest LoRA module comprises the LoRA module having the highest similarity to the new chunk embedding;adding the index of each assigned chunk embedding to an embedding indices list of the corresponding LoRA module; updating the global mapping of chunk indices to their corresponding LoRA indices; processing the chunk embeddings using the pretrained encoder model together with the matrices of the corresponding LoRA module and updating them in the chunk vector database; and clustering all chunk embeddings and updating the cluster centroids in the memory.

11. An apparatus (500) comprising: at least one processor (501); and at least one memory (502) storing instructions (503) that, when executed by the at least one processor (501), cause the apparatus (500) at least to perform a method according to any preceding claim.

12. Apparatus according to claim 11 , wherein the at least one processor (503) includes: a host processor (601 ); a second processor; configured to provide hardware acceleration for the host processor (601 ); a third processor, configured to provide hardware acceleration for the host processor (601); wherein the host processor (601) performs the indexing, defining, creating and storing steps of claim 1; and the second processor performs the generating and applying steps of claim 1.

13. Apparatus according to claim 12 and further caused at least to perform a method according to claim 2, wherein, under the control of the host processor (601 ) and upon receiving an online query: the host processor (601) performs the selecting, concatenating and decoding steps of claim 2; and the third processor performs the computing, retrieving and processing the combined text using the pre-trained LLM steps of claim 2.

14. Apparatus according to claim 12 and further caused at least to perform a method according to claim 6, wherein, under the control of the host processor (601) and upon receiving an offline query : the host processor (601) performs the emptying and retaining steps of claim 6; and the second processor performs the calculating similarity, retrieving, measuring a relevance, modifying the chunk embeddings, clustering and updating steps of claim 6.

15. A computer program stored on a non-transitory medium and including code instructions, which, when executed on more or more processors, cause the one or more processors to execute the method of any of claims 1 to 10.18

Images