Large language model privacy protection type retrieval enhancement generation method and system

Abstract

The invention provides a large language model privacy protection type retrieval enhancement generation method and system, and belongs to the field of artificial intelligence security, and the method comprises the following steps: parameterizing each initial document in a text form knowledge base to generate a parameter form knowledge base; retrieving in the text form knowledge base based on user query to obtain a plurality of document indexes; obtaining parameterized representations corresponding to the document indexes from a parametric form knowledge base; sending the user query and the parameterized representation to a server side; receiving a reasoning result returned by the server side; the reasoning result is obtained after the server side conducts reasoning on user query through a reasoning model; the reasoning model is obtained after the server side updates the weight of the large language model based on each parameterized representation. According to the method, the private document of the user is converted into the model parameter for representation, and the parameter is fused into the large language model during reasoning, so that the large language model can still keep reasoning accuracy on the premise of ensuring data privacy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and in particular to a method and system for enhancing privacy-preserving retrieval based on large language models. Background Technology

[0002] With the rapid development of large language model technology, Retrieval-Augmented Generation (RAG) has become an important technical approach to improve model performance. In a typical RAG system, users need to send their private documents to an API provider. The system retrieves relevant documents and uses them as context input into the large language model to achieve question answering based on user knowledge.

[0003] However, this architecture has serious flaws in terms of privacy protection. Users need to send the original document content, which often contains sensitive business information or personal privacy, to an external server, thus losing control over the data and creating the risk of malicious acquisition, storage, or misuse. Existing data privacy protection technologies are difficult to apply directly to this scenario. While traditional encryption methods can ensure data confidentiality, the encrypted text becomes unreadable to large language models, thereby compromising their semantic understanding and reasoning capabilities. Differential privacy technology protects privacy by injecting noise, but this can impair the accuracy of knowledge and affect the reliability of the generated results. Homomorphic encryption, while theoretically supporting ciphertext computation, is impractical for large-scale language models due to its enormous computational overhead. Therefore, how to ensure data privacy while maintaining the reasoning accuracy of large language models is a pressing problem that needs to be solved in current technological development. Summary of the Invention

[0004] This invention provides a method, system, electronic device, storage medium, and computer program product for generating privacy-preserving retrieval enhancements for large language models, in order to overcome the deficiencies in the prior art and enable large language models to maintain inference accuracy while ensuring data privacy.

[0005] This invention provides a privacy-preserving retrieval enhancement generation method for large language models, applied to the user end, including: Each initial document in the user's private text-based knowledge base is parameterized to generate a parametric knowledge base, and a mutual index is established between the text-based knowledge base and the parametric knowledge base. Based on the user query, a retrieval is performed in the text-based knowledge base to obtain multiple document indexes related to the user query; Based on the mutual indexing mechanism, the parameterized representation corresponding to each document index is obtained from the parameterized knowledge base; The user query and the parameterized representation are sent to the server. The system receives the inference result returned by the server; the inference result is obtained by the server through inference model after inferring the user query; the inference model is obtained by the server after updating the weights of the large language model based on each parameterized representation.

[0006] According to the privacy-preserving retrieval enhancement generation method for large language models provided by the present invention, the parameterization of each initial document in a user-private text-based knowledge base to generate a parametric knowledge base includes: For any of the initial documents, the initial documents are processed to generate multiple sets of question-answer pairs; The initial document is semantically rewritten to generate multiple rewritten documents; Construct a training dataset; the training dataset includes multiple training samples, and each set of training samples includes a set of question-answer pairs and a rewritten document; Based on the training dataset, the large language model is fine-tuned using low-rank adaptation to obtain the parameterized representation of the initial document; All the parameterized representations are indexed and stored to generate the parameterized knowledge base.

[0007] According to the present invention, a privacy-preserving retrieval enhancement generation method for a large language model includes performing low-rank adaptive fine-tuning on the large language model based on the training dataset to obtain the parameterized representation corresponding to the initial document, comprising: In the Transformer structure of each layer of the large language model, the linear mapping weight matrix that needs to be updated is selected. While keeping the original weights unchanged, a low-rank increment composed of a low-rank decomposition matrix is ​​injected into the linear mapping weight matrix. The low-rank decomposition matrix is ​​trained using a language modeling loss function. During the training process, a set of question-answer pairs and a concatenated sequence of rewritten documents from the training dataset are used as input samples. The parameters of the low-rank decomposition matrix are optimized by minimizing the language modeling loss function. The optimized set of all the low-rank decomposition matrices is used as the parameterized representation of the initial document.

[0008] According to the privacy-preserving retrieval enhancement generation method for large language models provided by the present invention, the low-rank increment is expressed by the following formula: ; in, For the low-rank increment, Let be the original weight matrix of the j-th linear mapping in the l-th layer, where α is the scaling factor and r is the lower rank order. and These are low-rank decomposition matrices that match the dimensions of the original weight matrix; The language modeling loss function is expressed by the following formula: ; in, Let t be the target word in the input sample. Let T be the sequence of tokens preceding the target token, and T be the total number of tokens in the input sample.

[0009] This invention also provides a privacy-preserving retrieval enhancement generation method for large language models, applied on the server side, including: The system receives user queries and parameterized representations sent by the user terminal. The parameterized representations are generated by the user terminal parameterizing each initial document in the user's private text-based knowledge base, creating a parameterized knowledge base, establishing a mutual index between the text-based knowledge base and the parameterized knowledge base, and retrieving multiple document indexes related to the user query from the text-based knowledge base based on the user query, and obtaining them from the parameterized knowledge base based on the mutual indexing mechanism. The weights of the large language model are updated based on the parameterized representation to obtain the inference model; The user query is input into the inference model, and the inference result output by the inference model is obtained; The inference result is sent to the user terminal.

[0010] According to the present invention, a privacy-preserving retrieval enhancement generation method for large language models, wherein updating the weights of the large language model based on the parameterized representation to obtain an inference model includes: In the Transformer structure of each layer of the large language model, determine the original weight matrix corresponding to each layer; For each of the original weight matrices, the original weight matrices are linearly fused and updated based on the low-rank decomposition matrix contained in the parameterized representation to obtain the adapted weight matrix. All the aforementioned adaptation weight matrices are combined to form an inference model, which is used to infer the user query.

[0011] This invention also provides a privacy-preserving retrieval enhancement generation system for large language models, applied to the user end, including: The first processing module is used to parameterize each initial document in the user's private text-based knowledge base, generate a parametric knowledge base, and establish a mutual index between the text-based knowledge base and the parametric knowledge base. The first processing module is also used to perform a retrieval in the text-based knowledge base based on the user query to obtain multiple document indexes related to the user query. The first processing module is also used to obtain the parameterized representation corresponding to each document index from the parameterized knowledge base based on the mutual indexing mechanism; The first sending module is used to send the user query and the parameterized representation to the server. The first receiving module is used to receive the inference result transmitted back by the server; the inference result is obtained by the server through inference model after inferring the user query; the inference model is obtained by the server after updating the weights of the large language model based on each parameterized representation.

[0012] This invention also provides a privacy-preserving retrieval enhancement generation system for large language models, applied on the server side, comprising: The second receiving module is used to receive user queries and parameterized representations sent by the user terminal; the parameterized representation is generated by the user terminal parameterizing each initial document in the user's private text-based knowledge base, generating a parameterized knowledge base, establishing a mutual index between the text-based knowledge base and the parameterized knowledge base, and searching in the text-based knowledge base based on the user query to obtain multiple document indexes related to the user query, and obtaining them from the parameterized knowledge base based on the mutual indexing mechanism; The second processing module is used to update the weights of the large language model based on the parameterized representation to obtain the inference model; The second processing module is also used to input the user query into the inference model and obtain the inference result output by the inference model; The second sending module is used to send the inference result to the user terminal.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the privacy-preserving retrieval enhancement generation method for large language models as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the privacy-preserving retrieval enhancement generation method for large language models as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the privacy-preserving retrieval enhancement generation method for large language models as described above.

[0016] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: By irreversibly converting the user's private initial document into a parameterized representation, the original textual knowledge is encoded into model parameters invisible to the server, fundamentally protecting the privacy of the user's statically stored data. By retrieving relevant documents locally on the user's end and extracting only the corresponding parameterized representation, it is ensured that the user's query intent and document selection process are not known externally, avoiding privacy leaks in intermediate stages. Furthermore, by sending only the user's query and parameterized representation to the server, and having the server dynamically update the large language model based on this parameterized representation, the large language model can effectively utilize private knowledge for reasoning without transmitting the original document content. This achieves the goal of maintaining reasoning accuracy while ensuring data privacy. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts of the privacy-preserving retrieval enhancement generation method for large language models provided by the present invention.

[0019] Figure 2 This is the second flowchart of the privacy-preserving retrieval enhancement generation method for large language models provided by this invention.

[0020] Figure 3 This is the third flowchart of the privacy-preserving retrieval enhancement generation method for large language models provided by this invention.

[0021] Figure 4 This is the fourth flowchart of the privacy-preserving retrieval enhancement generation method for large language models provided by this invention.

[0022] Figure 5 This is one of the structural schematic diagrams of the privacy-preserving retrieval enhancement generation system for large language models provided by the present invention.

[0023] Figure 6 This is the second schematic diagram of the structure of the privacy-preserving retrieval enhancement generation system for large language models provided by this invention.

[0024] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] It should be noted that in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships according to the accompanying drawings, are only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the system or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0027] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0028] The following is combined Figures 1-7 This invention describes the privacy-preserving retrieval enhancement generation method, system, electronic device, storage medium, and computer program product provided by the present invention.

[0029] This method can be applied to the user end, see reference. Figure 1 , Figure 1 This is one of the flowcharts illustrating the privacy-preserving retrieval enhancement generation method for large language models provided by this invention, such as... Figure 1 As shown, steps 101 to 105 are included: Step 101: Parameterize each initial document in the user's private text-based knowledge base to generate a parametric knowledge base, and establish a mutual index between the text-based knowledge base and the parametric knowledge base.

[0030] Specifically, a text-based knowledge base refers to a collection of user knowledge that uses natural language text as the basic storage unit, typically including internal enterprise documents, technical reports, operating procedures, and other knowledge-based documents.

[0031] Parametric knowledge bases refer to the transformation of textual knowledge into a set of representations composed of numerical parameters through parameterization. Each parameter representation corresponds to an initial document and is used to express the semantic information contained in the document without exposing the original text content.

[0032] Cross-indexing refers to establishing a bidirectional association between two types of knowledge bases to ensure that users can quickly locate the corresponding parameter representation when searching a text-based knowledge base.

[0033] In this implementation, when parameterizing documents in a text-based knowledge base, a large language model with language understanding and generation capabilities is first invoked to process each initial document offline according to a preset parameterization strategy, generating parameterized results related to the document content.

[0034] The parameterized results are then structured, indexed, and stored to form a parametric knowledge base. Each parameterized result corresponds to a numerical representation of the document's semantic features, and can replace the original document as knowledge input in subsequent reasoning processes.

[0035] Through the indexing process, a bidirectional mapping is established between the text-based knowledge base and the parameter-based knowledge base, ensuring that each initial document has a unique correspondence in both types of knowledge bases, thus supporting subsequent retrieval and retrieval.

[0036] In one specific implementation, refer to Figure 2 , Figure 2 This is the second flowchart of the privacy-preserving retrieval enhancement generation method for large language models provided by the present invention. Step 101 specifically includes steps 201 to 205: Step 201: For any initial document, process the initial document to generate multiple sets of question-answer pairs.

[0037] Specifically, the initial document refers to the basic document unit that constitutes the user's private text-based knowledge base, and each document contains knowledge content that the language model can learn.

[0038] A question-and-answer pair consists of a question and an answer. The question represents the knowledge points contained in the document, and the answer corresponds to the content in the document that can be directly or indirectly deduced.

[0039] In this implementation, for each initial document, the document content is first preprocessed to remove formatting symbols, whitespace, or semantically irrelevant tags, thereby obtaining standardized input text.

[0040] Subsequently, a large language model is invoked to perform semantic analysis on the standardized text. This model, possessing text understanding, semantic analysis, and generation capabilities, is used to extract knowledge from the original document and construct structured question-and-answer samples. Through its contextual understanding capabilities, the model identifies key concepts, factual descriptions, and logical relationships within the text, and automatically generates multiple sets of questions covering the document's main knowledge points based on these semantic elements. For each question, the large language model further generates corresponding answers through content backtracking and contextual reasoning, ultimately forming a set of question-and-answer pairs. ,in For document indexing, This serves as the question-answer pair index. The generated set of question-answer pairs is stored as intermediate data for building subsequent training datasets.

[0041] Step 202: Perform semantically equivalent rewriting on the initial document to generate multiple rewritten documents.

[0042] Rewriting a document refers to obtaining multiple text versions by adjusting the expression or language style while maintaining semantic consistency with the original document. Each rewritten document expresses the same core knowledge.

[0043] In this implementation, for each initial document, the document content is first input into a large language model. The core semantic vectors are extracted through the semantic understanding layer of the large language model, and different language expressions are generated based on preset rewriting instructions. Rewriting instructions include, but are not limited to, syntactic structure changes, synonym replacements, narrative order adjustments, and style transformations.

[0044] During the generation process, the large language model performs semantic consistency checks on each candidate rewriting result to ensure that the rewritten text remains consistent with the original text in terms of factual content and logical relationships. For multiple generated rewriting results, a text similarity algorithm is further used to filter them, removing duplicate or excessively similar texts and retaining only a set of rewritten documents with broad semantic coverage and diverse linguistic styles. ,in, This is used to rewrite the version index. Multiple rewritten documents are stored as diverse input samples for parameterized training, which improves the model's robustness and generalization ability during knowledge learning.

[0045] Step 203: Construct the training dataset; the training dataset includes multiple training samples, each set of training samples includes a question-answer pair and a rewritten document.

[0046] During implementation, the multiple rewritten documents generated from the initial document are first paired with their corresponding question-and-answer pairs. For each question-and-answer pair, a rewritten version is selected from the rewritten document set, and this rewritten document is combined with the question-and-answer pair to form a training sample. Subsequently, all training samples are uniformly organized into a data sequence conforming to the model input format, and data encoding is completed according to the preset sample structure. All encoded training samples together constitute the training dataset for each initial document i. .

[0047] Step 204: Perform low-rank adaptive fine-tuning on the large language model based on the training dataset to obtain the parameterized representation of the initial document.

[0048] Specifically, low-rank adaptive fine-tuning is a method for updating parameters of large language models. By introducing additional low-rank incremental parameters while keeping the original model weights unchanged, it achieves model customization for specific tasks or knowledge with relatively low computational and storage overhead. Parameterized representation refers to the model representation composed of low-rank incremental parameters after fine-tuning. This representation is used to capture the semantic information contained in the initial document and can be used to replace the original text in subsequent reasoning processes.

[0049] In one specific implementation, refer to Figure 3 , Figure 3 This is the third flowchart of the privacy-preserving retrieval enhancement generation method for large language models provided by this invention. Step 204 specifically includes steps 301 to 303: Step 301: In the Transformer structure of each layer of the large language model, select the linear mapping weight matrix that needs to be updated. While keeping the original weights unchanged, inject the low-rank increment composed of low-rank decomposition matrices into the linear mapping weight matrix.

[0050] Specifically, the Large Language Model (LLM) is a pre-trained language model based on the Transformer architecture. This architecture consists of multiple stacked Transformer layers, each containing components such as an attention mechanism and a feedforward neural network. A typical implementation for selecting the linear mapping weight matrices is to update two linear mapping weight matrices within the feedforward network (FFN) of each Transformer layer. These two weight matrices are responsible for transforming and dimensionality-upgrading / reducing the features during the model's inference process.

[0051] When injecting low-rank increments, the original weights of the large language model remain unchanged, meaning that no gradient updates are performed on the original weights in subsequent training stages. The advantage of this is that it preserves the generalization ability of the base model while learning document-specific knowledge through the increments. Low-rank increments are constructed from low-rank factorization matrices; this representation approximates a high-dimensional matrix with fewer parameters, significantly reducing the computational resources and storage overhead required for training.

[0052] The process of injecting low-rank increments into the linear mapping weight matrix is ​​specifically represented by the following formula: ; in, For low-rank increments, Let be the original weight matrix of the j-th linear mapping in layer l. α is the scaling factor, which is a hyperparameter used to adjust the influence of the low-rank increment on the original weights. r is the low-rank order, which is also a hyperparameter that determines the intermediate dimension of the low-rank decomposition matrix, thus affecting the capacity and complexity of the parameterized representation. and These are the low-rank decomposition matrices that match the dimensions of the original weight matrix. If the original weight matrix... If the dimension is d1×d2, then the low-rank decomposition matrix is... The dimension is d1×r, a low-rank decomposition matrix. The dimension is r×d². In subsequent steps, only and The parameters will be trained and optimized, while the original weight matrix... The document remains frozen. In this way, the document-specific knowledge is encoded into these low-rank decomposition matrices.

[0053] Step 302: Train the low-rank decomposition matrix using the language modeling loss function. During training, use a concatenated sequence of a question-answer pair and a rewritten document from the training dataset as input samples, and optimize the parameters of the low-rank decomposition matrix by minimizing the language modeling loss function.

[0054] Specifically, the goal of the training process is to enable the large language model, after injecting low-rank increments, to generate correct answers based on document content and corresponding questions. To this end, the training samples constructed in the preceding steps—a rewritten document, a question, and an answer—are concatenated into a continuous text sequence as input. This input sample's structure allows the model to simultaneously access the document content as background knowledge, the guiding question, and the answer as the target for generation when processing the sequence.

[0055] The core of training lies in optimizing the language modeling loss function. This function aims to improve the accuracy of the model's prediction of the next word in a sequence. The language modeling loss function is specifically expressed by the following formula: ; in, L This represents the total loss value for the input sample. Let t be the target word in the input sample. Let T be the sequence of tokens preceding the target token, and T be the total number of tokens in the input sample. The model represents the sequence of preceding words. When, predict the next word / character. The probability of x. x can be specifically expressed as... .

[0056] At each step of training, by minimizing the loss function L To perform parameter optimization, this process typically employs gradient descent or its variants. When calculating gradients and updating parameters, only the low-rank factorization matrix is ​​considered. and Adjustments were made to the original weights of the large language model. This process remains frozen. By iterating this process repeatedly, the parameters of the low-rank factorization matrix are continuously optimized, increasing the probability that the model will generate a sequence of correct answers given the document and the question. Maximize. In this way, the knowledge related to the initial document is effectively encoded into these optimized low-rank decomposition matrices.

[0057] Step 303: Use the optimized set of all low-rank decomposition matrices as the parameterized representation of the initial document.

[0058] Specifically, the optimized set of all low-rank decomposition matrices refers to a set of low-rank decomposition matrices associated with each selected linear mapping weight matrix in the chosen Transformer structure of each layer in the large language model, whose parameters have been fully trained by minimizing the language modeling loss function. Let represent the parameterized representation of the i-th initial document, then this set can be denoted as This set encompasses all trainable parameters introduced and optimized for learning the knowledge of this initial document.

[0059] This parameterized representation is a purely numerical data structure that encodes the semantic knowledge contained in the initial document into a series of specific model parameter increments. This representation does not contain any of the original text or word sequences from the initial document. Because this parameterized representation is generated from high-dimensional text data through a complex model training process, it lacks direct human readability and is difficult to reverse engineer to recover the complete content of the initial document. In this way, the document's knowledge is effectively preserved in the parameters, while the privacy information of its original text form is protected. This parameterization process is independently repeated for each initial document in the text-based knowledge base, thereby generating a unique, corresponding parameterized representation for each document.

[0060] After generating corresponding parameterized representations for all initial documents in the text-based knowledge base, step 205 is performed: all parameterized representations are indexed and stored to generate the parameterized knowledge base.

[0061] Specifically, indexed storage refers to each parameterized representation generated in the preceding steps. The process of organizing and managing documents begins with assigning a unique document index, such as document number i, to each initial document. Subsequently, this document index is associated with its corresponding parameterized representation. Associative storage is employed. This storage method constitutes a key-value database, where the document index serves as the key, and the complete parameterized representation (i.e., a set of low-rank decomposition matrices) serves as the value. After storing the parameterized representations of all initial documents in this manner, a parametric knowledge base is formed.

[0062] While generating the parametric knowledge base, it is also necessary to establish a mutual index between the text-based knowledge base and the parametric knowledge base. The mutual indexing mechanism is a bidirectional mapping relationship. On one hand, it records the mapping from each document index to its storage location in the parametric knowledge base; on the other hand, it also records the mapping from each document index to its original document storage location in the text-based knowledge base. Through this mutual indexing, in the subsequent online retrieval stage, the corresponding parametric representation can be quickly and accurately located and extracted from the parametric knowledge base based on the document index retrieved in the text-based knowledge base. This step completes all the work in the offline parametric stage, laying the data foundation for subsequent online privacy-preserving inference.

[0063] After the offline parameterization phase is completed, when a user initiates a query, step 102 is executed: based on the user query, a retrieval is performed in the text-based knowledge base to obtain multiple document indexes related to the user query.

[0064] This retrieval process is executed locally on the user's device, ensuring that the user's query content and original document data are not sent to external servers.

[0065] Specifically, a user query is a question posed by a user in natural language. The text-based knowledge base is a private collection of original documents stored by the user. The goal of the retrieval is to identify the batch of documents most relevant to the user's query from this knowledge base. This retrieval operation is implemented using either sparse or dense retrieval methods.

[0066] One implementation method is to use a sparse retrieval approach. This method calculates relevance based on the principles of word frequency statistics and keyword matching. For example, using the BM25 algorithm or TF-IDF algorithm, a relevance score is calculated for each document in the text-based knowledge base and the user query. This score reflects the distribution and importance of the keywords in the query within the documents.

[0067] Another approach is to employ a dense retrieval method. This method utilizes deep learning models to understand the semantics of the text. First, a pre-trained semantic encoding model is used to convert the user query and each document in the knowledge base into a high-dimensional dense vector. These vectors capture the deep semantic information of the text. Then, the relevance of the document to the query is determined by calculating the similarity (e.g., cosine similarity) between the user query vector and each document vector.

[0068] After calculating the relevance of all documents to the user query, the documents are sorted from highest to lowest relevance score. A predetermined number (e.g., k) of the top-ranked documents are selected. The final output of this step is not the content of the documents themselves, but rather the unique identifiers of these k most relevant documents in the knowledge base, i.e., the document indexes. This set of document indexes, for example, is denoted as... This will be used in subsequent steps to extract the corresponding parameterized representation.

[0069] After obtaining multiple document indexes related to the user query, step 103 is executed: based on the mutual indexing mechanism, the parameterized representation corresponding to each document index is obtained from the parametric knowledge base.

[0070] This operation is a crucial step in connecting local search results with the privacy-preserving knowledge representation to be sent to the server.

[0071] Specifically, the mutual indexing mechanism is a mapping relationship pre-established during the offline parameterization stage. It uniquely associates each document index with its corresponding parameterized representation stored in the parametric knowledge base. The parametric knowledge base is a structured storage system that stores the parameterized representations of all initial documents in the text-based knowledge base.

[0072] This retrieval operation is performed one by one for each document index output from the previous step. For the document index set... any index in The search is performed in a parametric knowledge base using a mutual indexing mechanism. This search process is equivalent to a key-value query, based on the document index. As a key, its corresponding value, i.e., the initial document, is retrieved from the knowledge base. Parameterized representation This parameterized representation contains all the low-rank factorization matrices optimized for training this document. The data set.

[0073] This process is repeated for all retrieved document indexes, ultimately resulting in a set consisting of multiple parameterized representations, denoted as . This collection contains the complete knowledge encoding of multiple documents most relevant to the user's query, but in the form of purely numerical model parameters, without containing any raw text.

[0074] After obtaining the set of parameterized representations related to the user query, step 104 is executed: the user query and parameterized representations are sent to the server.

[0075] This operation marks the beginning of the interaction between the user and the server, and its purpose is to securely submit all the information required for reasoning to the server for processing.

[0076] Specifically, the data payload of this transmission operation consists of two parts: one part is the original natural language question input by the user, without any modification, i.e., the user query; the other part is the set of parameterized representations corresponding to the multiple documents most relevant to the user query obtained in the previous step, i.e. The set of parameterized representations consists of a series of numerically quantified low-rank decomposition matrices and does not contain any textual information from the original document.

[0077] In one specific implementation, this sending operation encapsulates the user query (typically in string format) along with a set of parameterized representations within a single data packet. This data packet is transmitted from the user's device to a pre-configured server address via a secure network communication protocol, such as HTTPS. This transmission process ensures the confidentiality of the data over the transmission link.

[0078] The key to this operation is that only the text of the question and the parameterized form of the knowledge are transmitted. The user's original document content, as well as any documents or their parameterized representations unrelated to the query, remain locally on the user's device and are not involved in this network transmission, thus protecting the privacy of the user's private knowledge at the data source.

[0079] After the user sends the user query and parameterized representation to the server on the user side, step 105 is executed: receive the inference result returned by the server; the inference result is obtained by the server through inference model after inferring the user query; the inference model is obtained by the server after updating the weights of the large language model based on each parameterized representation.

[0080] This operation is the final step performed by the user in a complete question-and-answer interaction, and its purpose is to obtain and present answers generated based on private knowledge.

[0081] Specifically, the inference result is the final answer generated by the server in response to the user query, typically in the form of natural language text. This inference result is obtained by the server through inference modeling the user query. Furthermore, the inference model is not a fixed model, but a temporary model specifically built by the server for this query. This inference model is obtained by updating the weights of the large language model based on the parameterized representations sent by the user.

[0082] This process means that the server dynamically integrates the knowledge contained in the received parameterized representation into its underlying large language model, thereby generating a customized reasoning model that incorporates the user's relevant private knowledge. Subsequently, the server uses this reasoning model to process user queries and generate answers.

[0083] Therefore, the inference result received by the user is generated by a language model that incorporates relevant document knowledge, without exposing the original document content. After receiving this inference result via network communication, the user presents it to the user, thus completing a full, privacy-protected question-and-answer process.

[0084] This method can also be applied to the server side, see [reference]. Figure 4 , Figure 4 This is the fourth flowchart of the privacy-preserving retrieval enhancement generation method for large language models provided by this invention, the method including steps 401 to 404: Step 401: Receive the user query and parameterized representation sent by the user terminal; the parameterized representation is the parameterization of each initial document in the user's private text-based knowledge base by the user terminal, generating a parameterized knowledge base, establishing a mutual index between the text-based knowledge base and the parameterized knowledge base, and searching in the text-based knowledge base based on the user query to obtain multiple document indexes related to the user query, and obtaining them from the parameterized knowledge base based on the mutual indexing mechanism.

[0085] Specifically, this receiving operation is completed through the server-side network communication interface to obtain data transmitted from the user. The received data contains two parts: one part is the original question raised by the user, i.e., the user query; the other part is a set of parameterized representations related to the query.

[0086] The parameterized representation is the result obtained after the user completes a series of local operations. These local operations include: first, parameterizing each initial document in the user's private text-based knowledge base to generate a parameterized knowledge base and establishing a mutual index between the text-based and parameterized knowledge bases; then, searching the text-based knowledge base based on the user's query to obtain multiple document indexes related to the user's query; and finally, based on this mutual indexing mechanism, retrieving the parameterized representations corresponding to these document indexes from the parameterized knowledge base. The parameterized representation received by the server is essentially a set of numerically quantified low-rank decomposition matrices, which does not contain any directly interpretable original text information, thus ensuring that the user's private document content remains invisible to the server throughout the entire interaction process.

[0087] After receiving the user query and parameterized representation, step 402 is executed: the weights of the large language model are updated based on the parameterized representation to obtain the inference model.

[0088] This operation is the core step in dynamically injecting user-provided private knowledge into the basic model on the server side.

[0089] Specifically, the large language model is the foundational language model deployed on the server side. To ensure parameter compatibility and validity, the large language model maintains consistency with the large language model used by the user side to generate parameterized representations in terms of model architecture and original weights. The update process first determines the original weight matrix for each layer of the Transformer structure in the large language model. These determined original weight matrices correspond to the linear mapping weight matrices selected by the user side during offline parameterization, such as the weight matrices in a feedforward neural network.

[0090] In one specific implementation, step 402 is implemented by the following method: In the process of updating the weights of the large language model to obtain the specific implementation of the inference model, the original weight matrix corresponding to each layer of the Transformer structure of the large language model is first determined.

[0091] To ensure that the parameterized representation can be correctly applied to large language models, the original weight matrix here corresponds completely in position and function to the linear mapping weight matrix selected by the user during the offline parameterization stage for injecting low-rank increments. A typical implementation is to determine the original weight matrices of the two linear mapping layers inside the feedforward neural network in each Transformer structure.

[0092] After determining the original weight matrices, for each original weight matrix, a linear fusion update is performed based on the low-rank decomposition matrix contained in the parameterized representation received from the user, to obtain the fitting weight matrix. This update process is a summation operation, linearly superimposing the parameterized representations corresponding to all documents (e.g., k) related to the user query onto the original weights. For the j-th original weight matrix in the l-th layer of the large language model, its fitting weight matrix is ​​generated by the following formula: in, It is the updated adaptation weight matrix. This is the original weight matrix corresponding to the large language model. The summation term represents the linear fusion of all k parameterized representations sent by the user. This represents the index of the s-th related document. and It is the parameterized representation contained in the s-th related document. In this context, is the low-rank decomposition matrix corresponding to the current weight matrix. α and r are the scaling factor and low-rank order set during offline parameterization, respectively.

[0093] After linearly fusing and updating all selected original weight matrices, the resulting adapted weight matrices are combined to form the inference model. This combination process means that in subsequent inference calculations, these adapted weight matrices, which contain user-private knowledge, will replace the corresponding original weight matrices in the large language model. Other parts of the large language model that are not updated (such as the weights of the attention mechanism, word embedding layers, etc.) remain unchanged. The resulting inference model is a temporary model dynamically customized for this user query and incorporating relevant document knowledge. This model will be directly used to infer the user query.

[0094] After the inference model is built, step 403 is executed: the user query is input into the inference model, and the inference result output by the inference model is obtained.

[0095] This step is the core computational process for generating answers using a customized model that incorporates the user's private knowledge.

[0096] Specifically, the original user query received from the user's end is used as input and provided to the inference model generated in the preceding steps. After receiving the user query, the inference model performs a forward inference calculation. This calculation process typically adopts an autoregressive approach, whereby the model generates a text sequence word by word based on the context of the user query until a complete answer is generated or a preset termination condition is met.

[0097] Because the weights of this inference model incorporate parameterized representations relevant to the user query, the generated text content naturally utilizes the knowledge contained in these documents. The final text sequence output by this process is the inference result produced by the inference model.

[0098] After the inference model outputs the inference result, step 404 is executed: the inference result is sent to the user terminal.

[0099] This operation is the final step completed on the server side, designed to return the generated answer to the requester.

[0100] In practice, the text sequence obtained as the answer in the previous step is encapsulated into a data response packet. This response packet is then transmitted back to the user's device via a network communication protocol along the established communication link.

[0101] This step completes the server-side processing flow for a single user query. After the query is sent, the dynamically generated inference model can be released, and server resources are reclaimed to prepare for the next request.

[0102] It should be noted that the method described in this invention is not limited to a specific hardware implementation or software environment. The embodiments described below are merely one specific implementation of this invention. In an exemplary implementation environment, a hardware device is used, configured with a 3.6 GHz central processing unit, 32 GB of memory, and at least one graphics processing unit (GPU) with 32 GB of video memory. On this device, the method described in this invention is implemented using the Python 3.10 programming language and the PyTorch 2.4.0 deep learning framework.

[0103] Reference Figure 5 , Figure 5 This is one of the structural diagrams of the privacy-preserving retrieval enhancement generation system for large language models provided by the present invention. This system is applied to the user end and includes: The first processing module is used to parameterize each initial document in the user's private text-based knowledge base, generate a parametric knowledge base, and establish a mutual index between the text-based knowledge base and the parametric knowledge base. The first processing module is also used to perform retrieval in a text-based knowledge base based on user queries, and obtain multiple document indexes related to the user queries. The first processing module is also used to obtain the parameterized representation corresponding to each document index from the parameterized knowledge base based on the mutual indexing mechanism; The first sending module is used to send the user query and parameterized representation to the server. The first receiving module is used to receive the inference results transmitted back from the server. The inference results are obtained by the server through inference model after inferring the user query. The inference model is obtained by the server after updating the weights of the large language model based on each parameterized representation.

[0104] Reference Figure 6 , Figure 6 This is the second schematic diagram of the privacy-preserving retrieval enhancement generation system for large language models provided by this invention. The system is applied on the server side and includes: The second receiving module is used to receive user queries and parameterized representations sent by the user terminal. The parameterized representation is generated by the user terminal parameterizing each initial document in the user's private text-based knowledge base, establishing a mutual index between the text-based knowledge base and the parameterized knowledge base, and retrieving multiple document indexes related to the user query in the text-based knowledge base based on the user query, and obtaining them from the parameterized knowledge base based on the mutual indexing mechanism. The second processing module is used to update the weights of the large language model based on the parameterized representation to obtain the inference model; The second processing module is also used to input user queries into the inference model and obtain the inference results output by the inference model. The second sending module is used to send the inference results to the user terminal.

[0105] It should be noted that the privacy-preserving retrieval enhancement generation system for large language models provided by the present invention can execute the privacy-preserving retrieval enhancement generation method for large language models in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0106] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute the large language model privacy-preserving retrieval enhancement generation method provided in the above embodiments.

[0107] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the large language model privacy-preserving retrieval enhancement generation method provided in the above embodiments.

[0109] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the privacy-preserving retrieval enhancement generation method for large language models provided in the above embodiments.

[0110] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A privacy-preserving retrieval enhancement generation method for large language models, characterized in that, Applied to the user end, including: Each initial document in the user's private text-based knowledge base is parameterized to generate a parametric knowledge base, and a mutual index is established between the text-based knowledge base and the parametric knowledge base. Based on the user query, a retrieval is performed in the text-based knowledge base to obtain multiple document indexes related to the user query; Based on the mutual indexing mechanism, the parameterized representation corresponding to each document index is obtained from the parameterized knowledge base; The user query and the parameterized representation are sent to the server. The system receives the inference result returned by the server; the inference result is obtained by the server through inference model after inferring the user query; the inference model is obtained by the server after updating the weights of the large language model based on each parameterized representation.

2. The privacy-preserving retrieval enhancement generation method for large language models according to claim 1, characterized in that, The parameterization of each initial document in the user's private text-based knowledge base to generate a parametric knowledge base includes: For any of the initial documents, the initial documents are processed to generate multiple sets of question-answer pairs; The initial document is semantically rewritten to generate multiple rewritten documents; Construct a training dataset; the training dataset includes multiple training samples, and each set of training samples includes a set of question-answer pairs and a rewritten document; Based on the training dataset, the large language model is subjected to low-rank adaptive fine-tuning to obtain the parameterized representation of the initial document; All the parameterized representations are indexed and stored to generate the parameterized knowledge base.

3. The privacy-preserving retrieval enhancement generation method for large language models according to claim 2, characterized in that, The step of performing low-rank adaptive fine-tuning on the large language model based on the training dataset to obtain the parameterized representation corresponding to the initial document includes: In the Transformer structure of each layer of the large language model, the linear mapping weight matrix that needs to be updated is selected. While keeping the original weights unchanged, a low-rank increment composed of a low-rank decomposition matrix is ​​injected into the linear mapping weight matrix. The low-rank decomposition matrix is ​​trained using a language modeling loss function. During the training process, a set of question-answer pairs and a concatenated sequence of rewritten documents from the training dataset are used as input samples. The parameters of the low-rank decomposition matrix are optimized by minimizing the language modeling loss function. The optimized set of all the low-rank decomposition matrices is used as the parameterized representation of the initial document.

4. The privacy-preserving retrieval enhancement generation method for large language models according to claim 3, characterized in that, The low-rank increment is expressed by the following formula: ； in, For the low-rank increment, Let be the original weight matrix of the j-th linear mapping in the l-th layer, where α is the scaling factor and r is the lower rank order. and These are low-rank decomposition matrices that match the dimensions of the original weight matrix; The language modeling loss function is expressed by the following formula: ； in, Let t be the target word in the input sample. Let T be the sequence of tokens preceding the target token, and T be the total number of tokens in the input sample.

5. A privacy-preserving retrieval enhancement generation method for large language models, characterized in that, Applied to the server side, including: The system receives user queries and parameterized representations sent by the user terminal. The parameterized representations are generated by the user terminal parameterizing each initial document in the user's private text-based knowledge base, creating a parameterized knowledge base, establishing a mutual index between the text-based knowledge base and the parameterized knowledge base, and retrieving multiple document indexes related to the user query from the text-based knowledge base based on the user query, and obtaining them from the parameterized knowledge base based on the mutual indexing mechanism. The weights of the large language model are updated based on the parameterized representation to obtain the inference model; The user query is input into the inference model, and the inference result output by the inference model is obtained; The inference result is sent to the user terminal.

6. The privacy-preserving retrieval enhancement generation method for large language models according to claim 5, characterized in that, The step of updating the weights of the large language model based on the parameterized representation to obtain the inference model includes: In the Transformer structure of each layer of the large language model, the original weight matrix corresponding to each layer is determined; For each of the original weight matrices, the original weight matrices are linearly fused and updated based on the low-rank decomposition matrix contained in the parameterized representation to obtain the adapted weight matrix. All the aforementioned adaptation weight matrices are combined to form an inference model, which is used to infer the user query.

7. A privacy-preserving retrieval enhancement generation system for large language models, characterized in that, Applied to the user end, including: The first processing module is used to parameterize each initial document in the user's private text-based knowledge base, generate a parametric knowledge base, and establish a mutual index between the text-based knowledge base and the parametric knowledge base. The first processing module is also used to perform a retrieval in the text-based knowledge base based on the user query to obtain multiple document indexes related to the user query. The first processing module is also used to obtain the parameterized representation corresponding to each document index from the parameterized knowledge base based on the mutual indexing mechanism; The first sending module is used to send the user query and the parameterized representation to the server. The first receiving module is used to receive the inference result transmitted back by the server; the inference result is obtained by the server through inference model after inferring the user query; the inference model is obtained by the server after updating the weights of the large language model based on each parameterized representation.

8. A privacy-preserving retrieval enhancement generation system for large language models, characterized in that, Applied to the server side, including: The second receiving module is used to receive user queries and parameterized representations sent by the user terminal; the parameterized representation is generated by the user terminal parameterizing each initial document in the user's private text-based knowledge base, generating a parameterized knowledge base, establishing a mutual index between the text-based knowledge base and the parameterized knowledge base, and searching in the text-based knowledge base based on the user query to obtain multiple document indexes related to the user query, and obtaining them from the parameterized knowledge base based on the mutual indexing mechanism; The second processing module is used to update the weights of the large language model based on the parameterized representation to obtain the inference model; The second processing module is also used to input the user query into the inference model and obtain the inference result output by the inference model; The second sending module is used to send the inference result to the user terminal.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the privacy-preserving retrieval enhancement generation method for large language models as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the privacy-preserving retrieval enhancement generation method for large language models as described in any one of claims 1 to 6.

Images