Document content recall method and system, storage medium and program product

Abstract

The invention relates to the technical field of computers, in particular to a document content recall method, a document content recall system for implementing the method, a computer readable storage medium for implementing the method and a computer program product. The method comprises the following steps: acquiring an original result output by a retrieval enhancement generation model for user questions; based on the original result and the cue word, utilizing a large language model to obtain an expansion result aiming at the user question; semantic similarity matching is carried out on the expansion result and information vectors in a vector database, so that the information vectors matched with the expansion result and index information of the information vectors are obtained, and the vector database is used in the retrieval stage of the retrieval enhancement generation model; and integrating the original document content corresponding to the information vector, the index information, the original result and / or the expansion result into a final result and feeding back the final result to the user, wherein the index information indicates entry information of the original document content.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically to a document content recall method, a document content recall system implementing the method, a computer-readable storage medium implementing the method, and a computer program product. Background Art

[0002] Generative artificial intelligence (AI), especially technologies based on deep learning models such as the GPT series and BERT, has made significant progress in the field of natural language processing (NLP), especially in text generation tasks. However, generative AI has a series of inherent limitations in practical applications, such as expertise limitations, hallucination problems, and knowledge timeliness problems.

[0003] In response to this, Retrieval-augmented Generation (RAG) technology came into being. RAG refers to retrieving relevant information from an external knowledge base before using a large language model to answer questions. By citing the source of information, users can verify the accuracy of the answer, thereby enhancing trust in the model output. However, existing RAG technology also has a series of limitations in terms of retrieval quality, generation quality, and enhancement process, making it impossible to guarantee that the responses generated based on the existing RAG model are credible and auditable.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] In order to solve or at least alleviate one or more of the above problems, the embodiments of the present application provide a document content recall method, a document content recall system implementing the method, a computer-readable storage medium and a computer program product implementing the method, which are capable of reverse reasoning on the original results output by the model and accurately locating the source of the returned information, thereby ensuring the credibility and auditability of the output content.

[0006] According to a first aspect of the present application, a document content recall method is provided, the method comprising the following steps: obtaining the original result output by a retrieval enhancement generation model in response to a user's question; obtaining an extended result for the user's question using a large language model based on the original result and a prompt word; performing semantic similarity matching between the extended result and an information vector in a vector database to obtain an information vector matching the extended result and its index information, wherein the vector database is a vector database used by the retrieval enhancement generation model in the retrieval stage; and integrating the original document content corresponding to the information vector, the index information, the original result and / or the extended result into a final result and feeding it back to the user, wherein the index information indicates the entry information of the original document content.

[0007] As an alternative or supplement to the above scheme, in a document content recall method according to an embodiment of the present application, the vector database is constructed based on the following steps: extracting text content from local original information to generate knowledge text; segmenting the knowledge text to generate knowledge text blocks, wherein each knowledge text block corresponds to one or more entry information; and vectorizing the knowledge text blocks using an embedding model to generate the information vector.

[0008] As an alternative or supplement to the above scheme, in a document content recall method according to an embodiment of the present application, the vector database is constructed based on the following steps: the information vector and the index information corresponding to the information vector are stored in the vector database in an associated manner.

[0009] As an alternative or supplement to the above scheme, the document content recall method according to an embodiment of the present application also includes: using an embedding model to vectorize the user question to obtain a query vector; performing semantic similarity matching on the query vector with the information vector in the vector database to obtain an information vector matching the query vector; and inputting the original document content corresponding to the information vector matching the query vector and the user question into the large language model to generate the original result.

[0010] As an alternative or supplement to the above solution, the document content recall method according to an embodiment of the present application further includes: establishing a prompt word library, wherein the prompt word library includes one or more initialization prompt words set in response to the user's question.

[0011] As an alternative or supplement to the above scheme, in a document content recall method according to an embodiment of the present application, the initialization prompt words are set based on one or more of the following steps: inputting the user question, the user's historical questions and the historical final results into the large language model to generate an extended question for the user question; using the retrieval enhanced generation model to determine the user's intention and generate intention prompt words; directly inputting the user question into the large language model to obtain a hypothetical response.

[0012] As an alternative or supplement to the above scheme, the document content recall method according to an embodiment of the present application also includes: according to a generative adversarial mechanism, using a random gradient generator and the large language model as generators, and using the vector database as a discriminator, optimizing the one or more initialization prompt words to obtain one or more optimized prompt words.

[0013] As an alternative or supplement to the above scheme, in a document content recall method according to an embodiment of the present application, according to a generative adversarial mechanism, a random gradient generator and the large language model are used as generators, and the vector database is used as a discriminator to optimize the one or more initialization prompt words to obtain one or more optimized prompt words, including: in each iteration, extracting an initialization prompt word from the one or more initialization prompt words; vectorizing the initialization prompt word using an embedding model to obtain a first vector; perturbing the first vector using the random gradient generator to generate multiple similar vectors of the first vector; restoring each similar vector to a corpus; for each corpus, inputting the corpus, the original result and Gaussian noise into the large language model to obtain a random result; determining an optimal vector from the multiple similar vectors based on one or more of the difference between the original result and the random result, the difference between the first vector and the similar vector, and the difference between the random result and the information vector in the vector data volume; and replacing the initialization prompt word with the corpus corresponding to the optimal vector as the basis for the next iteration.

[0014] As an alternative or supplement to the above scheme, in a document content recall method according to an embodiment of the present application, based on one or more of the difference between the original result and the random result, the difference between the first vector and the similarity vector, and the difference between the random result and the information vector in the vector data volume, determining an optimal vector from the multiple similarity vectors includes: calculating the difference in word and sentence form between the original result and the random result as a reconstruction loss; calculating the difference between the first vector and the similarity vector as a perturbation loss; calculating the average value of the difference between the word vector of the random result and the multiple nearest information vectors in the vector database as an adversarial loss; and taking the weighted sum of the reconstruction loss, the perturbation loss and the adversarial loss as an optimization objective function, calculating the value of the optimization objective function for each similarity vector; and determining the similarity vector that minimizes the value of the optimization objective function as the optimal vector.

[0015] As an alternative or supplement to the above scheme, in a document content recall method according to an embodiment of the present application, the stopping condition of the iteration is: the number of iteration repetitions reaches a first threshold; the minimum value of the optimization objective function during this iteration is less than a second threshold; or the difference between the minimum value of the optimization objective function during this iteration and the minimum value of the optimization objective function during the previous iteration is less than a third threshold.

[0016] As an alternative or supplement to the above scheme, in a document content recall method according to an embodiment of the present application, based on the original results and prompt words, using a large language model to obtain extended results for the user's question includes: segmenting the original results to obtain multiple original fragments; and inputting each original fragment and the prompt words in the prompt word library into the large language model to obtain multiple extended results.

[0017] As an alternative or supplement to the above scheme, the document content recall method according to an embodiment of the present application also includes: segmenting each expansion result to obtain multiple expansion fragments; and vectorizing each expansion fragment to obtain a corresponding expansion vector.

[0018] As an alternative or supplement to the above scheme, in a document content recall method according to an embodiment of the present application, the extended result is matched with the information vector in the vector database for semantic similarity to obtain the information vector matching the extended result and its index information, including: for each extended vector, by calculating the distance between vectors, determining the information vector closest to the extended vector from the vector database, and determining the index information of the closest information vector.

[0019] As an alternative or supplement to the above solution, the document content recall method according to an embodiment of the present application also includes: if the information vector matching the extended vector does not exist in the vector database, discarding the extended result corresponding to the extended vector.

[0020] As an alternative or supplement to the above scheme, in a document content recall method according to an embodiment of the present application, the original document content corresponding to the information vector, the index information, the original result and / or the extended result are integrated into the final result, including: for each extended vector, determining the original fragment corresponding to the extended vector by reverse reasoning; adding the index information and / or the original document content corresponding to the information vector after the corresponding original fragment.

[0021] As an alternative or supplement to the above scheme, in a document content recall method according to an embodiment of the present application, integrating the original document content corresponding to the information vector, the index information, the original results and / or the extended results into the final result includes: inputting the original results and the multiple extended results into the large language model, using the multiple extended results as a corpus, and using the sentence structure of the original results as prompt words to generate the final result; adding the index information and / or the original document content corresponding to the information vector after the final result.

[0022] According to the second aspect of the present application, a document content recall system is provided, comprising: a memory; a processor; and a computer program stored on the memory and executable on the processor, wherein the execution of the computer program causes any one of the document content recall methods described in the first aspect of the present application to be executed.

[0023] According to a third aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes instructions, and the instructions, when run, execute any one of the document content recall methods described in the first aspect of the present application.

[0024] According to a fourth aspect of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements any one of the document content recall methods described in the first aspect of the present application.

[0025] Compared with the existing RAG scheme, the document content recall scheme according to one or more embodiments of the present application adds reconstruction and expansion of the original results, reverse reasoning based on the vector database, and the return step of the information source on the basis of the original results generated by the RAG model, thereby significantly improving the comprehensiveness, accuracy and credibility of the returned results. First, by introducing prompt words and reconstructing and expanding the original results with the help of a large language model, the scheme broadens the potential coverage of subsequent information vector retrieval, enabling the system to locate and find more evidence in the original document. Secondly, by mapping the output results of the RAG model to the vector database used by the RAG model in the retrieval stage, the scheme enables the information vectors and their index information that match the extended results to be retrieved. This step uses reverse reasoning of the extended results to deepen the system's understanding of the source of the results, and also provides users with a convenient way to trace the source of the results, thereby enhancing the auditability of the output content. Finally, after completing the above reconstruction, expansion and reverse reasoning, the scheme presents the corresponding original document content and its index information to the user, ensuring that the user can directly access the corresponding information source and improving the credibility of the output content. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or other aspects and advantages of the present application will become clearer and easier to understand through the following description of various aspects in conjunction with the accompanying drawings, in which the same or similar units are represented by the same reference numerals. In the accompanying drawings:

[0027] Figure 1 is a schematic flow chart of a method 10 for generating raw results according to one or more embodiments of the present application;

[0028] Figure 2 is a schematic flow chart of a document content recall method 20 according to one or more embodiments of the present application;

[0029] Figure 3 An exemplary schematic diagram of a Prompt optimization architecture according to one or more embodiments of the present application;

[0030] Figure 4 An exemplary schematic diagram of a reconstruction and reverse reasoning architecture of raw results according to one or more embodiments of the present application; and

[0031] Figure 5 Schematic block diagram of a document content recall system 50 according to one or more embodiments of the present application. DETAILED DESCRIPTION

[0032] The description of the following specific embodiments is merely exemplary in nature and is not intended to limit the disclosed technology or the application and use of the disclosed technology. In addition, it is not intended to be bound by any express or implied theory presented in the aforementioned technical field, background technology or the following specific embodiments.

[0033] In the following detailed description of the embodiments, many specific details are set forth in order to provide a more thorough understanding of the disclosed technology. However, it is apparent to one of ordinary skill in the art that the disclosed technology can be practiced without these specific details. In other instances, well-known features are not described in detail to avoid unnecessarily complicating the description.

[0034] Terms such as "comprising" and "including" indicate that in addition to the units and steps directly and clearly stated in the specification, the technical solution of the present application does not exclude the situation of having other units and steps that are not directly or clearly stated. Terms such as "first" and "second" do not indicate the order of units in terms of time, space, size, etc., but are only used to distinguish between units.

[0035] Hereinafter, various exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings.

[0036] Referring to the accompanying drawings, Figure 1 It is a schematic flowchart of a raw result generating method 10 according to one or more embodiments of the present application.

[0037] The original results described in this article refer to the output of the RAG model after performing the three basic steps of indexing, retrieving, and generating answers to user questions. Figure 1 Step S110 shown refers to an indexing step of a RAG model, step S120 refers to a retrieval step of a RAG model, and step S130 refers to a generation step of a RAG model.

[0038] The indexing step in step S110 is intended to embed the source data and construct a vector database. A vector database is a database specifically used to store and query vectors, and the vectors stored in it come from the vectorization of text, speech, image, video and other data. Because it has a powerful search and analysis engine, the vector database can be used for information query, aggregation and analysis. One or more possible implementations of constructing a vector database are given below.

[0039] First, text content is extracted from local original information to generate knowledge text. For example, text extraction can be performed on the locally collected safety system related information (e.g., text files, voice files, video education, picture files, etc.). Local original information is dynamically updated, that is, the vector database can be dynamically updated based on the updated local original information, thereby ensuring that the vector database can quickly adapt to changes in information timeliness and provide more real-time query results.

[0040] Next, the knowledge text is segmented to generate knowledge text blocks, each of which corresponds to one or more entry information. Text segmentation helps to process and retrieve information more efficiently, thereby reducing the burden on the model and improving the accuracy of information retrieval. Exemplarily, text segmentation can be performed according to the following process: segment the knowledge text in units of words, phrases, sentences or paragraphs; associate each knowledge text block after segmentation with the entry information of the original document content corresponding to the knowledge text block (for example, Article 10, Item 1 of the "Data Security Rules").

[0041] Next, the knowledge text block is vectorized using an embedding model to generate an information vector. Exemplarily, the embedding model includes a self-attention mechanism model, such as BERT (Bidirectional Encoder Representations from Transformers), which can dynamically adjust the meaning of a word according to the context, so that the same word has different vector representations in different contexts. In addition, the information vector can capture the semantic information in the text, so that semantically similar texts are closer in the vector space.

[0042] Finally, the information vector and the index information corresponding to the information vector are stored in association in a vector database. Exemplarily, the information vector v and its index information i may be stored in the form of a vector group (i, v).

[0043] The retrieval step in step S120 is intended to use the vector database generated in the indexing phase to locate and obtain the knowledge text block that is most semantically similar to the user's question through similarity search technology. Optionally, the retrieval step can be decomposed into user question processing and vector database retrieval.

[0044] In the user question processing stage, the user question can be parsed first to understand its intention and keywords, and then the parsed user question is converted into a query vector in a high-dimensional vector space through an embedding model (such as BERT, GPT, etc.). This query vector can capture the semantic information of the user's question. In the vector database retrieval stage, the query vector can be matched with the information vector in the vector database for semantic similarity (for example, through cosine similarity, Euclidean distance, etc.) to obtain an information vector that matches the query vector. Exemplarily, based on the similarity matching results, the information vectors in the vector database that are most similar to the query vector are sorted from high to low according to the similarity, and a certain number of the most relevant information vectors and their corresponding knowledge text blocks are screened out from the sorted results according to a preset threshold or return quantity limit.

[0045] The generation step in step S130 is intended to utilize the generation capability of the Large Language Model (LLM) to generate original results based on the retrieved knowledge text blocks and the user's questions. For example, a Prompt template may be pre-designed, which includes the keywords of the user's question, the retrieved original document content, and possible guide words or prompts to guide the LLM to generate an expected answer.

[0046] The original result generated by the above method 10 may have the following defects:

[0047] (1) The credibility of the original results. Although RAG technology enhances the generation capability of LLM by retrieving a vector database created based on external knowledge, due to the unexplainability of LLM itself, the answers generated by LLM may not always be based on the retrieved content. For example, when the user asks "What are the backup requirements within the company?" and the retrieved original document content is "The internal requirement is to fully back up the database every six months," the original result generated by LLM may be "In accordance with the provisions of the Data Security Management Measures, regular backups, incremental backups, multi-location storage, and encrypted backups should be performed." That is, even if the retrieved original document content is provided, the answer generated by LLM may not be consistent with the retrieved original document content, which means that the credibility of the generated content cannot be guaranteed.

[0048] (2) The problem of accurate location of information sources. As shown in the above example, although the original results output by the RAG model can provide the document name of the information source, it often cannot provide the specific terms or details of the information source, which limits the user's accurate location and verification of the information source and affects the auditability of the original results.

[0049] In this regard, the present application proposes to add reconstruction and expansion of the original results, reverse reasoning based on the vector database, and return steps of the information source on the basis of the original RAG solution (for example, the original result generation method 10), so as to improve the comprehensiveness, accuracy and credibility of the returned results. Figure 2-5 This paper introduces the document content recall scheme based on the original RAG technology in detail.

[0050] Figure 2 Schematic flowchart of a document content recall method 20 according to one or more embodiments of the present application.

[0051] like Figure 2 As shown, in step S210, the original result output by the RAG model for the user's question is obtained. For the method of generating the original result, please refer to the above Figure 1 The description is not repeated here.

[0052] In step S220, based on the original result and Prompt, LLM is used to obtain an extended result. Step S220 aims to reconstruct and extend the original result output by the RAG model by introducing Prompt and using LLM, thereby broadening the potential coverage of subsequent information vector retrieval so as to locate and find more evidence in the original document.

[0053] In one or more possible implementations, a Prompt library is first established for the current user question and / or the original result, and after optimization, the optimized Prompt in the Prompt library is input into the LLM together with the original result.

[0054] In such an implementation, a prompt library is first established, wherein the prompt library includes one or more initialization prompts set for user questions and / or original results. Exemplarily, the prompt library can be initialized according to the aspects of generating result supplements, prompt generation selection, hypothetical reply generation, generalization generation, and associated query generation. In one example, user questions, user historical questions, and historical final results are input into the LLM to generate extended questions for user questions. For example, when the user asks "What are the backup requirements within the company", the extended questions generated by the LLM may include "What regulations should be followed for internal backup" and "What are the standards for company data backup". In another example, the RAG model is used to determine the user's intention and generate an intention prompt. For example, when the user asks "What are the backup requirements within the company", the RAG model determines several intention prompts based on information vector matching and intention judgment, including "data security protection measures query", "information security backup cycle query", and ""System Grading and Classification Guidelines" query" and provides them to the user, who selects to further locate the content of the required query. In another example, the user's question is directly input into the LLM to obtain a hypothetical reply. Hypothetical responses may contain non-real content, but contain hidden keywords that help find similar documents in the knowledge base. For example, prompts obtained using hypothetical responses may include "data backup frequency", "backup storage location", "data encryption and security", "backup verification and testing", "disaster recovery plan", "data retention compliance", etc.

[0055] Since the initialization Prompt in the Prompt library may not be a valid prompt word, for example, it is semantically different from the user's question or the original result, it is impossible to achieve the intention of reconstructing and expanding the original result. In response to this, one or more embodiments of the present application add an optimization step for Prompt, that is, one or more initialization Prompts are optimized according to the generative adversarial mechanism to obtain one or more optimized Prompts.

[0056] The principle of generative adversarial learning is based on game theory and information theory. Its core idea is to make the generator learn how to generate artificial samples close to the real data distribution by training the generator and the discriminator in adversarial training. In the embodiment of the generative adversarial mechanism optimized for Prompt, the generator is a stochastic gradient generator and LLM, and the discriminator is a vector database used by the RAG model in the retrieval stage.

[0057] Specifically, in this adversarial training framework, the stochastic gradient generator and LLM jointly play the role of the generator. The stochastic gradient generator can introduce a certain degree of randomness to help the model explore more generation space. LLM uses its powerful language understanding and generation capabilities to generate new generation results based on the randomly reconstructed prompt. The vector database acts as a discriminator, responsible for evaluating the credibility of the generated content (that is, the new generation results) and the relevance of the generated content to the original document content. It determines whether the generated content is credible and whether sufficient evidence can be found in the original document content by calculating the similarity between the generated content vector and the information vector.

[0058] The following will refer to Figure 3 The following example illustrates the optimized architecture of Prompt. Figure 3 The Prompt optimization architecture shown includes multiple iterations, and each iteration is performed only on one Prompt in the Prompt library.

[0059] Specifically, in each iteration, a prompt to be optimized is extracted from one or more prompts, which may be an initialized prompt or an iteratively optimized prompt. The initialized prompt is vectorized using the embedded model to form a vector V p Afterwards, the vector V is conditioned using a random gradient generator p Perturb to generate multiple vectors V p A new vector that is similar but with a small difference (i.e., a similar vector V1-V n ). Each similarity vector V1-V n Through the inverse process of the embedding model, it is restored to a corresponding corpus (i.e., the new prompt word S1-S n ). Next, for each corpus S1-S n , a corpus (for example, S1), an original result R (that is, the original result obtained in step S210) and Gaussian noise z are input into LLM to obtain a random result R'. In order to obtain the optimal random gradient from the multiple random gradients introduced in this round of iteration, a loss function can be designed to evaluate the introduced random gradient (that is, perturbation).

[0060] In one embodiment, a reconstruction loss function Loss is established. R , which is used to calculate the difference in word form between the original result R and the random result R'. R The calculation is as follows:

[0061] Loss R =D(RR′)

[0062] The function D() is used to calculate the difference between words and sentences, rather than vector differences. R The purpose is to make the results generated by LLM dissimilar to the original results in terms of word form, that is, the literal difference is large.

[0063] In another embodiment, a perturbation loss function Loss is established. N , which is used to calculate the vector V before and after random reconstruction p Similar vectors (e.g., V1, V n ). For example, Loss N The calculation is as follows:

[0064] Loss N =Norm(V P -V1)

[0065] The function Norm() is used to calculate the length or size of a vector. N The purpose is to prevent the change of the vector generated by the random gradient from being too large. Hinge loss is usually used for regularization to optimize the training process.

[0066] In another embodiment, an adversarial loss function Loss is established. D , which is used to calculate the word vector V of the random result R' R' The average value of the difference between the vector and the nearest information vectors in the vector database. D The calculation is as follows:

[0067]

[0068] Among them, V i (i=1,2,...,n) refers to the vectors retrieved from the vector database and V R' The set of the nearest n information vectors, where the function D() is used to calculate the vector difference. D The purpose is to make the result R' generated by LLM semantically similar to the original result R. For example, the above vector difference can be calculated by means of mean absolute error MAE or mean square error MSE. For example, when using MAE, D(V R' -V i )=|V R' -V i |; When using MSE, D(V R' -V i )=(V R' -V i ) 2 .

[0069] For example, according to the above Loss R 、Loss N 、Loss D One or more of the above establishes a total loss function L as the final optimized objective function. Exemplarily, the total loss function L is calculated as follows:

[0070] L = LossR + αLossD + βLossN

[0071] Among them, α and β are hyperparameters that control the target weights and can be determined in advance during the training phase.

[0072] It is understandable that in the optimization process, due to the existence of LLM, it is impossible to directly optimize from the parameter gradient, so it is possible to compare all randomly generated new prompts (that is, S1-S n ), select the new Prompt that minimizes L as the optimal vector, and use the corpus corresponding to the optimal vector to replace the initialized Prompt as the basis for the next iteration. Furthermore, the stochastic gradient corresponding to the selected optimal vector can represent in which direction a small change in the current Prompt space can minimize the L value, which helps determine how to adjust the Prompt (i.e., the optimization direction) in the next iteration.

[0073] Optionally, the above-mentioned stopping condition for the optimization iteration of Prompt can be any of the following: (1) a stopping condition based on the number of iterations - the number of iterations reaches a first threshold, that is, if the number of iterations exceeds a preset value, the algorithm will stop, which helps to prevent the algorithm from infinite looping; (2) a stopping condition based on the objective function value - the minimum value of the optimization objective function in this iteration is less than a second threshold, that is, if the algorithm finds a solution in this iteration so that the value of the objective function is less than or equal to the preset threshold, then it is considered that the algorithm is close enough to the optimal solution and the iteration can be stopped; (3) a stopping condition based on convergence - the difference between the minimum value of the optimization objective function in this iteration and the minimum value of the optimization objective function in the previous iteration is less than a third threshold, that is, check whether the relative change of the minimum value of the objective function between two consecutive iterations is small enough. If so, it is considered that the algorithm has converged and the iteration can be stopped.

[0074] Next, return to step S220, and compare the original result with one or more Prompts in the Prompt library (for example, via Figure 3 The Prompt optimization architecture in the Optimized Prompt is input into the LLM to reconstruct and expand the original results. The expanded results output by the LLM broaden the potential coverage of subsequent information vector retrieval, making it possible to locate and find more evidence in the original document.

[0075] In step S230, semantic similarity matching is performed between the extended result and the information vector in the vector database to obtain the information vector matching the extended result and its index information. It should be noted that the vector database used in step S230 is the same as the vector database used by the RAG model in the retrieval stage. This step aims to use reverse reasoning on the extended results to deepen the system's understanding of the source of the results and provide users with a convenient way to trace the source of the results. In one example, after searching the vector database for similarity matching, the information vector matching the extended result and its index information are as follows: Article 10, Item 1 of the "Data Security Regulations", V1; Article 10, Item 1 of the "Data Security Regulations", V2. Optionally, since the indexes of the two information vectors in the above example are consistent, they can be merged into: Article 10, Item 1 of the "Data Security Regulations", V1 and V2.

[0076] In step S240, the original document content, index information, original results and / or extended results corresponding to the information vector are integrated into the final result and fed back to the user to ensure that the user can directly access the corresponding information source, thereby improving the credibility of the output content. For example, in the above example, V1 and V2 are restored to "data security protection measures" and "data backup" respectively. When integrating the final result, the original document content, original results and / or extended results are first integrated into "implementing corresponding data security protection measures during the development process, including data encryption, data desensitization, data tagging, data backup, log audit, etc.", and then add "This item comes from Article 10, Item 1 of the "Data Security Rules"".

[0077] The "integration" in step S240 is intended to retain the key information in the original document content, original results and / or extended results and merge duplicate information. In one example, the original results and the original document content can be directly merged for deduplication, and then the index information can be added. In another example, the extended results and the original document content can be directly merged for the total loss function, and then the index information can be added. In yet another example, the original results and the extended results can be input into LLM, with multiple extended results as the corpus and the word structure of the original results as the prompt to generate the final result, and the original document content corresponding to the index information and / or information vector is added after the final result.

[0078] Figure 4 FIG. 1 shows an exemplary schematic diagram of the reconstruction and reverse reasoning architecture of the original result according to one or more embodiments of the present application. Figure 4 The process in the illustrated architecture is merely used as a possible implementation of steps S220 - S230 , and is not intended to limit steps S220 - S230 to a particular implementation.

[0079] like Figure 4 As shown in the figure, before the original result is input into LLM for reconstruction and expansion, the original result is first split to obtain multiple original fragments (s1-s n ), and associate each original shard with a Prompt in the Prompt library (e.g., via Figure 3 The Prompt optimization architecture in the optimized Prompt) is input into the LLM to obtain multiple expansion results (r1-r n Then, for each extended result (r1-r n ) to obtain multiple extended shards (for example, r1 is split into s 11 -s 1n ), and vectorize each extended slice to obtain the corresponding extended vector. For each extended vector, by calculating the distance between vectors, the information vector closest to the extended vector is determined from the vector database, and its index information is determined. For example, for the vectorized s 11 Get the vector group (i 11 , v 11 ).

[0080] Optionally, for each extended vector, the original fragment corresponding to the extended vector can be determined by reverse reasoning, and the index information and / or the original document content corresponding to the information vector can be added after the corresponding original fragment. Optionally, if the information vector matching the extended vector does not exist in the vector database, the extended result corresponding to the extended vector is discarded.

[0081] Figure 5 5 is a schematic block diagram of a document content recall system 50 according to one or more embodiments of the present application. The document content recall system 50 includes a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. The execution of the computer program 530 enables the following Figure 1 The method shown in 10, Figure 2 The method shown in 20, Figure 3 The process shown, Figure 4 One or more of the processes shown are performed.

[0082] In addition, as described above, the present application can also be implemented as a computer-readable storage medium, in which is stored a computer-readable storage medium for making the Figure 1 The method shown in 10, Figure 2 The method shown in 20, Figure 3 The process shown, Figure 4Here, as the computer-readable storage medium, various computer-readable storage media such as disks (e.g., magnetic disks, optical disks, etc.), cards (e.g., memory cards, optical cards, etc.), semiconductor memories (e.g., ROMs, nonvolatile memories, etc.), and tapes (e.g., magnetic tapes, cassette tapes, etc.) can be used.

[0083] The present application may also be implemented as a computer program product, which includes a computer program. When the computer program is executed by a processor, the following Figure 1 The method shown in 10, Figure 2 The method shown in 20, Figure 3 The process shown, Figure 4 The procedure of the process shown.

[0084] In the applicable situation, hardware, software or a combination of hardware and software can be used to realize the various embodiments provided by the application. Moreover, in the applicable situation, without departing from the scope of the application, the various hardware components and / or software components set forth herein can be combined into a composite component comprising software, hardware and / or both. In the applicable situation, without departing from the scope of the application, the various hardware components and / or software components set forth herein can be divided into subcomponents comprising software, hardware or both. In addition, in the applicable situation, it is contemplated that the software component can be implemented as a hardware component, and vice versa.

[0085] Software according to the present application (such as program code and / or data) can be stored on one or more computer-readable storage media. It is also contemplated that the software identified herein can be implemented using one or more general or special-purpose computers and / or computer systems, networked and / or otherwise. Where applicable, the order of the various steps described herein can be changed, combined into composite steps, and / or divided into sub-steps to provide the features described herein.

[0086] The embodiments and examples set forth herein are provided to best illustrate embodiments according to the present application and its specific applications, and thereby enable those skilled in the art to implement and use the present application. However, those skilled in the art will appreciate that the above description and examples are provided only for ease of illustration and example. The description set forth is not intended to cover all aspects of the present application or to limit the present application to the precise form disclosed.

Claims

1. A document content recall method, characterized in that: The method comprises the following steps: Obtain the original results output by the retrieval enhancement generation model in response to user questions; Based on the original result and the prompt word, using a large language model to obtain an extended result for the user's question; Performing semantic similarity matching between the expansion result and the information vector in a vector database to obtain the information vector and index information thereof that match the expansion result, wherein the vector database is the vector database used by the retrieval enhancement generation model in the retrieval stage; and The original document content corresponding to the information vector, the index information, the original result and / or the extended result are integrated into a final result and fed back to the user, wherein the index information indicates the item information of the original document content.

2. The document content recall method according to claim 1, characterized in that: The vector database is constructed based on the following steps: Extract text content from local original information to generate knowledge text; Segmenting the knowledge text to generate knowledge text blocks, wherein each knowledge text block corresponds to one or more item information; as well as The knowledge text block is vectorized using an embedding model to generate the information vector.

3. The document content recall method according to claim 1 or 2, characterized in that: The vector database is constructed based on the following steps: The information vector and index information corresponding to the information vector are stored in the vector database in association with each other.

4. The document content recall method according to claim 1, characterized in that: The method further comprises: Vectorizing the user's question using an embedding model to obtain a query vector; Performing semantic similarity matching between the query vector and the information vector in the vector database to obtain an information vector matching the query vector; The original document content corresponding to the information vector matching the query vector and the user question are input into the large language model to generate the original result.

5. The document content recall method according to claim 1, characterized in that: The method further comprises: A prompt word library is established, wherein the prompt word library includes one or more initialization prompt words set in response to the user's question.

6. The document content recall method according to claim 5, characterized in that: The initialization prompt word is set based on one or more of the following steps: Inputting the user question, the user's historical questions and the historical final results into the large language model to generate an extended question for the user's question; Determine the user's intention and generate intention prompt words by using the retrieval enhancement generation model; The user question is directly input into the large language model to obtain a hypothesized response.

7. The document content recall method according to claim 5, characterized in that: The method further comprises: According to the generative adversarial mechanism, the random gradient generator and the large language model are used as generators, and the vector database is used as a discriminator to optimize the one or more initialization prompt words to obtain one or more optimized prompt words.

8. The document content recall method according to claim 7, characterized in that: According to the generative adversarial mechanism, the random gradient generator and the large language model are used as generators, and the vector database is used as a discriminator to optimize the one or more initialization prompt words to obtain one or more optimized prompt words including: In each iteration, an initialization prompt word is extracted from the one or more initialization prompt words; Vectorizing the initialization prompt word using an embedding model to obtain a first vector; Using the random gradient generator to perturb the first vector to generate a plurality of similar vectors of the first vector; Restore each similarity vector into a corpus; For each corpus, input the corpus, the original result and Gaussian noise into the large language model to obtain a random result; Determine an optimal vector from the plurality of similar vectors based on one or more of a difference between the original result and the random result, a difference between the first vector and the similar vector, and a difference between the random result and an information vector in the vector data volume; and The initialization prompt word is replaced with the corpus corresponding to the optimal vector to serve as the basis for the next iteration.

9. The document content recall method according to claim 8, characterized in that: Based on one or more of a difference between the original result and the random result, a difference between the first vector and the similar vector, and a difference between the random result and an information vector in the vector data volume, determining an optimal vector from the plurality of similar vectors comprises: Calculating the difference in word form between the original result and the random result as a reconstruction loss; Calculating a difference between the first vector and the similar vector as a perturbation loss; Calculate the average of the differences between the word vector of the random result and the multiple nearest information vectors in the vector database as the adversarial loss; and A weighted sum of the reconstruction loss, the perturbation loss and the adversarial loss is used as an optimization objective function, and a value of the optimization objective function is calculated for each similarity vector; The similarity vector that minimizes the value of the optimization objective function is determined as the optimal vector.

10. The document content recall method according to claim 9, characterized in that: The stopping condition of the iteration is: The number of iterations reaches a first threshold; The minimum value of the optimization objective function in this iteration is less than the second threshold; or The difference between the minimum value of the optimization objective function in the current iteration and the minimum value of the optimization objective function in the previous iteration is less than the third threshold.

11. The document content recall method according to claim 1, characterized in that: Based on the original result and the prompt word, the expanded result for the user's question obtained by using the large language model includes: Slicing the original result to obtain a plurality of original slices; and Each original segment and the prompt word in the prompt word library are input into the large language model to obtain multiple expansion results.

12. The document content recall method according to claim 11, characterized in that: The method further comprises: Splitting each expansion result to obtain multiple expansion shards; and Each extended slice is vectorized to obtain a corresponding extended vector.

13. The document content recall method according to claim 12, characterized in that: The expansion result is matched with the information vector in the vector database for semantic similarity to obtain the information vector matching the expansion result and its index information, including: For each extended vector, by calculating the distance between vectors, the information vector closest to the extended vector is determined from the vector database, and the index information of the closest information vector is determined.

14. The document content recall method according to claim 13, characterized in that: The method further comprises: If the information vector matching the extended vector does not exist in the vector database, the extended result corresponding to the extended vector is discarded.

15. The document content recall method according to claim 13, characterized in that: Integrating the original document content corresponding to the information vector, the index information, the original result and / or the extended result into a final result includes: For each extended vector, determine the original slice corresponding to the extended vector by reverse reasoning; The index information and / or the original document content corresponding to the information vector are added after the corresponding original fragment.

16. The document content recall method according to claim 13, characterized in that: Integrating the original document content corresponding to the information vector, the index information, the original result and / or the extended result into a final result includes: Inputting the original result and the multiple expanded results into the large language model, using the multiple expanded results as a corpus and the word and sentence structure of the original result as a prompt word to generate the final result; The index information and / or the original document content corresponding to the information vector are added after the final result.

17. A document content recall system, characterized in that: Include: Memory; Processor; and A computer program stored on the memory and executable on the processor, the execution of which results in the following operations: Obtain the original results output by the retrieval enhancement generation model in response to user questions; Based on the original result and the prompt word, using a large language model to obtain an extended result for the user's question; Performing semantic similarity matching between the expansion result and the information vector in a vector database to obtain the information vector and index information thereof that match the expansion result, wherein the vector database is the vector database used by the retrieval enhancement generation model in the retrieval stage; and The original document content corresponding to the information vector, the index information, the original result and / or the extended result are integrated into a final result and fed back to the user, wherein the index information indicates the item information of the original document content.

18. The document content recall system according to claim 17, characterized in that: The vector database is constructed based on the following steps: Extract text content from local original information to generate knowledge text; Segmenting the knowledge text to generate knowledge text blocks, wherein each knowledge text block corresponds to one or more item information; as well as The knowledge text block is vectorized using an embedding model to generate the information vector.

19. The document content recall system according to claim 17 or 18, characterized in that: The vector database is constructed based on the following steps: The information vector and index information corresponding to the information vector are stored in the vector database in association with each other.

20. The document content recall system according to claim 17, characterized in that: The execution of the computer program further results in the following operations: Vectorizing the user's question using an embedding model to obtain a query vector; Performing semantic similarity matching between the query vector and the information vector in the vector database to obtain an information vector matching the query vector; The original document content corresponding to the information vector matching the query vector and the user question are input into the large language model to generate the original result.

21. The document content recall system according to claim 17, characterized in that: The execution of the computer program further results in the following operations: A prompt word library is established, wherein the prompt word library includes one or more initialization prompt words set in response to the user's question.

22. The document content recall system according to claim 21, characterized in that: The initialization prompt word is set based on one or more of the following steps: Inputting the user question, the user's historical questions and the historical final results into the large language model to generate an extended question for the user's question; Determine the user's intention and generate intention prompt words by using the retrieval enhancement generation model; The user question is directly input into the large language model to obtain a hypothesized response.

23. The document content recall system according to claim 21, characterized in that: The execution of the computer program further results in the following operations: According to the generative adversarial mechanism, the random gradient generator and the large language model are used as generators, and the vector database is used as a discriminator to optimize the one or more initialization prompt words to obtain one or more optimized prompt words.

24. The document content recall system according to claim 23, characterized in that: According to the generative adversarial mechanism, the random gradient generator and the large language model are used as generators, and the vector database is used as a discriminator to optimize the one or more initialization prompt words to obtain one or more optimized prompt words including: In each iteration, an initialization prompt word is extracted from the one or more initialization prompt words; Vectorizing the initialization prompt word using an embedding model to obtain a first vector; Using the random gradient generator to perturb the first vector to generate a plurality of similar vectors of the first vector; Restore each similarity vector into a corpus; For each corpus, input the corpus, the original result and Gaussian noise into the large language model to obtain a random result; Determine an optimal vector from the plurality of similar vectors based on one or more of a difference between the original result and the random result, a difference between the first vector and the similar vector, and a difference between the random result and an information vector in the vector data volume; and The initialization prompt word is replaced by the corpus corresponding to the optimal vector to serve as the basis for the next iteration.

25. The document content recall system according to claim 24, characterized in that: Based on one or more of a difference between the original result and the random result, a difference between the first vector and the similar vector, and a difference between the random result and an information vector in the vector data volume, determining an optimal vector from the plurality of similar vectors comprises: Calculating the difference in word form between the original result and the random result as a reconstruction loss; Calculating a difference between the first vector and the similar vector as a perturbation loss; Calculate the average of the differences between the word vector of the random result and the multiple nearest information vectors in the vector database as the adversarial loss; and A weighted sum of the reconstruction loss, the perturbation loss and the adversarial loss is used as an optimization objective function, and a value of the optimization objective function is calculated for each similarity vector; The similarity vector that minimizes the value of the optimization objective function is determined as the optimal vector.

26. The document content recall system according to claim 25, characterized in that: The stopping condition of the iteration is: The number of iterations reaches a first threshold; The minimum value of the optimization objective function in this iteration is less than the second threshold; or The difference between the minimum value of the optimization objective function in the current iteration and the minimum value of the optimization objective function in the previous iteration is less than the third threshold.

27. The document content recall system according to claim 17, characterized in that: Based on the original result and the prompt word, the expanded result for the user's question obtained by using the large language model includes: Slicing the original result to obtain a plurality of original slices; and Each original segment and the prompt word in the prompt word library are input into the large language model to obtain multiple expansion results.

28. The document content recall system according to claim 27, characterized in that: The execution of the computer program further results in the following operations: Splitting each expansion result to obtain multiple expansion shards; and Each extended slice is vectorized to obtain a corresponding extended vector.

29. The document content recall system according to claim 28, characterized in that: The expansion result is matched with the information vector in the vector database for semantic similarity to obtain the information vector matching the expansion result and its index information, including: For each extended vector, by calculating the distance between vectors, the information vector closest to the extended vector is determined from the vector database, and the index information of the closest information vector is determined.

30. The document content recall system according to claim 29, characterized in that: The execution of the computer program further results in the following operations: If the information vector matching the extended vector does not exist in the vector database, the extended result corresponding to the extended vector is discarded.

31. The document content recall system according to claim 29, characterized in that: Integrating the original document content corresponding to the information vector, the index information, the original result and / or the extended result into a final result includes: For each extended vector, determine the original slice corresponding to the extended vector by reverse reasoning; The index information and / or the original document content corresponding to the information vector are added after the corresponding original fragment.

32. The document content recall system according to claim 29, characterized in that: Integrating the original document content corresponding to the information vector, the index information, the original result and / or the extended result into a final result includes: Inputting the original result and the multiple expanded results into the large language model, using the multiple expanded results as a corpus and the word and sentence structure of the original result as a prompt word to generate the final result; The index information and / or the original document content corresponding to the information vector are added after the final result.

33. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions, which, when executed, execute the document content recall method according to any one of claims 1-16.

34. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the document content recall method according to any one of claims 1 to 16.

Images