LangChain model-based pdf file processing method and system

Abstract

The invention discloses a pdf file processing method and a pdf file processing system based on a LangChain model. The method comprises the following steps: firstly, acquiring a target pdf file uploaded by a user based on an interface provided by Gradio, and segmenting a text in the target pdf file to form a plurality of segmented documents; and establishing an abstract scene and a question and answer scene based on a LangChain model. The abstract scene transmits the segmented grades into LLM to return to obtain segmented abstracts, and the abstracts are aggregated to obtain the abstract of the whole article; and according to the question and answer scene, the embedding of the original text is matched by using the similarity, and the LLM is further called again to obtain the final answer of the question through proper cue word assembly. According to the method, the summary and question answering of the pdf file are realized through the two established scenes.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence counting, and in particular to a PDF file processing method and system based on a LangChain model. Background Art

[0002] However, the emergence of ChatGPT makes the life of this application more intelligent and convenient. Through ChatGPT, natural language processing and dialogue can be performed. ChatGPT's fluent and natural multi-round dialogue ability close to human expression and powerful problem-solving ability allow it to not only interact with humans, but also complete various specific tasks such as writing copy, papers, codes and translation.

[0003] However, the existing ChatGPT has three important limitations:

[0004] API token restrictions: ChatGPT's API has token restrictions, which limit the length of content returned at a single time and do not allow for conversations based on context.

[0005] Lag of the knowledge base: OpenAI’s Large Language Model (LLM) knowledge base is only up to the end of 2021 and has no retrieval and reasoning capabilities for updated knowledge.

[0006] Does not include local or proprietary data: OpenAI's LLM knowledge base is derived from public data on the Internet. There is no corresponding capability support for proprietary data in certain fields, such as papers published by universities or undisclosed proprietary data.

[0007] Therefore, for the current academic field that requires a lot of reading and retrieval of previous academic research results (especially academic research papers in PDF format), ChatGPT has the above three defects. Summary of the invention

[0008] Based on this, the embodiment of the present application provides a PDF file processing method and system based on the LangChain model, which can enable LangChain to perform PDF file summary and question-and-answer.

[0009] In a first aspect, a PDF file processing method based on a LangChain model is provided, the method comprising:

[0010] Based on the interface provided by Gradio, the target PDF file uploaded by the user is obtained, and the text in the target PDF file is segmented to form multiple segmented documents;

[0011] Based on the LangChain model, we build summary scenarios and question-answering scenarios. Specifically:

[0012] In the summary scenario, multiple segmented documents are passed into the LLM model for processing and returned to obtain the summary of each segment, and the summaries of all segments are aggregated to obtain the summary of the target PDF file;

[0013] In the question-answering scenario, the embedding of each segmented document is calculated and stored in the vector database. The question input by the user is embedded, and the similar answer set is determined by similarity matching the embeddings corresponding to each segmented document. Finally, the question and the similar answer set are assembled through prompt words, and LLM is called again to obtain the target answer to the question.

[0014] Optionally, obtain the target PDF file uploaded by the user based on the interface provided by Gradio, including:

[0015] The PDF is loaded into a document array through PyPDFLoader; each document array contains page content and metadata used to represent the number of pages.

[0016] Optionally, the text in the target PDF file is segmented to form a plurality of segmented documents, including:

[0017] An environment variable is set, wherein the environment variable is used to dynamically set the size of the text block;

[0018] The text in the target PDF file is segmented sequentially by a recursive character text segmenter until the segmented block reaches a set size.

[0019] Optionally, calculate the embeddings for each segment document, including:

[0020] The shibing624 / text2vec-base-chinese model in HuggingFace was selected for text embedding, and the segmented text data was converted into vector representation.

[0021] Optionally, each segment document is calculated to obtain its own embedding and stored in a vector database, including:

[0022] By calling the from_documents method, the segmented text collection and embedding are stored in the local vector database, and the data file is saved in the form of parquet; wherein the vector database is the Chroma vector database.

[0023] Optionally, multiple segmented documents are passed into the LLM model for processing and returned to obtain the summary of each segment, and the summaries of all the segments are aggregated to obtain the summary of the target PDF file, including:

[0024] Through the Map step in the Map Reduce mode, the summary of each segment is calculated and output separately; then the Refine mode is used for cyclic merging, and finally the summary of the target PDF file is obtained.

[0025] Optionally, embedding the question input by the user and determining a set of similar answers by similarity matching the embeddings corresponding to each segmented document also includes:

[0026] Combined with the context of the user's current question and historical conversation, appropriate prompt words are selected to construct a summary, and finally a result vector is produced.

[0027] Optionally, the abstract of the target PDF file is obtained, and the abstract also includes the title, author, organization, keywords, links and conclusion of the article.

[0028] In a second aspect, a pdf file processing system based on the LangChain model is provided, the system comprising:

[0029] The upload module is used to obtain the target PDF file uploaded by the user based on the interface provided by Gradio, and to segment the text in the target PDF file into multiple segmented documents;

[0030] Based on the LangChain model, we build a summary processing module and a question-answer processing module. Specifically:

[0031] In the summary processing module, multiple segmented documents are passed into the LLM model for processing and returned to obtain the summary of each segment, and the summaries of all segments are aggregated to obtain the summary of the target PDF file;

[0032] In the question-answering processing module, the embedding of each segmented document is calculated and stored in the vector database; the question input by the user is embedded, and the similar answer set is determined by similarity matching the embeddings corresponding to each segmented document; finally, the question and the similar answer set are assembled through prompt words, and LLM is called again to obtain the target answer to the question.

[0033] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0034] (1) Compared with the existing model, it saves tokens and improves the summary generation effect for PDF files;

[0035] (2) The usability and practicality of online PDF dialogue capabilities are improved through the set prompt words;

[0036] (3) The parameters in the model can be adjusted according to actual conditions and have strong adaptability in actual operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0038] Figure 1 A pdf file processing flow chart based on the LangChain model provided in an embodiment of the present application;

[0039] Figure 2 This is a schematic diagram of the stuff mode summary chain;

[0040] Figure 3 Schematic diagram of the process of dividing a PDF text into n segments;

[0041] Figure 4 A schematic diagram of the process of dividing another PDF text into n segments;

[0042] Figure 5 This is a schematic diagram of the specific processing process of the segmented document in this embodiment;

[0043] Figure 6 This is a schematic diagram of question and answer based on PDF content in this embodiment;

[0044] Figure 7 This is a schematic diagram of the process of uploading a local PDF file to the server;

[0045] Figure 8 Schematic diagram generated for the abstract;

[0046] Fig. 9 This is a diagram of the parameter adjustment page. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] In the description of the present invention, the terms "comprises", "has" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units added based on further optimization schemes conceived by the present invention.

[0049] At present, in the academic field that requires a lot of reading and retrieval of previous academic research results (especially academic research papers in PDF format), ChatGPT does not have the three capabilities mentioned above. Therefore, it is necessary to find a technical means that can not only utilize ChatGPT's powerful natural language processing capabilities, but also use a large number of academic papers as corpora and knowledge bases (such as the content of PDF files) for summarization, induction, dialogue, question and answer, and other operations. With the emergence of the LangChain framework, a dialogue method (or system) that can interact with PDF files using LLM can be developed. First, the LangChain model is introduced as follows.

[0050] LangChain is a development framework for applications driven by large language models. Applications can not only call large language models through APIs, but also be data-aware: connect large language models to other data sources; and have agent properties: allow large language models to interact with their environment.

[0051] LangChain provides support for several main modules. These modules are arranged in increasing complexity as follows:

[0052] (1) Models: various model types and model integrations supported by LangChain.

[0053] (2) Prompts: including prompt management, prompt optimization and prompt serialization.

[0054] (3) Memory: Memory is the concept of maintaining state between chain / proxy calls. LangChain provides a standard memory interface, a set of memory implementations, and chain / proxy examples that use memory.

[0055] (4) Indexes: used in conjunction with the user’s text data.

[0056] (5) Chains: A chain is not just a single LLM call, but a series of calls (whether it is a call to LLM or a different utility). LangChain provides a standard chain interface and integration with other tools. LangChain provides end-to-end chain calls for common applications.

[0057] (6) Agents: Agents involve LLMs making action decisions, executing the action, looking at an observation, and repeating the process until completion. LangChain provides a standard agent interface, a range of agents to choose from, and examples of end-to-end agents.

[0058] The above modules can be used in a variety of ways.

[0059] LangChain also provides guidance and assistance. Here are some common use cases supported by LangChain:

[0060] (1) Autonomous agents: Long-running agents take multiple steps to try to achieve a goal. AutoGPT and BabyAGI are typical examples.

[0061] (2) Agent simulations: placing agents in a closed environment to observe how they interact with each other and respond to events.

[0062] (3) Personal assistants: Personal assistants need to take actions, remember interactions, and have knowledge of the user's relevant data.

[0063] (4) Question answering: Answering questions in specific documents by constructing answers using only the information in those documents.

[0064] (5) Chatbots: Since language models are good at generating text, they are well suited for creating chatbots.

[0065] (6) Querying Tabular Data: This section allows users to learn how to use LLM to query data stored in a tabular format (csv, SQL, data frame, etc.).

[0066] (7) Code Understanding (code): Users can use this to learn how to use LLM to query source code from GitHub.

[0067] (8) Extraction: Extracting structured information from text.

[0068] (9) Summarization: Summarizes longer documents into shorter, more concise chunks of information. It is a type of data augmentation generation.

[0069] (10) Evaluation: Generative models are extremely difficult to evaluate using traditional metrics. A new evaluation method is to use the language model itself for evaluation. LangChain provides some hints / chains to assist in evaluation.

[0070] In summary, LangChain is an open source framework that aims to systematize the entire process of LLM calculation and application building. It allows this application to use different data sources; it can help this application manage and optimize related prompt words; it can help this application quickly develop applications, such as knowledge base question answering, chatbots, and various tasks in traditional natural language processing.

[0071] In academic application scenarios, it is usually necessary to summarize and retrieve information from academic and scientific research papers in PDF format. By leveraging the capabilities of LLM, people's reading process can be greatly accelerated and facilitated.

[0072] First, this application clarifies the two most common usage scenarios: 1. Summarizing and summarizing PDF texts; 2. Retrieving and answering questions based on PDF content. Figure 1 , which shows a flowchart of a PDF file processing method based on the LangChain model provided in an embodiment of the present application. The method may include the following steps:

[0073] Step 101, based on the interface provided by Gradio, obtain the target PDF file uploaded by the user, and segment the text in the target PDF file to form a plurality of segmented documents.

[0074] In order to realize the above two scenarios, this application needs to enable the target site to have the ability to upload and load PDF files. This application uses Python's Gradio library to build the entire dialogue framework.

[0075] Gradio is a Python library for building interactive interfaces. It helps developers quickly create machine learning and deep learning models with user interfaces. Gradio provides a simple and easy-to-use interface, making it very easy to deploy models as web applications. Based on Gradio, developers have implemented the function of users uploading PDF files and displaying the contents of PDF files on web pages. This function not only has the ability to upload data, but also can read PDF files, making it convenient for users to switch and compare perspectives when answering questions and reading. After the PDF file is uploaded, the capabilities of LangChain are needed. First, the long text input by the user is segmented to form segmented documents (Documents).

[0076] Step 102: Establish summary scenarios and question-answer scenarios based on the LangChain model. Specifically:

[0077] In the summary scenario, multiple segmented documents are passed into the LLM model for processing and the summaries of each segment are returned. The summaries of all the segments are aggregated to obtain the summary of the target PDF file.

[0078] In the embodiment of the present application, the summary scenario will pass the segmented Documents into the LLM API, return the summary of the current segment, and aggregate them according to the refine or map reduce mode to finally obtain the summary of the entire article.

[0079] In the question-answering scenario, the embedding of each segmented document is calculated and stored in the vector database. The question input by the user is embedded, and the similar answer set is determined by similarity matching the embeddings corresponding to each segmented document. Finally, the question and the similar answer set are assembled through prompt words, and LLM is called again to obtain the target answer to the question.

[0080] In the embodiment of the present application, the question-answering scenario calculates the respective embeddings of each Document and stores them in a vector database. The questions input by the user are also embedded, and the embeddings of the original text are matched by similarity to obtain the top K similar ones. Finally, the question and the top K possible answers are assembled through appropriate prompt words, and the LLM is called again to obtain the answer to the final question.

[0081] Specifically, the above steps include:

[0082] S1, load these data into "Document". LangChain provides loaders for various types of files. This application uses a loader for PDF files, namely: PyPDFLoader. Use PyPDFLoader to load PDF into a document array. Each document contains page content and metadata, including page numbers. The advantage of this method is that documents can be retrieved by page number.

[0083] S2, using text splitters, aims to split the text into chunks when processing long texts. Although it sounds simple, there are many potential complexities in the process. The text splitter does the following: split the text into small, semantically meaningful chunks (usually sentences); start combining these small chunks into a larger chunk until a certain size is reached (measured by some function); once the size is reached, treat the chunk as its own text chunk, and then start creating a new text chunk with some overlap (to maintain context between text chunks). This application sets an environment variable: SPLIT_NUM, so that the size of the text chunk can be set dynamically. This embodiment selects the recursive character text splitter (RecursiveCharacterTextSplitter), which tries to perform text splits in sequence until the chunk is small enough to the set chunkSize. The effect of this is to keep all paragraphs (then sentences, then words) together as much as possible, because they generally appear to be the strongest semantically related text fragments.

[0084] S3, select a suitable embedding model to vectorize the text. In this example, the shibing624 / text2vec-base-chinese model in HuggingFace is selected for text embedding, and the segmented text data in the previous step is converted into a vector representation. The embedding process has a series of benefits, such as: reducing the data dimension; retaining semantic information; texts with similar semantics will be close to each other in the embedding space, thereby improving the generalization ability of the algorithm.

[0085] S4, store the text vector obtained in the third step in the vector database (VectorDB). Vector storage is an important part of index construction. This application chooses Chroma, which is an open source, lightweight vector database. By calling the from_documents method, the segmented text collection and embedding converted in the above steps are stored in the local vector database, and the data file is saved in .parquet format.

[0086] The above four steps complete the process of converting a PDF file with many pages into vector storage: first use load to load, then use the text segmenter to segment the text, then use the embedding model to embed the text, and finally store the text vector in the vector database.

[0087] In the embodiment of the present application, in order to generate a long text summary, this solution has designed and tried three text summary solutions, namely: stuff mode, refine mode and map reduce mode.

[0088] Solution 1: Stuff mode

[0089] The stuff pattern is the most direct summary chain, such as Figure 2 As shown in , it accepts a list of documents, inserts them all into the prompt, and passes them to LLM. Since the ChatGPT 3.5 version API has a token limit of 4096, the stuff mode is only applicable to situations where the document content is very small, and it is not applicable to most long text situations.

[0090] Solution 2: Refine mode

[0091] The refine mode summary chain loops through the input document and iteratively updates its answer to construct prompt words. The input is the current document and the intermediate answer. It continues to construct prompt words for summary to obtain further intermediate summaries. In this way, the summary of the entire article can be finally obtained recursively. Figure 3 As shown in: the entire PDF text is divided into n segments from 1 to n. First, the content of segment 1 and segment 2 is constructed with prompt words and input into LLM to obtain intermediate summary 1 of segment 1 and segment 2. Then, the content of segment 3 and the content of intermediate summary 1 are constructed with prompt words and input into LLM to obtain intermediate summary 2. Then, the content of segment 4 and the content of intermediate summary 2 are constructed with prompt words and input into LLM to obtain intermediate summary 3. This cycle repeats until the content of segment n and the summary of the previous segments 1 to n-1 are input into LLM together to obtain an overall summary. This scheme is very suitable for tasks where the number of documents to be analyzed exceeds the model context capacity. Obviously, compared with the stuff mode, refine requires more LLM calls, and some tasks are difficult to complete in iterations, so the speed and performance may be affected.

[0092] Solution 3: Map Reduce Mode

[0093] The summary chain of the Map reduce mode is different from the above two modes. It summarizes the segmented documents separately (Map step). Then all new documents are passed to a separate merge document chain to obtain a single output (Reduce step). If necessary, it can choose to compress or fold the mapped documents first to ensure that they fit into the merge document chain (usually passing them to LLM). This compression step is performed recursively if necessary. Figure 4 As shown in , the entire PDF text is divided into n segments from 1 to n. First, the documents of segments 1 to n are summarized respectively to obtain Summary 1 to Summary n. Then, the contents of Summary 1 to Summary n are constructed into prompt words and input into LLM to obtain the final document summary. Here, it may be necessary to execute step by step. For example, if the documents of Summary 1 to Summary n are still too large, they need to be split, reorganized, and merged again to finally meet the requirements of LLMAPI token restrictions as the input for the final summary output.

[0094] The above three modes are the most commonly used text summary modes in LangChain. In this application, the advantages of the Refine and Map Reduce modes are combined, and the Map step in the Map Reduce mode is used to calculate and output the summary of each segment separately, and then the Refine mode is used for cyclic merging. This makes the overall summary effect better than the single refine mode, and the overall token overhead is less than the single map reduce mode, which is a major innovation of this application.

[0095] In the embodiment of the present application, a conversation tool is proposed that can use PDF files uploaded by users to allow users to have conversations with PDF files and answer questions about the content in PDF files. This application uses LangChain to create a question chain and uses Gradio to develop the user interaction page for the conversation. Unlike ChatGPT, which provides limited data context (only 4096 tokens at most), due to the use of embedding and vector storage, the chatbot of this application will be able to process PDF data and manage large databases. Figure 5 A schematic diagram of the specific processing process of segmented documents in this embodiment is given.

[0096] The overall process of the question-answer chain is as follows Figure 6 As shown in , it mainly includes the following steps:

[0097] Step 1: Use PyPDFLoader to load the PDF file into an array of documents. Each document contains page content and metadata, such as page number. This allows you to retrieve documents by page number.

[0098] Step 2: Use Text Splitters to split long text into semantically meaningful chunks, ensuring that context is preserved. Recursive Character Text Splitter is the splitter of choice in this solution, which performs text splitting sequentially until the set chunk size is reached.

[0099] Step 3: Select a suitable embedding model to convert the segmented text data into vector representation. In this solution, the shibing624 / text2vec-base-chinese model of HuggingFace is used for text embedding to maintain semantic information in the vector space.

[0100] Step 4: Store the text vector in a vector database (VectorDB). This solution uses Chroma as a lightweight vector database. By calling the from_documents method, the segmented text collection and the corresponding text vector are stored in the local vector database, and the data is saved in the .parquet file format.

[0101] Step 5: The questions input by the user will also go through the embedding process and be stored in the vector database in the form of vectors. The session_state['history'] capability is also used here to retain the context of the user's conversation. By combining the context of the user's current question and the historical conversation, appropriate prompt words are selected to construct a summary, and finally a result vector is produced.

[0102] Step 6: Use the similarity matching function of the vector database to perform similarity matching on the user question vector and the segmented vector of the PDF document in the vector space. Use the similarity_search_with_scroe method to calculate the similarity score and obtain the top-K most similar vectors and the corresponding document text.

[0103] Step 7: Use appropriate prompt words, for example: "You are now a geology expert, read the reference materials and answer the user's questions, and reply if you cannot answer them."

[0104] The top-K possible questions are sorted and input into LLM again.

[0105] Step 8: LLM will answer the user's question based on the top-K possible related content to obtain a more accurate answer.

[0106] In the above process, in order to obtain an accurate answer, the model and prompt words need to be debugged. The points that need to be debugged are:

[0107] Setting the chunkSize of document segmentation: Generally speaking, if the chunkSize is set too large, the meaning of segmentation will be lost. If the chunkSize is set too small, it will have an adverse effect on subsequent vectorization and similarity pair retrieval. After many attempts, it was found that the effect is better when the chunkSize is around 1000. In addition, this application defines an environment variable for chunkSize, which allows dynamic jumps.

[0108] Selection of a suitable embedding model: Different HuggingFace models have different processing capabilities for different types of text and text in different languages. After multiple rounds of testing, this embodiment uses the shibing624 / text2vec-base-chinese model. Similarly, this application also defines environment variables here so that it can be switched according to different PDF types and languages.

[0109] Similarity search top-K (setting of K value): If the K value is set too small, it will cause information loss, resulting in some documents with very similar semantics not being retrieved; if the K value is set too large, some documents with not very similar semantics will be mixed in, affecting the final LLM judgment and causing some token waste. In actual application, this application found that the best effect is when K = 3 to 5. Similarly, environment variables are defined here, which can be adjusted dynamically.

[0110] Use of prompt words: Prompt words play a key role in the application of LLM capabilities. The functions of prompt words include but are not limited to: instructions and telling the model what to do; providing additional information to LLM; what kind of answer this application expects LLM to give, etc. This application tried a variety of prompt word projects and finally obtained the prompt words that worked best in this case. It also supports user-defined prompt words.

[0111] In an optional embodiment of the present application, the user can use the summary and question-and-answer functions. This embodiment uses the Python open source library: Gradio.

[0112] Gradio is an open source project of MIT. It has a very low entry barrier and has the following advantages: it can automatically generate interactive pages; it can be completed with only a small amount of code; it supports the customization of multiple inputs and outputs; and it supports the generation of externally accessible links for sharing.

[0113] The basic functions include: supporting users to upload PDF files; supporting users to preview PDF files on the page; supporting users to enter questions for conversation.

[0114] like Figure 7 As shown, the user clicks the upload button in the file box to upload the local PDF file to the server; after the upload is completed, the document segmentation and vectorization process in the algorithm background is also executed synchronously; after completion, the entire PDF will be displayed on the right side of the page, and it supports translation browsing and marking, just as convenient as using a PDF browsing tool in this application.

[0115] There are two buttons preset on the page, one is summary, and the other corresponds to the description function. These two basic buttons will call the summary chain in the background to generate a summary, such as Figure 8 As shown, list the article's title, author, institution, keywords, links, abstract, conclusion and other information.

[0116] There is also a dialog box at the bottom left of the page for users to enter questions. After the user enters the question, the question-answer chain will be called in the background, and the results will be displayed. There are also some shortcut buttons, such as retry, cancel, clear, etc., to facilitate user operations.

[0117] In this embodiment, a parameter adjustment tab page is also set up, such as Fig. 9 As shown. Considering that most users are PDF article readers and scientific researchers, in order to better utilize the capabilities of LLM, users can customize prompt words (fill in the prompt word input box), customize the K value of recall, customize the current semantics to Chinese or English, etc. By adjusting parameters, users can get a more flexible and effective usage experience.

[0118] The core code of document slicing is as follows:

[0119]

[0120]

[0121] The core code for summary generation is as follows:

[0122]

[0123] The core code for question and answer generation is as follows:

[0124]

[0125]

[0126] The core code of the online conversation tool built based on Gradio is as follows:

[0127]

[0128]

[0129] In summary, the following beneficial effects are specifically achieved in this application:

[0130] This application combines the advantages of the Refine and MapReduce modes, as Figure 6 shown. This application first uses the Map step in the MapReduce mode to calculate and output the abstracts of each segment separately, and then uses the Refine mode for cyclic merging, and finally outputs the abstract of the overall text. Such a solution has the following two advantages:

[0131] Save tokens compared to the MapReduce mode: Taking a pdf text as an example, this application is divided into n segments, with K tokens in each segment, and the abstract of a single segment is summarized into A tokens:

[0132] Using the pure Refine mode, the total number of tokens that need to be consumed is:

[0133] ∑ t = 2K + A + (K + A) * (n - 2) = nK + (n - 1)A

[0134] Using the pure MapReduce mode, the total number of tokens that need to be consumed is:

[0135]

[0136] Adopting the dual-mode combination method of this application, the total number of tokens that need to be consumed is:

[0137] ∑ t = n * (K + A) + 3 * (n - 1) * A = (4n - 3)A + nK

[0138] Generally speaking, in terms of the result of token consumption, the Refine mode < this embodiment < the MapReduce mode, and the solution of this embodiment consumes fewer tokens than the MapReduce mode.

[0139] Better summary effect than Refine mode: The single Refine mode loops through the input segmented documents, outputs the current document and the intermediate summary, continues to construct prompt words for summary to obtain further intermediate summaries, and finally recursively obtains the summary of the entire article. Although this mode is more economical in the use of tokens, there is a lot of context loss, especially if there are too many segments. The more the subsequent segmented documents are looped, the less full the summary content of the intermediate state that saves all the previous information is, and the lower the correlation with the initial content. Therefore, the overall summary effect may not be satisfactory. In the scheme of this embodiment, the Map method is first used to summarize each segment, and the integrity of the current segment information is retained as much as possible. Then the Refine mode is called for loop execution to obtain the final summary. During the implementation process, college students were summoned to test and compare the summary effects of hundreds of papers, verifying that the scheme is better than the single Refine mode in summary effect.

[0140] The key to the question-answering chain is to improve the recall rate of questions and answers. The key to improving the recall rate is to select a suitable embedding model and debug a suitable K value.

[0141] Embedding selection: This application uses two models from OpenAIEmbedding and HuggingFaceEmbedding:

[0142] (1)shibing624 / text2vec-base-chinese;

[0143] (2)sentence-transformers / distiluse-base-multilingual-cased-v1;

[0144] K value: In the process of many debugging, the application finally determined K value = 3;

[0145] This application uses an excerpt from an experiment as a brief explanation. Through different questions and answers (the actual answers are textual discussions or self-written reviews to ensure correctness), the effects of three embedding models are compared when K=3.

[0146] This application can see that sentence-transformers / distiluse-base-multilingual-cased-v1 has the worst effect and cannot be recalled correctly in many cases, while OpenAIEmbedding and shibing624 / text2vec-base-chinese have similar effects. In order to record the user's question and answer data locally, this application finally selected HuggingFace's shibing624 / text2vec-base-chinese model for recall. In the recall rate comparison experiment conducted in this application, the K value was selected from 3 to 20. A total of 10 recall rate test pdf files were selected, with a total of about 850 recall rate experiments. A total of more than 10,000 questions and answers were manually recalled. The recall rate results of the three models are finally obtained as shown in Table 1:

[0147] Table 1 Recall results of different models

[0148]

[0149] Based on the above recall rate results, this application finally selected shibing624 / text2vec-base-chinese as the embedding model of this application. In the subsequent prompt word debugging process, this application tried a total of about 50 prompt words, covering the guidance of LLM from the recall content reference, the sorting of the recalled content, the comprehensiveness and tone of guiding the user's answer, and other aspects for testing. Finally, this application used the prompt word content, and after more than 10,000 manual question and answer tests, the overall answer success rate and response rate exceeded 85%, verifying the availability and practicality of the online PDF dialogue capability.

[0150] In order to obtain accurate answers, this application needs to debug the model and prompt words to optimize the effect. At the same time, in order to facilitate users to debug and use large models by themselves, some parameters are necessary to open and customize. Current solutions often only provide parameter configurations for developers to choose, but ignore the special requirements of users, resulting in insufficient flexibility in use and failure to give full play to the capabilities of LLM and the user's own algorithm capabilities.

[0151] Therefore, in this solution, the following parameters are open for configuration, and some parameters can be configured through the program's environment variables. For example: setting the chunkSize of document segmentation; selecting an appropriate embedding model, and Fig. 9 There are two important parameter settings that can be configured through the chatbot's parameter adjustment page:

[0152] Top K value setting for similarity search: Choosing an appropriate top K value is important for accurate similarity search. Setting it too small will cause information loss, and setting it too large will confuse semantics. Here, the default setting is K=3, which can be customized on the parameter adjustment page.

[0153] Prompt word setting: The use of prompt words is also critical and will greatly affect the accuracy of LLM's answers. Good prompt words will guide LLM to answer well. Users can customize prompt words on the parameter adjustment page.

[0154] Select the appropriate language: The language of the uploaded PDF file (Chinese or English) corresponds to different embedding processing models, and users can switch between different languages.

[0155] The user configuration permission development of the above parameters allows users to flexibly utilize the capabilities of LLM and their own algorithm capabilities. It has strong adaptability in actual operations.

[0156] The embodiment of the present application also provides a pdf file processing system based on the LangChain model. The system includes:

[0157] The upload module is used to obtain the target PDF file uploaded by the user based on the interface provided by Gradio, and to segment the text in the target PDF file into multiple segmented documents;

[0158] Based on the LangChain model, we build a summary processing module and a question-answer processing module. Specifically:

[0159] In the summary processing module, multiple segmented documents are passed into the LLM model for processing and returned to obtain the summary of each segment, and the summaries of all segments are aggregated to obtain the summary of the target PDF file;

[0160] In the question-answering processing module, the embedding of each segmented document is calculated and stored in the vector database; the question input by the user is embedded, and the similar answer set is determined by similarity matching the embeddings corresponding to each segmented document; finally, the question and the similar answer set are assembled through prompt words, and LLM is called again to obtain the target answer to the question.

[0161] The PDF file processing system provided in the embodiment of the present application is used to implement the above-mentioned PDF file processing method. For the specific definition of the PDF file processing system, please refer to the definition of the PDF file processing method in the above text, which will not be repeated here. Each part of the above-mentioned PDF file processing system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the device in the form of hardware, or can be stored in the memory in the device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0162] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A PDF file processing method based on LangChain model, characterized in that: The method comprises: Based on the interface provided by Gradio, the target PDF file uploaded by the user is obtained, and the text in the target PDF file is segmented to form multiple segmented documents; Based on the LangChain model, we build summary scenarios and question-answering scenarios. Specifically: In the summary scenario, multiple segmented documents are passed into the LLM model for processing and returned to obtain the summary of each segment, and the summaries of all segments are aggregated to obtain the summary of the target PDF file; In the question-answering scenario, the embedding of each segmented document is calculated and stored in the vector database. The question input by the user is embedded, and the similar answer set is determined by similarity matching the embeddings corresponding to each segmented document. Finally, the question and the similar answer set are assembled through prompt words, and LLM is called again to obtain the target answer to the question.

2. The pdf file processing method according to claim 1, characterized in that: Get the target PDF file uploaded by the user based on the interface provided by Gradio, including: The PDF is loaded into a document array through PyPDFLoader; each document array contains page content and metadata used to represent the number of pages.

3. The pdf file processing method according to claim 1, characterized in that: Segment the text in the target PDF file to form multiple segmented documents, including: An environment variable is set, wherein the environment variable is used to dynamically set the size of the text block; The text in the target PDF file is segmented sequentially by a recursive character text segmenter until the segmented block reaches a set size.

4. The pdf file processing method according to claim 1, characterized in that: Calculate each segment document to get its own embedding, including: The shibing624 / text2vec-base-chinese model in HuggingFace was selected for text embedding, and the segmented text data was converted into vector representation.

5. The pdf file processing method according to claim 4, characterized in that: Calculate the embeddings of each segmented document and store them in the vector database, including: By calling the from_documents method, the segmented text collection and embedding are stored in the local vector database, and the data file is saved in the form of parquet; wherein the vector database is the Chroma vector database.

6. The pdf file processing method according to claim 1, characterized in that: Multiple segmented documents are passed into the LLM model for processing and returned to obtain the summary of each segment, and the summaries of all segments are aggregated to obtain the summary of the target PDF file, including: Through the Map step in the Map Reduce mode, the summary of each segment is calculated and output separately; then the Refine mode is used for cyclic merging, and finally the summary of the target PDF file is obtained.

7. The pdf file processing method according to claim 1, characterized in that: Embed the questions input by the user, and determine the similar answer set by similarity matching the embeddings corresponding to each segment document, including: Combined with the context of the user's current question and historical conversation, appropriate prompt words are selected to construct a summary, and finally a result vector is produced.

8. The pdf file processing method according to claim 1, characterized in that: The abstract of the target PDF file is obtained, and the abstract also includes the title, author, organization, keywords, links and conclusion of the article.

9. A pdf file processing system based on LangChain model, characterized in that: The system comprises: The upload module is used to obtain the target PDF file uploaded by the user based on the interface provided by Gradio, and to segment the text in the target PDF file into multiple segmented documents; Based on the LangChain model, we build a summary processing module and a question-answer processing module. Specifically: In the summary processing module, multiple segmented documents are passed into the LLM model for processing and returned to obtain the summary of each segment, and the summaries of all segments are aggregated to obtain the summary of the target PDF file; In the question-answering processing module, the embedding of each segmented document is calculated and stored in the vector database; the question input by the user is embedded, and the similar answer set is determined by similarity matching the embeddings corresponding to each segmented document; finally, the question and the similar answer set are assembled through prompt words, and LLM is called again to obtain the target answer to the question.

Images