RAG question and answer traceability positioning method and device based on text coordinate index

Abstract

The invention discloses an RAG question and answer traceability positioning method and device based on a text coordinate index, relates to the technical field of RAG and traceability positioning, and solves the problems that in the prior art, coordinate positioning precision is poor, and traceability in an original text is difficult after deconstruction. The method comprises the following steps of: analyzing and storing a text and corresponding coordinates into a mixed retrieval mode of Elasticsearch, meanwhile, carrying out vectorization storage on text fragments, deconstructing an original text paragraph or keyword in advance, storing the original text paragraph or keyword into mongoDB, carrying out vector similarity matching with Elasticsearch to find the original text when in use, and carrying out uniform format conversion on an input file, so that the extraction precision of the original text coordinates is higher, and the extraction efficiency is improved. The problem of tracing difficulty caused by RAG model text reconstruction is avoided to the greatest extent by utilizing an Elasticsearch technology, so that the most accurate tracing result can be obtained.

Description

Technical Field

[0001] This application relates to the field of RAG and source tracing technology, and in particular to a RAG question-answering source tracing method and apparatus based on text coordinate indexing. Background Technology

[0002] Currently, AI's RAG (Retrieval-Augmented Generation) technology is widely used in text analysis, intelligent question answering and other fields. With the development of business, there are higher requirements for the accuracy and source of AI-generated copy.

[0003] Original text location technology: refers to the technology of extracting text coordinates (such as (x,y) positions) and building a searchable index by parsing the document structure or OCR technology. It is often used in contract analysis, academic literature retrieval, etc.

[0004] In its early stages, PDF relied on manual annotation and basic text parsing, lacking automated coordinate positioning capabilities. With technological breakthroughs, Adobe launched PDF in 1993, which became an ISO standard (ISO 32000-1) in 2008, supporting interactive functions such as hyperlinks and forms. Open-source OCR tools such as Tesseract initially achieved text recognition of scanned documents, but their coordinate accuracy was limited. With ES 5.0+ supporting the ingest-attachment plugin, enterprises began indexing PDF text into ES, but coordinate positioning still required the use of external parsing libraries.

[0005] In existing technologies, location based on PDF parsing libraries involves using PyMuPDF (Python) or PDFBox (Java) to extract text block coordinates and storing them in Elasticsearch for retrieval. However, text coordinate extraction for different file types suffers from the following accuracy issues:

[0006] 1. Differences in PDF coordinate systems: Different PDF generation tools may customize the origin of the coordinate system or rotate the page;

[0007] 2. Inconsistent document formats: Uploaded documents may be in formats such as PDF, WORD, JPG, PPT, etc. Each format has a different parser, resulting in inconsistent parsing results;

[0008] 3. Page size not normalized: The parser does not unify PDF pages of different sizes into a standard range;

[0009] 4. OCR recognition error: Complex layouts (such as multi-column, tables) may be incorrectly segmented by the OCR model.

[0010] The aforementioned coordinate system accuracy issues cause problems with the accuracy of the correspondence between content and layout coordinates when extracting content from different types of documents such as PDF, Word, PPT, and JPG. Secondly, when the RAG model performs question answering using RAG technology, due to the irreversible nature of knowledge deconstruction and reconstruction, when the RAG model references text chunks, it will reorder and reorganize information according to semantics, resulting in the final generated text not being able to correspond effectively with the original text, making it difficult to find the source of the answer in the original text. Summary of the Invention

[0011] The purpose of this application is to overcome the problems in the existing technology of poor coordinate positioning accuracy, which makes it difficult to cover the original text content corresponding to the entry, and the problem that the entry is obtained by deconstructing and reprocessing the original text, making it difficult to trace the source in the original text. The application provides a RAG question-answering source tracing method and device based on text coordinate index.

[0012] Firstly, a RAG question-answering source tracing and localization method based on text coordinate indexing is provided, including:

[0013] Obtain the file to be processed and store it in the first database;

[0014] Convert all files to be processed into PDF files;

[0015] The PDF file is parsed to obtain PDF content, wherein the PDF content includes the original text content, as well as the page number, coordinates, width, and height of the original text content;

[0016] The parsed PDF content is stored in Elasticsearch;

[0017] Upload the files to be processed from the first database to the RAG model and return the parsing results;

[0018] Locable terms are extracted from the parsing results and stored in the second database. The locable terms are then associated with the files to be processed and the PDF content.

[0019] Elasticsearch is used to perform vector similarity matching on locatable terms to find the corresponding original text content;

[0020] The parsing results and the file to be processed are visualized and displayed to the user. The user can jump to the corresponding position in the file to be processed through the locatable entries in the parsing results.

[0021] In some possible implementations, the first database is a MySQL database, and the second database is a MongoDB database.

[0022] In some possible implementations, the files to be processed are uniformly converted into PDF files, including:

[0023] Determine the file type of the file to be processed, wherein the file type includes PDF files and non-PDF files;

[0024] If the file to be processed is not a PDF file, it is converted into a PDF file using a PDF conversion tool and then re-uploaded to the first database.

[0025] In some possible implementations, parsing the PDF file to obtain PDF content includes:

[0026] Determine whether the PDF file is a scanned version;

[0027] If the PDF file is a scanned PDF, the open-source tool minerU is used to perform OCR recognition on the PDF file to extract the PDF content;

[0028] If the PDF file is a standard PDF, the PDFbox open-source toolkit is used to parse the PDF file to obtain the PDF content.

[0029] In some possible implementations, the parsed PDF content is stored in Elasticsearch, including:

[0030] Create an Elasticsearch index;

[0031] The index creates content, x, y, width, height, page, pageWidth, and pageHeight fields. The content field stores the parsed original text content, which is then segmented using an IK word segmenter. The x field stores the X-axis coordinates of the original text content. The y field stores the Y-axis coordinates of the original text content. The width field stores the width of the original text content. The height field stores the height of the original text content. The page field stores the page number of the original text content. The pageWidth field stores the width of the page containing the original text content. The pageHeight field stores the height of the page containing the original text content.

[0032] The PDF content is stored in Elasticsearch based on the matched field information.

[0033] In some possible implementations, after uploading the files to be processed from the first database to the RAG model, a separate knowledge base is established, and the files to be processed are bound to the corresponding separate knowledge base. The parsing template of the files to be processed is obtained, and the preset prompt words corresponding to the parsing module are obtained. The RAG model reads the files to be processed according to the prompt words and parses them to obtain the parsing results. The parsing results include parsing item information, the parsing content in the PDF corresponding to the parsing item, and the page number information and coordinate information of the referenced content. The parsing results are stored in the first database.

[0034] In some possible implementations, the RAG model returns a parsing result in markdown format, and the content after the colon and the non-heading body text in the parsing result are obtained as locatable terms. The locatable terms are stored in a second database, and the locatable terms are associated with the file to be processed and the PDF content.

[0035] In some possible implementations, Elasticsearch is used to perform vector similarity matching on locatable terms to find the corresponding original text content, including:

[0036] Retrieve locatable terms from the second database;

[0037] Retrieve the page number information referenced by the locatable term from the first database;

[0038] The process of matching keywords from Elasticsearch involves directly matching keywords from Elasticsearch using the fuzzy search function `matchPhraseQuery`. If no match is found using the fuzzy search function, Elasticsearch's similarity matching function `MoreLikeThisQueryBuilder` is used to perform similarity matching based on word segmentation, and the ten most similar results are returned.

[0039] The query results from Elasticsearch are matched with the content of the page numbers referenced by the locatable terms. If no match is found, it is recorded as a false query to remove false queries from the query results.

[0040] Expand the values ​​of the y and height fields in the original text corresponding to the query results;

[0041] Obtain locatable terms and their corresponding query results in Elasticsearch, and take the query results that have an intersection with the locatable terms and have a similarity score higher than a preset value as the final query results;

[0042] Returns the locatable term and its corresponding original text data, as well as the page number, coordinates, width, and height of the original text data. The Y-axis and height of the coordinates are the values ​​after being assigned and enlarged.

[0043] In some possible implementations, during the process of visually displaying the parsing results and the files to be processed to the user, the user can identify the locatable terms by hovering the mouse over the different states displayed on the locatable terms in the parsing results, or by using color indicators to identify the locatable terms in the parsing results.

[0044] Secondly, a RAG question-answering source tracing and localization device based on text coordinate indexing is provided, including:

[0045] The file acquisition module is used to acquire the file to be processed and store the file to be processed in the first database;

[0046] The format conversion module is used to convert all files to be processed into PDF files.

[0047] The PDF parsing module is used to parse the PDF file to obtain the PDF content, wherein the PDF content includes the original text content, as well as the page number, coordinates, width, and height of the original text content;

[0048] The ES search module is used to store the parsed PDF content via Elasticsearch.

[0049] The model processing module is used to upload the files to be processed in the first database to the RAG model and return the parsing results;

[0050] The term extraction module is used to extract locatable terms from the parsing results and store the locatable terms in the second database, and associate the locatable terms with the file to be processed and the PDF content;

[0051] The matching module is used to perform vector similarity matching on locatable terms using Elasticsearch to find the corresponding original text content;

[0052] The results display module is used to visually display the parsing results and the file to be processed to the user. The user can jump to the corresponding position in the file to be processed through the locatable entries in the parsing results.

[0053] This application has the following beneficial effects: By uniformly converting the input file format, the extraction accuracy of the original text coordinates is improved. The text and corresponding coordinates are parsed and stored in a hybrid retrieval method of Elasticsearch. At the same time, the text fragments are vectorized and stored. The original text paragraphs or keywords are deconstructed and stored in a MongoDB database in advance. When needed, the vector similarity is matched with Elasticsearch to find the original text. Of course, there may be multiple matched original texts. Then, the intersection of the RAG slices cited by the RAG model in this answer and the matched original text is taken and the similarity is sorted. This avoids the problem of difficult source tracing caused by text reconstruction in the RAG model to the greatest extent, so as to obtain the most accurate source tracing results. Attached Figure Description

[0054] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart of the RAG question-answering source tracing and localization method based on text coordinate index in Embodiment 1 of this application;

[0057] Figure 2 This is a simplified flowchart of the RAG question-answering source tracing and localization method based on text coordinate index in Embodiment 1 of this application;

[0058] Figure 3 This is a structural block diagram of the RAG question-answering tracing and positioning device based on text coordinate index in Embodiment 2 of this application.

[0059] Figure label:

[0060] 100. File Acquisition Module; 200. Format Conversion Module; 300. PDF Parsing Module; 400. ES Search Module; 500. Model Processing Module; 600. Terminology Extraction Module; 700. Matching Module; 800. Result Display Module. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Example 1

[0063] like Figure 1 and Figure 2 As shown in Embodiment 1 of this application, a RAG question-answering source tracing and localization method based on text coordinate indexing is proposed. To improve the accuracy of retrieving and locating the original text, we adopt a hybrid retrieval method that converts the input file into a unified format and stores the text and corresponding coordinates in Elasticsearch. At the same time, the text fragments are vectorized and stored. The original text paragraphs or keywords are deconstructed and stored in MongoDB in advance. When needed, the vector similarity is matched with Elasticsearch to find the original text. Of course, there may be multiple matched original texts. The intersection of the RAG slices referenced by the RAG model in this answer and the matched original texts is taken and then sorted by similarity to obtain the most accurate answer. The method includes:

[0064] S100. Obtain the file to be processed uploaded by the user and store the file to be processed in the first database, wherein the first database is a MySQL database.

[0065] S200. Convert all files to be processed into PDF files. Specifically, the file type of the files to be processed needs to be identified first. In this embodiment, the file type is divided into PDF files and non-PDF files. Determine whether the file to be processed is a PDF file. If it is not a PDF file, use LibreOffice technology to convert the file to the standard PDF format and upload the file to the cloud OSS. Store the file in the OSS link to the MySQL database and save the file ID.

[0066] In this embodiment, to address the coordinate extraction problem caused by non-standard files, which can lead to inaccurate coordinate extraction due to the lack of precise layout during parsing, a dual verification process is performed during file upload using both the file extension and MIME type (e.g., text / plain, application / PDF) to determine if the file is a non-standard PDF. LibreOffice is then used to convert it into a standardized PDF file, improving the accuracy of file parsing and positioning.

[0067] S300. The PDF file is parsed to obtain PDF content, wherein the PDF content includes the original text content, the page number, coordinates, width, and height of the original text content. Specifically, since the text in a scanned PDF cannot be directly recognized, before parsing the PDF file, it is necessary to determine whether the PDF file is a scanned PDF or a regular PDF. If it is a regular PDF, the PDF is parsed using the PDFbox open-source toolkit. If it is a scanned PDF, the OCR parsing is performed using the minerU open-source tool from the Shanghai Artificial Intelligence Laboratory. The parsed PDF content includes the original text content, coordinates, width and height of the content, and page number.

[0068] In this embodiment, the underlying layer uses a combination of multiple parsing technologies such as PyMuPDF, PDFtotext, and OCR engine. Complex PPF and Word documents can be converted into PDFs before parsing, which standardizes the parsing structure and avoids parsing deviations caused by differences in document types. It can intelligently extract complex element information such as images, tables, and formulas from documents, ensuring the accuracy of layout and positioning to the greatest extent.

[0069] S400 stores the parsed PDF content through Elasticsearch, specifically including the following steps:

[0070] S401. Create an Elasticsearch index, using the file ID saved in step S200 as the index name to avoid duplication.

[0071] S402. Index creation includes the content, x, y, width, height, page, pageWidth, and pageHeight fields. The following describes the information stored in each field and its numeric data type:

[0072] The `content` field stores the parsed content information, which is then segmented using the IK word segmenter.

[0073] x field: Stores the X-axis information of the coordinates, single-precision floating-point number type;

[0074] y field: Stores the Y-axis information of the coordinates, single-precision floating-point number type;

[0075] The width field stores the width information of the parsed content and is a single-precision floating-point number.

[0076] The height field stores the height information of the parsed content; it is a single-precision floating-point number.

[0077] The `page` field stores the page number where the parsed content is located; it is an integer.

[0078] The pageWidth field stores the width information of the page containing the parsed content; it is an integer.

[0079] The `pageHeight` field stores the height information of the page containing the parsed content; it is an integer.

[0080] S403. Store the data. The information parsed in step S300 is matched with the field information in step S402 and stored in Elasticsearch.

[0081] The corresponding business data will be parsed asynchronously in the background:

[0082] S500: Upload the files to be processed from the first database to the RAG model and return the parsing results of all parsing items. Specifically, after uploading the files to be processed from the first database that have not been parsed by RAG to the RAG model, establish a separate knowledge base, return the knowledge base ID, and bind the files to be processed to the corresponding separate knowledge base. Obtain the parsing template of the files to be processed. The parsing template is divided into: default interpretation, original text interpretation, and template interpretation. Obtain the preset optimal prompt words corresponding to the parsing module. The RAG model reads the files to be processed according to the prompt words and parses them to obtain the parsing results. The parsing results include parsing item information, the parsing content in the PDF corresponding to the parsing item, and the page number information and coordinate information (chunk) of the referenced content. Persistently store the parsing results in the first database.

[0083] S600. Obtain the parsing results of the RAG model for all parsing items, extract locatable terms from the parsing results, and based on the Markdown format parsing content returned by the RAG model, obtain the content after the colon and the non-heading body text content as locatable terms. Store the locatable terms in a MongoDB database, associating each locatable term with a file ID and parsing item.

[0084] Here is a code example of using text analysis and a RAG model to obtain locatable terms from the interpretation results:

[0085] Message message = new Message();

[0086] message.setRole(Role.USER.getValue());

[0087] String question = "Please parse the text strictly according to the following rules:\n" +

[0088] 1. Identify heading structure: Markdown headings begin with ##, ###, ####, etc. The content of each paragraph under a heading belongs to that heading's scope, until the next heading appears.\n

[0089] "2: Parse the entry content: If a line contains a colon (e.g., name: ABC Company), extract the content after the colon. Recursively process the extracted content: continue checking for nested colons, for example, address: province: Zhejiang, city: Wenzhou → extract "Zhejiang", "Wenzhou". If a line does not contain a colon, add the entire line content as a string element to the result array.\n" +

[0090] 3. For paragraphs without colons, please do not split them; add them directly as complete strings to the result array.

[0091] 4. Skip invalid content: Blank lines, lines containing only whitespace characters, and lines from which no valid information can be extracted should be skipped.\n

[0092] 5. When encountering text that forms a sentence with bolded text, the bolded text and the unbolded text need to be separated.

[0093] "6: The final output should be a pure string array. Please output the string array directly and do not output any other distracting words or symbols\n" +

[0094] Please parse the following markdown content:\n" + text;

[0095] message.setContent(question);

[0096] GenerationResult result = AliAIDialogueUtil.textCallWithMessage(Collections.singletonList(message), "qwen-plus", null);

[0097] String content = result.getOutput().getChoices().get(0).getMessage().getContent();

[0098] List <string>keywords = JSON.parseArray(content, String.class).

[0099] In this embodiment, to address the retrieval performance issue, an asynchronous approach is adopted. When the first database receives a new file, a background task is triggered to perform asynchronous parsing steps S500-S600, and the parsing progress can be viewed through a page or interface. To address the server resource consumption issues associated with text parsing, this embodiment employs distributed microservice processing, separating the task parsing server (for executing steps S100-S400) from the application server (for executing steps S500-S600), and utilizing a high-performance GPU server to maximize resource reliability.

[0100] S700 uses Elasticsearch to perform vector similarity matching on locatable terms to find the corresponding original text content, so that users can search for locatable terms in the MongoDB database on the client side. The parsed item information is rendered on the page. When the mouse hovers over a locatable term, the mouse icon is displayed as a floating icon. Users can click the mouse to jump to the corresponding original text location in the file to be processed.

[0101] Entry location service, including:

[0102] S701. Obtain the location keywords selected by the user.

[0103] S702. Retrieve the page number information of the RAG model parsing results from the MySQL database.

[0104] S703. Matching keywords from Elasticsearch: First, match keywords directly from Elasticsearch using fuzzy matching (matchPhraseQuery). If no fuzzy match is found, use Elasticsearch's MoreLikeThisQueryBuilder to perform similarity matching using word segmentation, returning the ten most similar results. For the results retrieved from Elasticsearch, match them with the page number in step S702. Unmatched query records are considered false queries and are filtered out.

[0105] In a further embodiment, for the optimization of Elasticsearch's MoreLikeThisQuery similarity matching, text vectors are generated using the technique of generating text vector representations (embedding), stored in a vector database, and the cosine similarity between the locating terms and the documents is calculated during the query.

[0106] S704. Coordinate reassignment: 1. For the Y-axis, add half of the original height value to the original value stored in Elasticsearch to make it cover a wider area; 2. For the height value, add height (original value) * 2 to the original value stored in Elasticsearch to make it cover a wider area.

[0107] In this embodiment, in order to expand the click hit area, coordinate optimization technology is used to process the Y coordinate more deeply, move the center point of the click area down to cover the vertical range of the text block, and at the same time increase the hit height to increase the vertical error tolerance space, avoid click failure due to accuracy error, thereby improving the coverage hit rate.

[0108] S705. Simultaneously, obtain RAG slice information from the MySQL database. The specific algorithm is as follows: the intersection of the RAG slice and the top 10 sorted data returned by Elasticsearch, and the result with a similarity score (out of 100) greater than 60 is returned.

[0109] In scenarios involving locating the original text of a knowledge base, Elasticsearch's hybrid retrieval solution can significantly optimize the problem of language expression variation. Its core value lies in bridging the semantic gap through multi-dimensional relevance calculation. For example, the original text "2024 Q4 revenue growth rate reached 25%, mainly contributed by the expansion of new markets in the Asia-Pacific region" might be decomposed by the RAG model into "The company achieved a 25% revenue increase in the fourth quarter of last year (source: doc2_chunk5)". In this case, Elasticsearch will retrieve the original text fragments based on similarity retrieval and select the TOP50 (adjustable) based on similarity ranking to compare with the knowledge base fragments. This approach minimizes the difficulty in matching and locating the original text caused by the text reconstruction of the RAG model.

[0110] S800. The parsing results and the file to be processed are visualized and displayed to the user. The user can jump to the corresponding position in the file to be processed through the locatable entries in the parsing results. The basis for locating the corresponding position in the file to be processed is the value of the page, x, y, height, width, pageWidth, and pageHeight fields. The y and height fields are the values ​​reassigned in step S704.

[0111] Among them, locatable entries can be identified by color, so that users can intuitively know which entries are locatable by the font color or font background color, thus quickly knowing which entries are locatable without having to hover the mouse over the text.

[0112] In this embodiment, to address the accuracy issue of coordinate extraction from PDF and Word documents: Word is a streaming document format and does not store precise layout information. Therefore, LibreOffice is used to convert documents of different formats into a unified PDF, solving the accuracy problem of file parsing and enabling the acquisition of precise coordinate information. MinerU is used to parse PDFs, maintaining a parsing accuracy rate of over 90%. Furthermore, prompt word optimization technology is employed, prioritizing the use of the original text in the RAG model's responses, ensuring that the output parsing results roughly correspond to the original text in over 80% of cases. Finally, ElasticSearch's similarity query technology, multi-level matching strategy, and IK score are used. The word technology enables Elasticsearch's similarity query technology to retrieve the specific document location even after the RAG model reorders semantics and reorganizes information. Combining the RAG model's return results with Elasticsearch's query matching degree optimizes query positioning and makes the positioning information more accurate. By building a knowledge base through RAGFlow, the RAG model can accurately return paragraph-style text slice references when answering, which can narrow down the matching range of the positioning. Finally, by assigning values ​​to the y and height fields to expand the highlighted area by 2-3 times, the visual hit effect of positioning by mouse click is better.

[0113] Example 2

[0114] like Figure 3 As shown, Embodiment 2 of this application relates to a RAG question-answering source tracing and localization device based on text coordinate indexing, comprising:

[0115] The file acquisition module 100 is used to acquire the file to be processed and store the file to be processed in the first database;

[0116] The format conversion module 200 is used to convert all files to be processed into PDF files.

[0117] PDF parsing module 300 is used to parse the PDF file to obtain PDF content, wherein the PDF content includes the original text content, as well as the page number, coordinates, width, and height of the original text content;

[0118] ES Search Module 400 is used to store the parsed PDF content via Elasticsearch;

[0119] The model processing module 500 is used to upload the files to be processed in the first database to the RAG model and return the parsing results;

[0120] The term extraction module 600 is used to extract locatable terms from the parsing results and store the locatable terms in the second database, and associate the locatable terms with the file to be processed and the PDF content;

[0121] The matching module 700 is used to perform vector similarity matching and RAG slice similarity matching of the locatable terms using Elasticsearch to find the corresponding original text content.

[0122] The results display module 800 is used to visually display the parsing results and the file to be processed to the user. The user can jump to the corresponding position in the file to be processed through the locatable entries in the parsing results.

[0123] It should be noted that other specific implementations of the RAG question-and-answer source tracing and localization device based on text coordinate index in this embodiment can be found in the specific implementations of the RAG question-and-answer source tracing and localization method based on text coordinate index described above. To avoid redundancy, they will not be repeated here.

[0124] The above are merely preferred embodiments of this application; however, the scope of protection of this application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in this application, based on the technical solution and its improved concept, should be covered within the scope of protection of this application.< / string>

Claims

1. A RAG question and answer traceability positioning method based on a text coordinate index, characterized in that, The method comprises the following steps: acquiring a to-be-processed file and storing the to-be-processed file into a first database; uniformly converting the to-be-processed file into a PDF file; analyzing the PDF file to obtain PDF content, wherein the PDF content comprises original text content, and the page number, coordinates, width and height of the original text content; storing the analyzed PDF content in Elasticsearch; uploading the to-be-processed file in the first database into a RAG model and returning an analysis result; extracting a locatable term from the analysis result and storing the locatable term into a second database, and associating the locatable term with the to-be-processed file and the PDF content; performing vector similarity matching on the locatable term by using Elasticsearch to find corresponding original text content; visualizing the analysis result and presenting the analysis result to a user, wherein the user can jump to a corresponding position in the to-be-processed file through the locatable term in the analysis result.

2. The text coordinate index based RAG question answer provenance localization method of claim 1, wherein, The first database is a MySQL database, and the second database is a MongoDB database.

3. The text coordinate index based RAG question answer provenance localization method of claim 2, wherein, The step of uniformly converting the to-be-processed file into a PDF file comprises the following steps: judging the file type of the to-be-processed file, wherein the file type comprises a PDF file and a non-PDF file; in response to the to-be-processed file being a non-PDF file, converting the to-be-processed file into a PDF file by using a PDF conversion tool, and re-uploading the to-be-processed file into the first database.

4. The text coordinate index based RAG question answer provenance localization method of claim 1, wherein, The step of analyzing the PDF file to obtain PDF content comprises the following steps: judging whether the PDF file is a scanned version; in response to the PDF file being a scanned version, performing OCR identification on the PDF file by using an open-source tool minerU to analyze the PDF content; in response to the PDF file being a normal version, analyzing the PDF file by using a PDFbox open-source tool package to obtain the PDF content.

5. The text coordinate index based RAG QAO method according to any one of claims 1-4, wherein, The step of storing the analyzed PDF content in Elasticsearch comprises the following steps: creating an Elasticsearch index; indexing a content field, an x field, a y field, a width field, a height field, a page field, a pageWidth field and a pageHeight field, wherein the content field is used to store the analyzed original text content and is segmented by using an IK segmenter, the x field is used to store X-axis information of original text content coordinates, the y field is used to store Y-axis information of original text content coordinates, the width field is used to store width information of original text content, the height field is used to store height information of original text content, the page field is used to store page number information of original text content, the pageWidth field is used to store width information of a page where original text content is located, and the pageHeight field is used to store height information of the page where original text content is located; storing the PDF content in Elasticsearch according to field information.

6. The text coordinate index based RAG question answer provenance localization method of claim 5, wherein, After uploading the to-be-processed file in the first database to the RAG model, a separate knowledge base is established, the to-be-processed file is bound to the corresponding separate knowledge base, an analysis template of the to-be-processed file is obtained, a preset prompt word corresponding to an analysis module is obtained, the RAG model reads the to-be-processed file according to the prompt word and analyzes the to-be-processed file to obtain an analysis result, the analysis result includes analysis item information, analysis content in a PDF corresponding to the analysis item, and page number information and coordinate information of referenced content, and the analysis result is stored in the first database.

7. The text coordinate index based RAG question answer provenance localization method of claim 6, wherein, The RAG model returns the analysis result in markdown format, obtains content after a colon and non-title text content in the analysis result as a locatable entry, stores the locatable entry in a second database, and associates the locatable entry with the to-be-processed file and PDF content.

8. The text coordinate index based RAG question answer provenance localization method of claim 6, wherein, Also includes providing entry positioning services for users, in particular: Obtain the selected positioning entry of the user; Obtain the page number information referenced by the positioning entry from the first database; Match the positioning entry from Elasticsearch, specifically, match the positioning entry directly from Elasticsearch through fuzzy search matchmatchPhraseQuery, if the fuzzy search does not match, use the similar matching of ElasticsearchMoreLikeThisQueryBuilder to perform similar matching using word segmentation, and return the top ten data with the highest similarity; Match the query results from Elasticsearch with the content in the page number referenced by the positioning entry, if the matching is not successful, record it as a false query to remove false queries from the query results; Assign and expand the y field and height field of the original text content corresponding to the query results; Obtain the RAG slice corresponding to the positioning entry and the query results in the corresponding Elasticsearch, and take the query results that have an intersection with the RAG slice and a similarity higher than a preset score as the final query results; Return the positioning entry, the corresponding original text data, and the page number, coordinates, width, and height occupied by the original text data, wherein the Y-axis of the coordinates and the height are the values after assignment and expansion.

9. The text coordinate index based RAG question answer provenance localization method of claim 6, wherein, In the process of visually displaying the analysis result and the to-be-processed file to the user, the user can identify the locatable entries in the analysis result by hovering the mouse over different states displayed by the locatable entries or identifying the locatable entries in the analysis result by color identification.

10. A RAG question answering provenance locating apparatus based on text coordinate indexing, characterized in that, It includes: A file acquisition module for acquiring to-be-processed files and storing the to-be-processed files in a first database; A format conversion module for uniformly converting to-be-processed files into PDF files; A PDF analysis module for analyzing the PDF files to obtain PDF content, wherein the PDF content includes original text content, and the page number, coordinates, width, and height occupied by the original text content; An ES search module for storing the analyzed PDF content through Elasticsearch; A model processing module configured to upload the to-be-processed file in the first database to a RAG model and return a parsing result; A term extraction module configured to extract a locatable term from the parsing result and store the locatable term in a second database, and associate the locatable term with the to-be-processed file and the PDF content; A matching module configured to perform vector similarity matching on the locatable term and RAG slice similarity matching on the locatable term corresponding to the RAG using Elasticsearch to find corresponding original content; A result display module configured to visually display the parsing result and the to-be-processed file to a user, wherein the user can jump to a corresponding position in the to-be-processed file through the locatable term in the parsing result.

Images