Periodical paper tracing method and device based on RAG technology and computer equipment

Abstract

The invention discloses a journal paper traceability method and device based on an RAG technology and computer equipment, relates to the technical field of paper indexing, and aims at enhancing the depth and breadth of retrieval based on the RAG technology while improving the traceability accuracy and efficiency by combining vector and keyword retrieval and capturing association and contextual relationships among literatures. The method comprises the steps of obtaining paragraph characters to be traced, and performing vectorization processing on the paragraph characters to obtain character vectors; performing vectorization retrieval on the character vectors through a vector index database to obtain a plurality of first related document blocks, the vector index database being constructed based on an RAG technology; performing keyword retrieval on the paragraph characters to obtain a plurality of second related document blocks; according to the document relevancy between each first related document block and the paragraph text and between each second related document block and the paragraph text, document tracing operation is executed, and a target original document is obtained and output.

Description

Technical Field

[0001] The present application relates to the field of paper indexing technology, and in particular to a journal paper source tracing method, device and computer equipment based on RAG technology. Background Art

[0002] In today's era of high information technology and knowledge explosion, the number and complexity of academic journal papers, as the core carrier of scientific research innovation and knowledge inheritance, are growing exponentially. With the rapid development of Internet technology, especially the widespread application of big data and cloud computing technology, a large number of academic papers have been digitized and stored in various online databases, such as China National Knowledge Infrastructure, which greatly facilitates the acquisition and sharing of academic resources. However, this huge information resource library also brings unprecedented challenges: how to efficiently and accurately trace and understand academic content in massive data has become a key issue that needs to be solved in the academic community.

[0003] In related technologies, academic literature tracing mainly relies on keyword-based search strategies. Users initiate queries by entering specific keywords or phrases, and the system matches and sorts the keywords according to their frequency of occurrence in the title, abstract or full text of the document, and returns the search results.

[0004] In the process of implementing the relevant technology, the applicant found that the relevant technology has at least the following problems:

[0005] The selection of keywords is highly dependent on the user's prior knowledge and understanding of the target field. For complex or novel academic concepts, it is often difficult for users to accurately extract all relevant keywords, resulting in incomplete and missing tracing results. Moreover, keyword retrieval cannot deeply understand the deep meaning and logical relationship of the document content. The relevance and accuracy of the tracing results are not high, and it is difficult to effectively reveal the complex citation network and knowledge flow path between documents, and it is impossible to provide academic researchers with efficient and reliable academic support. Summary of the invention

[0006] In view of this, the present application provides a journal paper tracing method, device and computer equipment based on RAG technology. The main purpose is to solve the problems that the current tracing results are incomplete and missing, and keyword retrieval cannot deeply understand the deep meaning and logical relationship of the document content. The relevance and accuracy of the tracing results are not high, and it is difficult to effectively reveal the complex citation network and knowledge flow path between documents, and it is impossible to provide academic researchers with efficient and reliable academic support.

[0007] According to the first aspect of the present application, a journal paper tracing method based on RAG technology is provided, the method comprising:

[0008] Obtaining paragraph text to be traced, and performing vectorization processing on the paragraph text to obtain a text vector;

[0009] Performing vectorized retrieval on the text vectors through a vector index library to obtain a plurality of first relevant document blocks, wherein the vector index library is constructed based on RAG technology;

[0010] Perform keyword search on the paragraph text to obtain multiple second related document blocks;

[0011] Referring to the document relevance between each of the first relevant document blocks and each of the second relevant document blocks and the paragraph text, a document tracing operation is performed to obtain and output the target original document.

[0012] Optionally, the obtaining of paragraph text to be traced and vectorization processing of the paragraph text to obtain a text vector includes:

[0013] In response to a journal article traceability request, obtaining uploaded text content as the paragraph file to be traced, and performing a preprocessing operation on the paragraph text;

[0014] A pre-trained deep learning model is obtained, and the pre-processed paragraph text is input into the deep learning model to obtain the text vector.

[0015] Optionally, when constructing the vector index library, the method includes:

[0016] Obtaining document collection requirements, and collecting multiple original documents according to the document collection requirements;

[0017] Performing data cleaning operations on the multiple original documents, and using NLP technology to perform text standardization processing on the cleaned multiple original documents;

[0018] Performing structural processing on each of the processed original documents, and generating a plurality of document blocks for each of the original documents;

[0019] Inputting multiple document blocks of each of the original documents into a pre-trained deep learning model for vector conversion to obtain multiple document vectors, wherein the deep learning model is constructed using a neural network model indicated by the RAG technology;

[0020] The multiple document vectors are sorted to obtain the vector index library.

[0021] Optionally, the step of performing structural processing on each of the processed original documents and generating a plurality of document blocks for each of the original documents includes:

[0022] For each of the original documents, identifying structural elements in the original document, and determining a plurality of document paragraphs included in the original document according to the identified structural elements, wherein the structural elements include one or more of a chapter title, a paragraph start, and a paragraph end;

[0023] Generate metadata information for each of the document paragraphs according to the structural elements in each of the document paragraphs, the metadata information including one or more of paragraph information, chapter information, title information, and author information of the corresponding document paragraph;

[0024] Traversing each of the document paragraphs, and splitting each of the document paragraphs into a plurality of independent sentences according to sentence boundaries detected in each of the document paragraphs;

[0025] A document block is constructed for each of the multiple sentences, and metadata information of the document paragraph to which each sentence belongs is stored in the document block of the sentence, thereby obtaining multiple document blocks of the original document.

[0026] Optionally, referring to the document relevance between each of the first relevant document blocks and each of the second relevant document blocks and the paragraph text, performing a document tracing operation to obtain and output a target original document includes:

[0027] Calculating the document relevance between each of the first relevant document blocks and each of the second relevant document blocks and the paragraph text, and querying whether there is at least one candidate relevant document block whose document relevance is greater than or equal to a relevance threshold among the plurality of first relevant document blocks and the plurality of second relevant document blocks;

[0028] When the query determines that the at least one candidate related document block exists, deduplication processing is performed on the at least one candidate related document block, the processed at least one candidate related document block is sorted in descending order of document relevance to obtain a first sorting result, and a first preset number of candidate related document blocks ranked first in the first sorting result are extracted, target metadata information is extracted from the first preset number of candidate related document blocks, and the document traced back through the target metadata information is used as the target original document and outputted;

[0029] When the query determines that at least one candidate related document block does not exist, NLP technology is used to analyze the entities and relationships of the paragraph text to obtain the analysis result of the paragraph text, and a graph search is performed on the analysis result through a graph database to obtain a retrieval result, and the document traced back based on the retrieval result, the multiple first related document blocks and the multiple second related document blocks is used as the target original document and output.

[0030] Optionally, when constructing the atlas database, the method includes:

[0031] Acquire multiple document blocks obtained by splitting multiple original documents, and identify entity information, attribute information, and relationship information of each document block;

[0032] Cleaning up duplicate and contradictory contents in the entity information, attribute information and relationship information of each document block, and generating triple data for each document block using the cleaned entity information, attribute information and relationship information to obtain a plurality of triple data of the plurality of document blocks;

[0033] The multiple triplet data are sorted to obtain the atlas database.

[0034] Optionally, performing a graph search on the analysis result through a graph database to obtain a search result, and outputting a document traced back based on the search result, the plurality of first related document blocks, and the plurality of second related document blocks as the target original document, including:

[0035] Performing a graph search on the analysis result in the graph database, and taking the retrieved at least one target triple data as the search result;

[0036] Respectively query the document blocks corresponding to each of the target triple data to obtain at least one third related document block, and calculate the document relevance between each of the third related document blocks and the paragraph text;

[0037] performing deduplication processing on the plurality of first related document blocks, the plurality of second related document blocks, and the at least one third related document block, and sorting the processed plurality of first related document blocks, the plurality of second related document blocks, and the at least one third related document block in descending order of document relevance to obtain a second sorting result;

[0038] A second preset number of designated related document blocks ranked first in the second sorting result are extracted, designated metadata information is extracted from the second preset number of designated related document blocks, and the document traced back through the designated metadata information is used as the target original document and output.

[0039] According to the second aspect of the present application, a journal article tracing device based on RAG technology is provided, the device comprising:

[0040] A processing module, used for obtaining paragraph text to be traced, and performing vectorization processing on the paragraph text to obtain a text vector;

[0041] A vector retrieval module, used to perform vectorized retrieval on the text vector through a vector index library to obtain a plurality of first relevant document blocks, wherein the vector index library is constructed based on the RAG technology;

[0042] A keyword search module, used for performing keyword search on the paragraph text to obtain a plurality of second related document blocks;

[0043] The output module is used to refer to the document relevance between each of the first related document blocks and each of the second related document blocks and the paragraph text, perform a document tracing operation, obtain a target original document and output it.

[0044] Optionally, the processing module is used to respond to a journal paper tracing request, obtain the uploaded text content as the paragraph file to be traced, and perform preprocessing operations on the paragraph text; obtain a pre-trained deep learning model, input the preprocessed paragraph text into the deep learning model, and obtain the text vector.

[0045] Optionally, when constructing the vector index library, a document collection requirement is obtained, and multiple original documents are collected according to the document collection requirement; data cleaning operations are performed on the multiple original documents, and text standardization processing is performed on the cleaned multiple original documents using NLP technology; structured processing is performed on each of the processed original documents, and multiple document blocks are generated for each of the original documents; the multiple document blocks of each of the original documents are input into a pre-trained deep learning model for vector conversion to obtain multiple document vectors, and the deep learning model is constructed using a neural network model indicated by the RAG technology; the multiple document vectors are sorted to obtain the vector index library.

[0046] Optionally, when constructing the vector index library, for each of the original documents, structural elements in the original document are identified, and based on the identified structural elements, multiple document paragraphs included in the original document are determined, wherein the structural elements include one or more of a chapter title, a paragraph start, and a paragraph end; metadata information is generated for each of the document paragraphs based on the structural elements in each of the document paragraphs, and the metadata information includes one or more of paragraph information, chapter information, title information, and author information of the corresponding document paragraph; each of the document paragraphs is traversed, and based on sentence boundaries detected in each of the document paragraphs, each of the document paragraphs is split into multiple independent sentences; a document block is constructed for each of the multiple sentences, and the metadata information of the document paragraph to which each of the sentences belongs is stored in the document block of the sentence, so as to obtain multiple document blocks of the original document.

[0047] Optionally, the output module is used to calculate the document relevance between each of the first relevant document blocks and each of the second relevant document blocks and the paragraph text, and query whether there is at least one candidate relevant document block whose document relevance is greater than or equal to a relevance threshold among the multiple first relevant document blocks and the multiple second relevant document blocks; when the query determines that the at least one candidate relevant document block exists, the at least one candidate relevant document block is deduplicated, the at least one processed candidate relevant document block is sorted in order of document relevance from high to low to obtain a first sorting result, and a first preset number of candidate relevant document blocks ranked first in the first sorting result are extracted, target metadata information is extracted from the first preset number of candidate relevant document blocks, and the document traced back through the target metadata information is used as the target original document and outputted; when the query determines that the at least one candidate relevant document block does not exist, the NLP technology is used to analyze the entities and relationships of the paragraph text to obtain the analysis result of the paragraph text, and the graph search is performed on the analysis result through the graph database to obtain the retrieval result, and the document traced back based on the retrieval result, the multiple first relevant document blocks and the multiple second relevant document blocks is used as the target original document and outputted.

[0048] Optionally, when constructing the graph database, multiple document blocks obtained by splitting multiple original documents are obtained, and the entity information, attribute information and relationship information of each document block are identified; the duplicate content and contradictory content in the entity information, attribute information and relationship information of each document block are cleaned up, and the cleaned entity information, attribute information and relationship information are used to generate triple data for each document block to obtain multiple triple data of the multiple document blocks; the multiple triple data are sorted to obtain the graph database.

[0049] Optionally, the output module is used to perform graph retrieval on the analysis result in the graph database, and use the retrieved at least one target triple data as the retrieval result; query the document block corresponding to each of the target triple data respectively to obtain at least one third related document block, and calculate the document relevance between each of the third related document blocks and the paragraph text; deduplicate the multiple first related document blocks, the multiple second related document blocks and the at least one third related document block, and sort the processed multiple first related document blocks, the multiple second related document blocks and the at least one third related document block in order of document relevance from high to low to obtain a second sorting result; extract a second preset number of specified related document blocks ranked first in the second sorting result, extract specified metadata information from the second preset number of specified related document blocks, and output the document traced back through the specified metadata information as the target original document.

[0050] According to a third aspect of the present application, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0051] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0052] Through the above technical scheme, the present application provides a journal paper tracing method, device and computer equipment based on RAG technology. The present application obtains the paragraph text to be traced, vectorizes the paragraph text, obtains the text vector, and vectorizes the text vector through the vector index library to obtain multiple first related document blocks. The vector index library is constructed based on RAG technology, and keyword search is performed on the paragraph text to obtain multiple second related document blocks. Referring to the document relevance between each first related document block and each second related document block and the paragraph text, the document tracing operation is performed to obtain the target original document and output it. By combining the multimodal retrieval strategy of vectorized retrieval and keyword retrieval, the advantages of different retrieval methods are fully utilized to capture the complex associations and contextual relationships between documents. While improving the accuracy and efficiency of document tracing, a vector index library is constructed based on RAG technology, which further enhances the depth and breadth of the retrieval, ensures the depth and comprehensiveness of the retrieval results, and provides strong technical support for document tracing.

[0053] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0055] Figure 1 A schematic diagram of a journal paper source tracing method based on RAG technology provided in an embodiment of the present application is shown;

[0056] Figure 2A A schematic diagram of another method for tracing the origin of a journal article based on RAG technology provided in an embodiment of the present application is shown;

[0057] Figure 2B A schematic diagram of a construction process of a vector index library provided in an embodiment of the present application is shown;

[0058] Figure 2C A schematic diagram of a construction process of a graph database provided in an embodiment of the present application is shown;

[0059] Figure 2D A schematic diagram of a RAG model design architecture provided in an embodiment of the present application is shown;

[0060] Figure 2E A schematic diagram of a process flow of another journal paper tracing method based on RAG technology provided in an embodiment of the present application is shown;

[0061] Figure 3 A schematic diagram of the structure of a journal paper tracing device based on RAG technology provided in an embodiment of the present application is shown;

[0062] Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0063] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0064] The present application embodiment provides a journal paper tracing method based on RAG technology, such as Figure 1 As shown, the method includes:

[0065] 101. Obtain the paragraph text to be traced, and vectorize the paragraph text to obtain a text vector.

[0066] The embodiment of the present application can be applied to a document search platform. When a user has a need to trace the source of a journal article, the document search platform will be provided with paragraph text to be traced, such as entering the paragraph text in an input box provided by the document search platform. After obtaining the paragraph text entered by the user, the document search platform will use the paragraph text entered by the user as the paragraph text to be traced.

[0067] In the field of document processing, especially in paper tracing and retrieval tasks, it is crucial to effectively convert document content (such as paper abstracts, full texts or specific paragraphs) into vector representations. These vector representations need to be able to capture the semantic information of the document in order to play a role in subsequent similarity comparisons, clustering or retrieval tasks. Therefore, in an embodiment of the present application, after obtaining the paragraph text, the document search platform will vectorize the paragraph text and obtain a text vector to capture the semantic features of the paragraph text and provide a basis for subsequent vectorized retrieval.

[0068] 102. Perform vectorized retrieval on the text vector through a vector index library to obtain multiple first relevant document blocks.

[0069] In order to achieve efficient and accurate vector retrieval, the embodiment of the present application proposes a method for constructing a vector index library based on RAG (Retriever-Augmented Generation) technology. The method aims to construct a vector index library rich in semantic information by fine-grained processing of structured paper documents and converting document content into vector representations in combination with an embedding model. The vector index library not only supports fast vectorized retrieval, but also makes full use of the advantages of RAG technology to improve the accuracy and efficiency of retrieval.

[0070] The vector index library includes multiple document vectors, each of which is obtained by vectorizing the document block, ensuring that the document block and the paragraph text are in the same semantic space, which is convenient for similarity calculation. Among them, the document block is generated by splitting the structured paper according to sentences and retaining its paragraphs, chapters, titles, authors and other metadata information. After generating the document block, it is necessary to use an embedding model to convert the generated document block into vector data, and the metadata information related to the document block is saved in the vector index library. These metadata information plays an important role in subsequent retrieval and relevance evaluation. It should be noted that the document vector can be a fixed length, or a fixed length vector obtained by pooling and other operations after dynamic length, and this application does not specifically limit this. In addition, efficient data structures (such as KD trees, ball trees, approximate nearest neighbor algorithms, etc.) can be used to organize and store document vectors to improve retrieval speed.

[0071] In this way, the vector index library is used to perform vectorized retrieval on the text vector, find the document vector with a high similarity to the text vector, and use the document block corresponding to the document vector as the first relevant document block. Specifically, when performing vector retrieval, for each document vector in the vector index library, it is necessary to calculate its similarity with the text vector. Among them, the similarity measurement methods used include cosine similarity, Euclidean distance, etc., and according to the specific needs of the task, a suitable similarity measurement method can be selected to find the document vector with a high similarity to the text vector, and use the document block corresponding to the document vector as the first relevant document block. In the actual application process, in order to control the number and quality of the first relevant document block returned, a similarity threshold can be set. Only when the similarity between the document vector and the text vector exceeds this threshold, the document block corresponding to the document vector is used as the first relevant document block.

[0072] 103. Perform keyword search on the paragraph text to obtain multiple second related document blocks.

[0073] In the embodiment of the present application, in addition to vectorized retrieval, a traditional keyword retrieval method is also used to extract keywords from paragraph text, perform keyword matching retrieval in the document database, and obtain document blocks related to the keywords as the second related document blocks.

[0074] Specifically, the paragraph and chapter structure data corresponding to the sentences can be split out in the document block for keyword index retrieval, and keyword search can be performed through the keyword inverted index library to obtain multiple second related document blocks. In this way, by combining keyword retrieval, it is possible to supplement the documents that may be missed by vectorized retrieval but directly contain keywords; moreover, by structuring the document block, when displaying the results, not only the sentences containing keywords are displayed, but also the paragraph and chapter information to which these sentences belong, which helps users understand the contextual position of keywords in the document.

[0075] 104. Referring to the document relevance between each first related document block and each second related document block and the paragraph text, a document tracing operation is performed to obtain and output the target original document.

[0076] In an embodiment of the present application, after obtaining multiple first-related document blocks and multiple second-related document blocks, the document search platform will refer to the document relevance between each first-related document block and each second-related document block and the paragraph text, perform a document tracing operation, obtain the target original document and output it.

[0077] First, the document search platform calculates the document relevance between each first relevant document block and the second relevant document block and the paragraph text to be traced. This can be achieved by calculating the semantic similarity, keyword overlap and other indicators between the document block and the paragraph text. Then, the document is sorted and screened according to the document relevance, and the most relevant document block is selected as the candidate document block. Finally, through further analysis and comparison (such as content consistency check, author information comparison, etc.), the final document block is determined, and the target original document is traced back through the metadata information of the document block, and it is output to the user.

[0078] It should be noted that a relevance threshold can be set in the document search platform, and the document relevance of each document block calculated is compared with the relevance threshold. If the document relevance of each document block is less than the relevance threshold, the user's paragraph text can be further parsed through NLP technology, the entities and relationships in the paragraph text can be identified, and the graph database can be searched, and the relevant document blocks can be retrieved again. Finally, the Rerank model is used to re-sort the paragraph text and the results according to the semantic relevance to obtain the top few document blocks for document tracing, and the traced documents are used as the target original documents and output. In this way, through the constructed knowledge graph and multi-level enhancement mechanism, not only the depth of retrieval is increased, but also a traceable academic evidence chain is provided, which enhances the user's trust in the retrieval results.

[0079] The method provided by the embodiment of the present application obtains the paragraph text to be traced, performs vectorization processing on the paragraph text to obtain a text vector, performs vectorized retrieval on the text vector through a vector index library to obtain multiple first relevant document blocks. The vector index library is constructed based on the RAG technology. Perform keyword retrieval on the paragraph text to obtain multiple second relevant document blocks. Refer to the document relevance between each first relevant document block and each second relevant document block and the paragraph text, perform document tracing operations, obtain the target original document and output it. By combining the multi-modal retrieval strategy of vectorized retrieval and keyword retrieval, make full use of the advantages of different retrieval methods, capture the complex associations and context relationships between documents, while improving the accuracy and efficiency of document tracing, construct a vector index library based on the RAG technology, further enhance the depth and breadth of retrieval, ensure the depth and comprehensiveness of retrieval results, and provide strong technical support for document tracing.

[0080] Further, as a refinement and extension of the specific implementation manner of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, the embodiment of the present application provides another method for tracing journal papers based on the RAG technology, as Figure 2A shown, this method includes:

[0081] 201. Obtain the paragraph text to be traced, and perform vectorization processing on the paragraph text to obtain a text vector.

[0082] The embodiment of the present application can be applied to a document search platform. When a user has a need to trace journal papers, the user will provide the paragraph text to be traced to the document search platform, such as entering the paragraph text in the input box provided by the document search platform. Alternatively, the document search platform can also provide a file upload function for the user, receive the paragraph file uploaded by the user, and use the paragraph content included in the file as the paragraph text to be traced in the journal paper. Among them, the document search platform can restrict the format of the file uploaded by the user to ensure that the uploaded file format supports text reading.

[0083] When the user confirms the provided paragraph text, the document search platform will default to receiving the user's journal paper tracing request. In response to the journal paper tracing request, the document search platform will obtain the uploaded text content as the paragraph file to be traced, and perform preprocessing operations on the paragraph text. The preprocessing operations can include text cleaning, word segmentation, stop word removal, stemming or lemmatization, etc. Among them, the text cleaning operation is used to remove non-text elements in the paragraph text, such as HTML tags, special characters, irrelevant symbols, etc.; word segmentation is used to segment the text according to language rules for subsequent processing; stop word removal is used to remove common words that are meaningless for analysis, such as "de", "shi", "zai", etc.; stemming or lemmatization is used to convert the vocabulary into its basic form to reduce the impact of vocabulary morphological changes on analysis.

[0084] In the field of document processing, especially in paper tracing and retrieval tasks, it is crucial to effectively convert document content (such as paper abstracts, full texts or specific paragraphs) into vector representations. These vector representations need to be able to capture the semantic information of the document in order to play a role in subsequent similarity comparisons, clustering or retrieval tasks. Therefore, in an embodiment of the present application, after completing the preprocessing operation, the document search platform will obtain a pre-trained deep learning model, input the preprocessed paragraph text into the deep learning model, obtain a text vector, and obtain a text vector to capture the semantic features of the paragraph text, providing a basis for subsequent vectorized retrieval.

[0085] Specifically, the document retrieval platform will use pre-trained deep learning models (such as BERT, RoBERTa, etc.) to convert paragraph text into vectors to capture semantic information. Among them, the appropriate pre-trained model can be selected according to the specific requirements of the task and resource constraints. BERT and RoBERTa can usually be selected for vector conversion, and they perform well in multiple natural language processing tasks. RoBERTa has been further optimized on the basis of BERT, such as longer training time, larger batch size, removal of NSP (Next Sentence Prediction) tasks, etc., to obtain better performance.

[0086] 202. Perform vectorized retrieval on the text vector through a vector index library to obtain multiple first related document blocks.

[0087] In order to achieve efficient and accurate vector retrieval, the embodiment of the present application proposes a method for constructing a vector index library based on RAG (Retriever-Augmented Generation) technology. The method aims to construct a vector index library rich in semantic information by fine-grained processing of structured paper documents and converting document content into vector representations in combination with an embedding model. The vector index library not only supports fast vectorized retrieval, but also makes full use of the advantages of RAG technology to improve the accuracy and efficiency of retrieval.

[0088] The vector index library includes multiple document vectors. Each document vector is obtained by converting the document block into a vector, ensuring that the document block and the paragraph text are in the same semantic space to facilitate similarity calculation. The following describes the generation process of the document block, which includes the following three steps:

[0089] Step 1: Obtain document collection requirements and collect multiple original documents according to the document collection requirements.

[0090] The specific purpose, theme scope, time span, required document types (such as academic papers, experimental reports, patent documents, technical white papers, etc.) of the collected documents, as well as possible specific sources or databases can be clarified first, so as to determine the document collection requirements; then, according to the document collection requirements, obtain eligible documents as the original documents. It should be noted that the original format and metadata information of the original documents should be maintained to facilitate subsequent traceability operations.

[0091] Step 2: Perform data cleaning operations on multiple original documents, and perform text standardization processing on the multiple cleaned original documents using NLP technology.

[0092] After obtaining multiple original documents, first perform data cleaning operations on the multiple original documents to remove irrelevant information, correct errors, and ensure data quality. Specifically, non-text content such as advertisements, headers, footers, copyright notices, watermarks, etc. in the original documents can be removed, duplicate paragraphs or documents can be deleted to reduce redundancy; the documents can also be converted to a unified format (such as plain text or HTML) for subsequent processing, and format errors in the text, such as inconsistent indents, line breaks, spaces, etc., can be corrected; noise data such as garbled characters, special characters, emojis, etc. can also be removed, and non-text data blocks such as pictures and tables can be identified and deleted; in addition, if the original documents contain multiple languages, the main analysis language can be determined, and whether to remove the text in other languages can be considered.

[0093] After completing the data cleaning operations on the original documents, then perform text standardization processing on the multiple cleaned original documents using NLP technology to convert the cleaned text into a more standardized and easier-to-analyze form. Among them, stemming, stop word removal, and lemmatization can be performed on the original documents based on NLP technology. Specifically, an appropriate tokenization tool needs to be used first to split the text into words or phrases, create a stop word list that contains common words that are not very helpful for text analysis, such as "of", "is", "in", etc., and remove these stop words from the text to reduce data sparsity and improve processing efficiency; then, the words are reduced to their basic form (stem) to reduce the lexical diversity caused by lexical morphological changes (such as tenses, singular and plural); then, based on the grammatical and semantic information of the words, the words are restored to the actual word form in the dictionary, retaining the original meaning of the words.

[0094] In the actual application process, when performing text standardization processing, all words in the text can also be converted to lowercase (or uppercase) to ensure that case does not affect subsequent analysis.

[0095] Step 3: Structurally process each processed original document and generate multiple document blocks for each original document.

[0096] For each original document, it is necessary to first identify the structural elements in the original document, and determine the multiple document paragraphs included in the original document based on the identified structural elements. Among them, the structural elements include one or more of the chapter title, the beginning of a paragraph, and the end of a paragraph. The NLP technology can be used to identify the above structural elements, and the hierarchical structure of the original document can be determined based on the identified structural elements, so as to further determine the multiple document paragraphs included in the original document. For example, the original document is first divided into multiple chapters based on the identified chapter titles, and then each chapter is divided into multiple paragraphs according to the markers of the beginning and end of the paragraph, thereby obtaining multiple document paragraphs.

[0097] In order to realize document traceability, while determining the document paragraphs, metadata information can also be generated for each document paragraph according to the structural elements in each document paragraph, wherein the metadata information includes one or more of the paragraph information, chapter information, title information and author information of the corresponding document paragraph.

[0098] Next, each document paragraph is traversed, and each document paragraph is split into multiple independent sentences according to the sentence boundaries detected in each document paragraph. Specifically, NLP tools can be used to detect sentence boundaries, accurately identify language structures and punctuation marks, and achieve sentence splitting.

[0099] Finally, a document block is constructed for each of the multiple sentences, and the metadata information of the document paragraph to which each sentence belongs is stored in the document block of the sentence, thereby obtaining multiple document blocks of the original document, so that the document block contains not only the sentence text, but also metadata information pointing to the paragraph, chapter, and higher-level structure (such as the document itself) to which it belongs. It should be noted that in the document block, metadata information can be stored in the form of a dictionary, a JSON object, etc., to ensure that for each sentence, the paragraph, chapter, and other relevant information of the document to which it belongs can be easily accessed.

[0100] The above is the process of generating document blocks. The following describes the process of building a vector index library:

[0101] After generating the document blocks, it is necessary to input multiple document blocks of each original document into a pre-trained deep learning model for vector conversion to obtain multiple document vectors. The deep learning model is constructed using a neural network model indicated by RAG technology. Specifically, BERT and RoBERTa can be selected for vector conversion to convert each document block into a high-dimensional vector (i.e., embedded representation), that is, to obtain a document vector for each document block. The document vector captures the semantic content of the document block, so that similar texts are closer in the vector space. Finally, multiple document vectors are sorted to obtain a vector index library.

[0102] In summary, the construction process of the vector index library is as follows: Figure 2B , obtain the original document, perform semantic splitting on the original document, and obtain multiple document blocks; perform vector conversion on the document block based on the embedding model to obtain the document vector, and collect metadata information of the document block to construct a vector index library.

[0103] After constructing a vector index library through the above process, the vector index library can be used to perform vectorized retrieval on the text vector, find the document vector with high similarity to the text vector, and use the document block corresponding to the document vector as the first relevant document block. Specifically, when performing vector retrieval, for each document vector in the vector index library, it is necessary to calculate its similarity with the text vector. Among them, the similarity measurement methods used include cosine similarity, Euclidean distance, etc., and according to the specific needs of the task, a suitable similarity measurement method can be selected to find the document vector with high similarity to the text vector, and use the document block corresponding to the document vector as the first relevant document block. In the actual application process, in order to control the number and quality of the first relevant document block returned, a similarity threshold can be set. Only when the similarity between the document vector and the text vector exceeds this threshold, the document block corresponding to the document vector is used as the first relevant document block.

[0104] 203. Perform keyword search on the paragraph text to obtain multiple second related document blocks.

[0105] In the embodiment of the present application, in addition to vectorized retrieval, a traditional keyword retrieval method is also used to extract keywords from paragraph text, perform keyword matching retrieval in the document database, and obtain document blocks related to the keywords as the second related document blocks.

[0106] Specifically, the paragraph and chapter structure data corresponding to the sentence can be split out in the document block for keyword index retrieval, and keyword search can be performed through the keyword inverted index library to obtain multiple second related document blocks. When performing keyword search through the keyword inverted index library, it is necessary to extract keywords from each sentence through word segmentation, part-of-speech tagging, stop word filtering, TF-IDF calculation or NLP technology, and use the extracted keywords to build a keyword inverted index library. The inverted index library maps each keyword to a list containing the location where the keyword appears. These location information include sentence ID, paragraph number, chapter title, etc. After the inverted index library is built, the keywords extracted from the paragraph text can be used to search in the inverted index library to quickly locate the sentences containing these keywords. For each matched sentence, the document search platform will group the sentences according to their structural data (paragraphs, chapters), extract relevant contextual information (such as adjacent sentences, paragraph summaries, etc.), and then sort the sentences according to relevance (such as TF-IDF scores, location information, user behavior data, etc.). Since the results are in sentences, but a larger document block is required as the return unit, the document search platform will regard the document block containing the matching sentences as the second relevant document block.

[0107] In this way, by combining keyword search, it is possible to supplement documents that may be missed by vectorized search but directly contain keywords; moreover, by structuring document blocks, when displaying the results, not only the sentences containing keywords are displayed, but also the paragraphs and chapters to which these sentences belong, which helps users understand the contextual position of keywords in the document.

[0108] 204. Calculate the document relevance between each first relevant document block and each second relevant document block and the paragraph text, and query whether there is at least one candidate relevant document block whose document relevance is greater than or equal to the relevance threshold among the plurality of first relevant document blocks and the plurality of second relevant document blocks. When the query determines that there is at least one candidate relevant document block, execute the following step 205; when the query determines that there is no at least one candidate relevant document block, execute the following steps 206 to 208.

[0109] In an embodiment of the present application, after obtaining multiple relevant document blocks through vector retrieval and keyword retrieval, the document search platform calculates the document relevance between each first relevant document block and each second relevant document block and the paragraph text, and queries whether there is at least one candidate relevant document block whose document relevance is greater than or equal to the relevance threshold among the multiple first relevant document blocks and the multiple second relevant document blocks, thereby screening the relevant document blocks, retaining some relevant document blocks with higher relevance to the paragraph text as candidate relevant documents, and further evaluating the candidate relevant documents.

[0110] In actual application, when calculating the document relevance between each first relevant document block and each second relevant document block and the paragraph text, a mathematical measurement method can be used to quantify the similarity or correlation between the two. Specifically, based on text vectorization technology, the text content can be converted into vectors in mathematical space, and then the geometric relationship between these vectors can be used to evaluate the similarity between texts:

[0111] First, we vectorize the text of each relevant document block and paragraph text, perform word segmentation on the relevant document blocks and paragraph text, divide the continuous text into independent vocabulary units, and extract features useful for calculating relevance from the word segmentation results. In the TF-IDF model, these features are usually the weights of the vocabulary, that is, the importance score of the vocabulary in the document, which comprehensively considers the frequency of occurrence of the vocabulary in the document (TF) and the inverse document frequency (IDF) in the entire document set. Based on the extracted features, the relevant document blocks and paragraph text are converted into high-dimensional vectors respectively.

[0112] Then, the document relevance is calculated based on the cosine similarity, Jaccard similarity, BM25 and other methods of the vector space model. Taking the use of cosine similarity to calculate the relevance between each related document block and paragraph text as an example, cosine similarity evaluates their similarity by measuring the angle between two vectors in the direction. The smaller the angle (that is, the closer the cosine value is to 1), the more similar the two vectors are, and thus the content of the related document block and the paragraph text is more relevant.

[0113] Finally, after obtaining the document relevance between all relevant document blocks and paragraph text, a relevance threshold is set in the embodiment of the present application. The threshold is used to distinguish between relevant document blocks with high relevance and low relevance. The document search platform will traverse these document relevances to check whether there is at least one candidate relevant document block whose document relevance is greater than or equal to the relevance threshold.

[0114] Accordingly, when the query determines that there is at least one candidate related document block, it means that these candidate related document blocks have a high correlation with the content of the paragraph text and may be useful results of the user's query. The document search platform will select some candidate related document blocks with higher document relevance to trace the original document, so as to provide the user with the tracing results, that is, execute the following step 205; and when the query determines that there is no at least one candidate related document block, it is necessary to further parse the user's paragraph text through NLP technology, identify the entities and relationships in the paragraph text, search through the graph database, and retrieve the relevant document blocks again. Finally, the Rerank model is used to re-sort the paragraph text and the results according to semantic relevance to obtain the top few document blocks for document tracing, and the traced documents are used as the target original documents and output, that is, execute the following steps 206 to 208.

[0115] 205. When the query determines that there is at least one candidate related document block, deduplicate the at least one candidate related document block, sort the processed at least one candidate related document block in descending order of document relevance to obtain a first sorting result, and extract a first preset number of candidate related document blocks that are ranked first in the first sorting result, extract target metadata information from the first preset number of candidate related document blocks, and output the document traced back through the target metadata information as the target original document.

[0116] In an embodiment of the present application, when the query determines that there is at least one candidate related document block, it means that these candidate related document blocks have a high relevance to the content of the paragraph text and may be useful results for the user's query. Therefore, the document search platform will select some candidate related document blocks with higher document relevance to trace the original document in order to provide the user with tracing results.

[0117] In order to further improve the quality of results and user experience, the document search platform will perform a series of processing on these candidate related document blocks. Among them, since multiple query conditions or algorithms may cause multiple candidate related document blocks to be different parts or different representations of the same document, the first is deduplication processing. The key to deduplication is to identify repeated candidate related document blocks, which can be achieved by comparing the unique identifiers (such as document IDs, paragraph numbers, etc.) or content hash values ​​of document blocks. After removing duplicates, the document search platform will sort the processed candidate related document blocks in order from high to low document relevance to obtain the first sorting result, ensuring that the results that are most relevant to the query and most likely to meet user needs are ranked first. After the sorting is completed, the document search platform will extract the first preset number of candidate related document blocks ranked first from the first sorting result. The first preset number is a value set according to actual needs, which determines how many most relevant results the user will see.

[0118] For the first preset number of candidate related document blocks extracted, the document search platform will further extract the target metadata information therein. As can be seen from the content of the above step 202, the metadata information includes one or more of the paragraph information, chapter information, title information and author information of the corresponding document paragraph. Therefore, the target metadata information refers to the metadata that can uniquely identify or point to the original document, which is crucial for subsequent tracing and document output. Finally, using the extracted target metadata information, the document search platform will trace back to the corresponding original document. The tracing process may involve querying a database, a file system or a network resource, etc.

[0119] Once the target original document is found, the document search platform will output it to the user. For documents that are shorter or that users want to browse quickly, the document search platform can directly display the content of the document on the user interface (such as a web page, application interface, etc.), so that users can immediately obtain the required information without leaving the current interface or performing additional operations. It should be noted that in order to improve the user experience, the document search platform may use technologies such as paging, scrolling loading or dynamic rendering to optimize the display of documents to ensure clear readability of the content and smooth loading.

[0120] When the document content is long or the user prefers to read in a specific environment, the document search platform can provide a link to the document (such as a URL address). After the user clicks the link, the document can be opened in a new browser tab or application for more in-depth viewing or saving.

[0121] For documents that users need to view offline or process further, the document search platform will provide a download option. Users can click a download button or link to save the document to their local device. During the download process, the platform may display information such as download progress and file size, and provide prompts after the download is complete. In addition, in order to help users quickly locate key information in the query results, the document search platform can mark the keywords in the document in yellow and highlight them. These keywords can be words or phrases in the paragraph text entered by the user when searching, and the locations where they appear in the document will be specially marked. The yellowing and highlighting functions not only improve the readability of the information, but also enhance the user's search experience. Users can use these marks to quickly understand the correlation between the document and the paragraph text they entered to be traced, and make further decisions.

[0122] 206. When the query determines that there is no at least one candidate related document block, NLP technology is used to analyze the entities and relationships of the paragraph text to obtain the analysis result of the paragraph text.

[0123] In the embodiment of the present application, when the query determines that there is no at least one candidate related document block, it is necessary to further parse the user's paragraph text through NLP technology, identify the entities and relationships in the paragraph text, search through the graph database, search the related document blocks again, and finally use the Rerank model to re-sort the paragraph text and the results according to semantic relevance to obtain the top few document blocks for document tracing, and output the traced documents as the target original documents. Therefore, the document search platform will use NLP technology to analyze the entities and relationships of the paragraph text, and obtain the analysis results of the paragraph text, so as to use the analysis results to realize the graph retrieval of the paragraph text later.

[0124] When using NLP technology to analyze the entities and relationships of paragraph text, it is necessary to extract paper entities, attributes, and relationships. Among them, the analyzed entities include papers, authors, institutions, funds, organizations, conferences, research fields, citations, publications, etc.; the analyzed relationships include writing, publishing, involved fields, citations, etc.; the analyzed attributes include title, abstract, publication year, keywords, etc. By integrating the extracted entity, relationship, and attribute information, the analysis results can be obtained. The analysis results can be in the form of triple data. Triple data not only shows the internal connection between academic papers, but also contains rich metadata information, which provides a basis for graph retrieval.

[0125] 207. Perform graph retrieval on the analysis results in the graph database, use the retrieved at least one target triple data as the retrieval result, query the document block corresponding to each target triple data respectively, obtain at least one third related document block, and calculate the document relevance between each third related document block and the paragraph text.

[0126] In an embodiment of the present application, after obtaining the analysis results of the paragraph text, the document search platform will perform a graph search on the analysis results in the graph database, and use the retrieved at least one target triple data as the search result. Graph search allows users to browse the relationships between academic papers in a graphical way and quickly locate relevant papers and research fields. At the same time, users can also search in the graph according to specific query conditions (such as author, institution, research field, etc.) to obtain more accurate and comprehensive results. In order to realize graph search, a graph database is pre-constructed in the document search platform. The graph database includes triple data extracted from multiple original documents. The construction process of the graph database is described below:

[0127] First, it is necessary to obtain multiple document blocks obtained by splitting multiple original documents, and identify the entity information, attribute information and relationship information of each document block. Among them, the process of splitting the document block is consistent with the process described in the above step 202, and will not be repeated here. After obtaining the document block, the document search platform will use the Named Entity Recognition (NER) method in NLP technology to extract key entities such as papers, authors, institutions, funds, conferences, research fields, citations, and publications from the document block as the entity information of the document block; on the basis of entity extraction, further use the relationship extraction technology to identify the semantic relationship between entities and form relationship information, which includes writing, publishing, involving fields, and citations; extract attribute information related to the entity, which includes title, abstract, publication year, keywords, etc.

[0128] Subsequently, the document search platform will clean up the duplicate and contradictory content in the entity information, attribute information, and relationship information of each document block. When cleaning up duplicate content, the document search platform will traverse the entity, attribute, and relationship information of all document blocks, use data structures such as hash tables or sets to record unique items that have appeared, and check whether each item (entity, attribute, relationship) in each document block already exists in the hash table or set. If duplicate items are found, decide whether to merge the information (such as merging different attributes of the same entity) or retain one instance and delete other duplicate items based on business needs. It should be noted that when merging information, it is necessary to pay attention to the priority of attributes and conflict resolution strategies to ensure that the merged information is meaningful. Contradictory content usually refers to the situation where there are conflicting descriptions of the same entity or relationship in the same document block or different document blocks. It is necessary to define which information is considered contradictory based on business rules or domain knowledge. When cleaning up contradictory content, the document search platform will use rule engines or machine learning models to detect contradictory content. For detected contradictions, a variety of resolution strategies are adopted, such as retaining the most trusted information source, prompting users to resolve conflicts, or automatically selecting a solution based on predefined rules.

[0129] Next, the document search platform will use the cleaned entity information, attribute information, and relationship information to generate triple data for each document block, and obtain multiple triple data for multiple document blocks. Triple data usually consists of "entity-relationship-entity" or "entity-attribute-value". For each document block, the document search platform will traverse its cleaned entities, attributes, and relationships, and for each entity, extract all its related attributes and relationships and construct them into triple data.

[0130] Finally, multiple triple data are sorted to obtain a graph database for subsequent query and retrieval. Furthermore, an index can be established for the triple data to improve query efficiency.

[0131] In summary, the construction process of the graph database is as follows: Figure 2C , obtain the original document, split the original document to obtain document blocks, and perform entity extraction, relationship extraction and attribute extraction on the document blocks. Among them, entity extraction includes extracting entities such as papers, authors, institutions, and research fields of document blocks; relationship extraction includes extracting relations such as works, publications, locations, research directions, and citations of document blocks; attribute extraction includes extracting attributes such as DOI and publication year of document blocks. Next, the extracted data is subjected to reference resolution and entity disambiguation, and triple data is constructed to obtain a graph database.

[0132] After the graph database is constructed through the above process, the document search platform will perform graph retrieval on the analysis results in the graph database, take the retrieved at least one target triple data as the retrieval result, query the document block corresponding to each target triple data respectively, obtain at least one third related document block, and calculate the document relevance between each third related document block and the paragraph text. The process of calculating the document relevance between each third related document block and the paragraph text is consistent with the process of calculating the document relevance described in the above step 205, and will not be repeated here.

[0133] 208. De-duplicate the multiple first related document blocks, the multiple second related document blocks, and the at least one third related document block, and sort the processed multiple first related document blocks, the multiple second related document blocks, and the at least one third related document block in order of document relevance from high to low to obtain a second sorting result, extract a second preset number of specified related document blocks that are ranked first in the second sorting result, extract specified metadata information from the second preset number of specified related document blocks, and output the document traced back through the specified metadata information as the target original document.

[0134] In the embodiment of the present application, since a batch of related document blocks have been retrieved previously through vector retrieval and keyword retrieval, the document search platform will merge multiple first related document blocks, multiple second related document blocks and at least one third related document block into a set, and perform deduplication processing on this set. The key to deduplication is to identify and remove duplicate document blocks, which can be achieved by comparing the unique identifiers of the document blocks (such as document IDs, hash values, etc.). After deduplication, a document block set that does not contain duplicate items can be obtained.

[0135] Next, the document search platform sorts the deduplicated document block set in descending order of document relevance to obtain a second sorting result. The purpose of sorting is to place the most relevant document block in front of the second sorting result, and then extract the second preset number of designated relevant document blocks ranked first from the second sorting result. The second preset number is a value set according to actual needs, which determines how many most relevant results the user will see. The second preset number may be consistent with the value of the first preset number mentioned above, or it may be inconsistent, and this application does not specifically limit this.

[0136] For the second preset number of specified related document blocks extracted, the document search platform will extract the specified metadata information therein. The specified metadata information refers to metadata that can uniquely identify or point to the original document. Using the extracted specified metadata information, the document search platform will trace back to the corresponding target original document. The tracing process may involve querying a database, a file system, or a network resource. Once the target original document is found, the document search platform will output it to the user. The output form may be to directly display the document content on the user interface, or to provide a link or download option for the document. The specific process is consistent with the process described in the above step 206, and will not be repeated here.

[0137] During actual application, the document search platform will encrypt all data to protect user privacy and set strict access rights to ensure data security. Moreover, the document search platform establishes a regular update mechanism to synchronize the knowledge base according to the update of the academic database to maintain the timeliness of the data. Furthermore, the document search platform collects user feedback, continuously optimizes the model and retrieval strategy, and optimizes the algorithm and resource allocation according to the system performance monitoring results to improve the retrieval speed. In addition, the document search platform trains specific RAG models for different subject areas to improve the pertinence and accuracy of the retrieval. Finally, the document search platform designs an intuitive user interface to facilitate users to enter queries and provide a personalized retrieval experience.

[0138] In summary, the design of the traceability model in this application actually includes three aspects, see Figure 2D :After obtaining the original document and cleaning and organizing it, the document search platform will split the original document into document blocks according to semantics, and each document block corresponds to metadata information. Next, on the one hand, the embedding model will be used to vectorize the document blocks and build a vector index library for vector indexing; on the other hand, the document blocks will be structured, and an inverted index database will be built to implement keyword indexing; on the third hand, the triple data of the document blocks will be identified and a graph database will be built for graph indexing. These aspects are also the functions implemented by the RAG model built based on RAG technology in this application.

[0139] Therefore, the logical process of the technical solution in this application is summarized as follows: Figure 2E, in response to the journal article traceability request, obtain the paragraph text to be traced, perform vector indexing and keyword indexing, and thus obtain a series of related document blocks. Then merge and remove duplicates of these related document blocks, calculate the document relevance between each related document block and the paragraph text, and determine whether there is a related document block with a document relevance greater than the relevance threshold in these related document blocks. If not, continue to analyze the entity relationship of the paragraph text, perform graph retrieval on the paragraph text, and then obtain another series of related document blocks, put the related document blocks obtained this time together with the previous related document blocks for rearrangement, select the top k related document blocks to obtain the metadata information of these related document blocks, and output the original documents traced based on these metadata information as the target original document to the user. If there are related document blocks with a document relevance greater than the relevance threshold in the related document blocks, directly sort the related document blocks in order from large to small in terms of document relevance, select the top k related document blocks to obtain the metadata information of these related document blocks, and output the original documents traced based on these metadata information as the target original document to the user.

[0140] The method provided in the embodiment of the present application, by combining a multimodal retrieval strategy of vectorized retrieval and keyword retrieval, fully utilizes the advantages of different retrieval methods, captures the complex associations and contextual relationships between documents, and while improving the accuracy and efficiency of document traceability, builds a vector index library based on RAG technology, further enhancing the depth and breadth of retrieval, ensuring the depth and comprehensiveness of the retrieval results, and providing strong technical support for document traceability.

[0141] Further, as Figure 1 The specific implementation of the method, the embodiment of the present application provides a journal paper tracing device based on RAG technology, such as Figure 3 As shown, the device includes: a processing module 301, a vector retrieval module 302, a keyword retrieval module 303 and an output module 304.

[0142] The processing module 301 is used to obtain paragraph text to be traced, and perform vectorization processing on the paragraph text to obtain a text vector;

[0143] The vector retrieval module 302 is used to perform vectorized retrieval on the text vector through a vector index library to obtain a plurality of first related document blocks, wherein the vector index library is constructed based on the RAG technology;

[0144] The keyword search module 303 is used to perform keyword search on the paragraph text to obtain multiple second related document blocks;

[0145] The output module 304 is used to perform a document tracing operation by referring to the document relevance between each of the first related document blocks and each of the second related document blocks and the paragraph text, obtain the target original document and output it.

[0146] In a specific application scenario, the processing module 301 is used to respond to a journal paper tracing request, obtain the uploaded text content as the paragraph file to be traced, and perform preprocessing operations on the paragraph text; obtain a pre-trained deep learning model, input the preprocessed paragraph text into the deep learning model, and obtain the text vector.

[0147] In a specific application scenario, when constructing the vector index library, the document collection requirements are obtained, and multiple original documents are collected according to the document collection requirements; data cleaning operations are performed on the multiple original documents, and NLP technology is used to perform text standardization on the cleaned multiple original documents; each of the processed original documents is structured, and multiple document blocks are generated for each of the original documents; the multiple document blocks of each of the original documents are input into a pre-trained deep learning model for vector conversion to obtain multiple document vectors, and the deep learning model is constructed using a neural network model indicated by the RAG technology; the multiple document vectors are sorted to obtain the vector index library.

[0148] In a specific application scenario, when constructing the vector index library, for each of the original documents, structural elements in the original document are identified, and based on the identified structural elements, multiple document paragraphs included in the original document are determined, wherein the structural elements include one or more of a chapter title, a paragraph start, and a paragraph end; metadata information is generated for each of the document paragraphs based on the structural elements in each of the document paragraphs, and the metadata information includes one or more of paragraph information, chapter information, title information, and author information of the corresponding document paragraph; each of the document paragraphs is traversed, and based on sentence boundaries detected in each of the document paragraphs, each of the document paragraphs is split into multiple independent sentences; a document block is constructed for each of the multiple sentences, and the metadata information of the document paragraph to which each of the sentences belongs is stored in the document block of the sentence, thereby obtaining multiple document blocks of the original document.

[0149] In a specific application scenario, the output module 304 is used to calculate the document relevance between each of the first relevant document blocks and each of the second relevant document blocks and the paragraph text, and query whether there is at least one candidate relevant document block whose document relevance is greater than or equal to the relevance threshold among the multiple first relevant document blocks and the multiple second relevant document blocks; when the query determines that the at least one candidate relevant document block exists, perform deduplication processing on the at least one candidate relevant document block, sort the processed at least one candidate relevant document block in order of document relevance from high to low, obtain a first sorting result, and extract the candidate relevant document blocks ranked in the first sorting result. A first preset number of candidate related document blocks are first selected, target metadata information is extracted from the first preset number of candidate related document blocks, and the document traced back through the target metadata information is used as the target original document and output; when the query determines that there is no at least one candidate related document block, the NLP technology is used to perform entity and relationship analysis on the paragraph text to obtain the analysis result of the paragraph text, and the graph search is performed on the analysis result through the graph database to obtain the retrieval result, and the document traced back based on the retrieval result, the multiple first related document blocks and the multiple second related document blocks is used as the target original document and output.

[0150] In a specific application scenario, when constructing the graph database, multiple document blocks obtained by splitting multiple original documents are obtained, and the entity information, attribute information and relationship information of each document block are identified; the duplicate content and contradictory content in the entity information, attribute information and relationship information of each document block are cleaned up, and the cleaned entity information, attribute information and relationship information are used to generate triple data for each document block to obtain multiple triple data of the multiple document blocks; the multiple triple data are sorted to obtain the graph database.

[0151] In a specific application scenario, the output module 304 is used to perform graph retrieval on the analysis result in the graph database, and use the retrieved at least one target triple data as the retrieval result; query the document block corresponding to each of the target triple data respectively to obtain at least one third related document block, and calculate the document relevance between each of the third related document blocks and the paragraph text; deduplicate the multiple first related document blocks, the multiple second related document blocks and the at least one third related document block, and sort the processed multiple first related document blocks, the multiple second related document blocks and the at least one third related document block in order of document relevance from high to low to obtain a second sorting result; extract a second preset number of specified related document blocks ranked first in the second sorting result, extract specified metadata information from the second preset number of specified related document blocks, and output the document traced back through the specified metadata information as the target original document.

[0152] The device provided in the embodiment of the present application obtains the paragraph text to be traced, vectorizes the paragraph text to obtain the text vector, performs vectorized retrieval on the text vector through a vector index library to obtain multiple first related document blocks, the vector index library is constructed based on RAG technology, performs keyword retrieval on the paragraph text to obtain multiple second related document blocks, refers to the document relevance between each first related document block and each second related document block and the paragraph text, performs document tracing operations, obtains the target original document and outputs it, and through a multimodal retrieval strategy combining vectorized retrieval and keyword retrieval, fully utilizes the advantages of different retrieval methods, captures the complex associations and contextual relationships between documents, and while improving the accuracy and efficiency of document tracing, constructs a vector index library based on RAG technology, further enhances the depth and breadth of the retrieval, ensures the depth and comprehensiveness of the retrieval results, and provides strong technical support for document tracing.

[0153] It should be noted that for other corresponding descriptions of the functional units involved in the journal paper tracing device based on RAG technology provided in the embodiment of the present application, please refer to Figure 1 and FIG. 2A to FIG. 2E The corresponding description in will not be repeated here.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0155] The above embodiments and the technical features in the embodiments can be arbitrarily combined with each other. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0156] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

[0157] In an exemplary embodiment, see Figure 4 , and also provides a computer device, which includes a bus, a processor, a memory and a communication interface, and may also include an input and output interface and a display device, wherein each functional unit can communicate with each other through the bus. The memory stores a computer program, and the processor is used to execute the program stored in the memory and execute the journal paper tracing method based on RAG technology in the above embodiment.

[0158] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the journal paper tracing method based on RAG technology.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by hardware, or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0160] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily required for implementing the present application.

[0161] Those skilled in the art will appreciate that the modules in the device in the implementation scenario can be distributed in the device in the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more devices different from the implementation scenario. The modules in the above implementation scenario can be combined into one module, or can be further split into multiple sub-modules.

[0162] The above application serial numbers are for description only and do not represent the advantages or disadvantages of the implementation scenarios.

[0163] The above disclosure only discloses several specific implementation scenarios of the present application. However, the present application is not limited thereto, and any changes that can be conceived by technicians in this field should fall within the scope of protection of the present application.

Claims

1. A journal article tracing method based on RAG technology, characterized in that: include: Obtaining paragraph text to be traced, and performing vectorization processing on the paragraph text to obtain a text vector; Performing vectorized retrieval on the text vectors through a vector index library to obtain a plurality of first relevant document blocks, wherein the vector index library is constructed based on RAG technology; Perform keyword search on the paragraph text to obtain multiple second related document blocks; Referring to the document relevance between each of the first relevant document blocks and each of the second relevant document blocks and the paragraph text, a document tracing operation is performed to obtain and output the target original document.

2. The method according to claim 1, characterized in that The step of obtaining the paragraph text to be traced and performing vectorization processing on the paragraph text to obtain a text vector includes: In response to a journal article traceability request, obtaining uploaded text content as the paragraph file to be traced, and performing a preprocessing operation on the paragraph text; A pre-trained deep learning model is obtained, and the pre-processed paragraph text is input into the deep learning model to obtain the text vector.

3. The method according to claim 1, characterized in that When constructing the vector index library, the method includes: Obtaining document collection requirements, and collecting multiple original documents according to the document collection requirements; Performing data cleaning operations on the multiple original documents, and using NLP technology to perform text standardization processing on the cleaned multiple original documents; Performing structural processing on each of the processed original documents, and generating a plurality of document blocks for each of the original documents; Inputting multiple document blocks of each of the original documents into a pre-trained deep learning model for vector conversion to obtain multiple document vectors, wherein the deep learning model is constructed using a neural network model indicated by the RAG technology; The multiple document vectors are sorted to obtain the vector index library.

4. The method according to claim 3, characterized in that The step of performing structural processing on each of the processed original documents and generating a plurality of document blocks for each of the original documents comprises: For each of the original documents, identifying structural elements in the original document, and determining a plurality of document paragraphs included in the original document according to the identified structural elements, wherein the structural elements include one or more of a chapter title, a paragraph start, and a paragraph end; Generate metadata information for each of the document paragraphs according to the structural elements in each of the document paragraphs, the metadata information including one or more of paragraph information, chapter information, title information, and author information of the corresponding document paragraph; Traversing each of the document paragraphs, and splitting each of the document paragraphs into a plurality of independent sentences according to sentence boundaries detected in each of the document paragraphs; A document block is constructed for each of the multiple sentences, and metadata information of the document paragraph to which each sentence belongs is stored in the document block of the sentence, thereby obtaining multiple document blocks of the original document.

5. The method according to claim 1, characterized in that: The step of referring to the document relevance between each of the first related document blocks and each of the second related document blocks and the paragraph text, performing a document tracing operation, obtaining a target original document and outputting the document comprises: Calculating the document relevance between each of the first relevant document blocks and each of the second relevant document blocks and the paragraph text, and querying whether there is at least one candidate relevant document block whose document relevance is greater than or equal to a relevance threshold among the plurality of first relevant document blocks and the plurality of second relevant document blocks; When the query determines that the at least one candidate related document block exists, deduplication processing is performed on the at least one candidate related document block, the processed at least one candidate related document block is sorted in descending order of document relevance to obtain a first sorting result, and a first preset number of candidate related document blocks ranked first in the first sorting result are extracted, target metadata information is extracted from the first preset number of candidate related document blocks, and the document traced back through the target metadata information is used as the target original document and outputted; When the query determines that at least one candidate related document block does not exist, NLP technology is used to analyze the entities and relationships of the paragraph text to obtain the analysis result of the paragraph text, and a graph search is performed on the analysis result through a graph database to obtain a retrieval result, and the document traced back based on the retrieval result, the multiple first related document blocks and the multiple second related document blocks is used as the target original document and output.

6. The method according to claim 5, characterized in that When constructing the atlas database, the method includes: Acquire multiple document blocks obtained by splitting multiple original documents, and identify entity information, attribute information, and relationship information of each document block; Cleaning up duplicate and contradictory contents in the entity information, attribute information and relationship information of each document block, and generating triple data for each document block using the cleaned entity information, attribute information and relationship information to obtain a plurality of triple data of the plurality of document blocks; The multiple triplet data are sorted to obtain the atlas database.

7. The method according to claim 6, characterized in that The step of performing a graph search on the analysis result through a graph database to obtain a search result, and outputting a document traced back based on the search result, the plurality of first related document blocks, and the plurality of second related document blocks as the target original document, includes: Performing a graph search on the analysis result in the graph database, and taking the retrieved at least one target triple data as the search result; Respectively query the document blocks corresponding to each of the target triple data to obtain at least one third related document block, and calculate the document relevance between each of the third related document blocks and the paragraph text; performing deduplication processing on the plurality of first related document blocks, the plurality of second related document blocks, and the at least one third related document block, and sorting the processed plurality of first related document blocks, the plurality of second related document blocks, and the at least one third related document block in descending order of document relevance to obtain a second sorting result; A second preset number of designated related document blocks ranked first in the second sorting result are extracted, designated metadata information is extracted from the second preset number of designated related document blocks, and the document traced back through the designated metadata information is used as the target original document and output.

8. A journal article tracing device based on RAG technology, characterized in that: include: A processing module, used for obtaining paragraph text to be traced, and performing vectorization processing on the paragraph text to obtain a text vector; A vector retrieval module, used to perform vectorized retrieval on the text vector through a vector index library to obtain a plurality of first relevant document blocks, wherein the vector index library is constructed based on the RAG technology; A keyword search module, used for performing keyword search on the paragraph text to obtain a plurality of second related document blocks; The output module is used to refer to the document relevance between each of the first related document blocks and each of the second related document blocks and the paragraph text, perform a document tracing operation, obtain a target original document and output it.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images