Literature retrieval and content extraction method and computer readable storage medium

Abstract

The invention provides a literature retrieval and content extraction method and a computer readable storage medium, and the method adopts a hierarchical processing architecture, and comprises the following steps: firstly, vectorizing literatures, and storing the vectorized literatures in a first vector database; vectorizing the retrieval requirements through a first artificial intelligence model and performing preliminary screening to obtain related literatures; fine screening is carried out through a second artificial intelligence model trained in a specific field, high-correlation literatures are obtained, and the retrieval efficiency of the first model is higher than that of the second model; then, high-correlation literatures are sliced and vectorized to construct a second vector database, and N most relevant text slices are recalled from the second vector database based on retrieval requirements; and finally, extracting and outputting a result related to the retrieval demand based on the text slices through a third artificial intelligence model. According to the method, through cooperative work of a plurality of models with clear division of labor, the professional literature screening precision and the final content extraction accuracy are remarkably improved while the processing efficiency is ensured, and the illusion risk of a large model is effectively inhibited.

Description

Technical Field

[0001] This invention relates to the field of information processing, and in particular to a method for document retrieval and content extraction, and a computer-readable storage medium. Background Technology

[0002] Scientific literature is a crucial carrier of knowledge in scientific research activities. With the rapid growth of scientific output, the number of documents included in various scientific literature databases has increased dramatically, posing a severe challenge to efficient knowledge retrieval and extraction. Traditional keyword-based retrieval methods are gradually showing limitations when dealing with professional literature searches: they struggle to effectively handle the synonymy and diversity of professional terms, have limited ability to understand complex semantic queries, and still require significant manual intervention to extract key information from the search results, resulting in overall efficiency that needs improvement.

[0003] Despite significant advancements in semantic understanding from existing AI technologies, and the improved inclusivity of semantic vector-based retrieval methods for diverse expressions, numerous challenges remain when processing specialized content such as scientific literature. General-purpose models lack a deep understanding of domain-specific knowledge, are prone to missing crucial information when handling long documents, and struggle to guarantee the accuracy and structure of extracted knowledge. While some existing improvements attempt to mitigate these issues by combining retrieval and generation, most are not optimized for the complexities of scientific literature, still exhibiting shortcomings such as biases in understanding technical terminology and difficulties in tracing the chain of evidence.

[0004] Therefore, there is an urgent need for a method that can retrieve and extract documents efficiently, traceably, and with high accuracy. Summary of the Invention

[0005] In order to overcome the above-mentioned technical defects, the purpose of this invention is to provide a document retrieval and content extraction method and a computer-readable storage medium.

[0006] This invention discloses a method for document retrieval and content extraction, comprising: vectorizing the documents and storing them in a first vector database; The first artificial intelligence model is used to vectorize the search requirements, and relevant documents are initially screened from the vector database based on vector similarity. The second AI model further filters relevant literature based on domain relevance to obtain highly relevant literature; the first AI model has a higher retrieval efficiency than the second AI model; the second AI model is trained based on a specific domain. Highly relevant literature is sliced ​​and vectorized to construct a second vector database; Based on the retrieval requirements, the N most relevant text slices are retrieved from the second vector data; The third artificial intelligence model extracts and outputs results relevant to the search requirements based on N text slices.

[0007] Preferably, storing the vectorized documents in the first vector database includes: Using DOI as a unique identifier for each document, we extract the title and abstract fields to construct metadata. Perform missing value processing, deduplication and merging on metadata, and complete field alignment and unified encoding conversion; The processed metadata is then vectorized in batches and imported into the first vector database.

[0008] Preferably, the retrieval requirements are vectorized using a first artificial intelligence model, and relevant documents are initially screened from the vector database based on vector similarity, including: The output dimension of the first artificial intelligence model is fixed at D, and the user query Q and the document metadata D are encoded into semantic vectors of a unified dimension h. The first AI is deployed in a memory-resident manner on the accelerated computing unit, resulting in a single batch similarity calculation latency of <1000ms, and calculating the cosine similarity between the two. Sort by cosine similarity in descending order, and select the top k articles as relevant articles.

[0009] Preferably, the training of the second artificial intelligence includes: Construct a question-and-answer dataset for a specific domain using relevant literature; in, Combine the text of the document title and abstract. To correspond to the full text excerpt, Indicate whether the document mentions content related to the search requirements; Define the reward function: in, Rewards for correct answers Rewards for effective classification To ensure strict format rewards, For lenient reward format, Rewards for XML counting; The weights w1, w2, w3, w4, and w5 are determined by a grid search of the validation set. The second artificial intelligence is trained using a group relative policy optimization algorithm: in each iteration, from Sample small batches of data, and generate predictions using the model. And calculate the reward Using the objective function Update the strategy parameters; where, for The expectation operator below represents the averaging of trajectory samples generated by the strategy; t is the time step index in the generated sequence; Indicates the state Next, select an action (generate a token). The probability is calculated until the reward converges or reaches the preset performance index, thus obtaining the trained second artificial intelligence model.

[0010] Preferably, the second artificial intelligence model further filters relevant literature based on domain relevance, resulting in highly relevant literature including: The titles and abstracts of relevant literature are used to construct prompts for input into a second artificial intelligence model; Generate a complete output including the reasoning path and classification results. ,from Extracting classification results with reasoning path ; Highly relevant literature was obtained.

[0011] Preferably, before slicing and vectorizing highly relevant documents, the document retrieval and content extraction method further includes: Identify the overall layout of highly relevant literature, distinguishing between titles, paragraphs, figures, and formula areas; Use the OCR pipeline for scanned pages and introduce syntax rules for correction in formula or chemical formula areas; Call the table structure recognition module to extract cell content and coordinates to generate standardized metadata; By detecting hierarchical headings and reconstructing the reading order, text, formulas, and table data are integrated and converted into hierarchical structured text.

[0012] Preferably, slicing and vectorizing highly relevant literature further includes: Preprocess the structured text to remove the reference section; A recursive character segmentation method is used to divide the text into multiple slice units of a preset length; For each slice unit, an extended context window containing a predetermined number of characters before and after it is generated and stored in metadata; An embedding model is used to encode the text content of each slice unit into a dense vector and then store it in a second vector database.

[0013] Preferably, based on the retrieval requirements, retrieving the N most relevant text slices from the second vector database includes: The search requirements are encoded into vectors, and the vector similarity scores of each slice are obtained using the vector index. Simultaneously, the search requirements are segmented into words, and the BM25 relevance scores of each slice are obtained based on keyword retrieval; The vector similarity score and the BM25 correlation score are weighted and fused to obtain a comprehensive correlation score; Sort by overall relevance score and select the top N slices.

[0014] Preferably, the third artificial intelligence model extracts and outputs results related to the retrieval requirements based on N text slices, including: N text slices are concatenated in reading order, multiple third-party AI models are called for parallel processing, and multiple outputs are fused through a voting mechanism by a third-party AI model to output the final result. And / or, using the coordinates of N text slices in the original document as indices, extract the complete original document as context, build a structured extraction tool and drive the large language model to call the tool to extract information from the context and output the results.

[0015] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the aforementioned document retrieval and content extraction methods.

[0016] Compared with existing technologies, the above technical solution has the following advantages: 1. This application employs a hierarchical processing architecture, utilizing multiple AI models with clearly defined roles to sequentially perform document screening and content extraction. The first AI model achieves rapid initial screening based on semantic vectors; the second AI model acquires professional knowledge through domain-adaptive training, enabling it to more accurately understand technical terms and complex concepts, thereby improving screening accuracy; the third AI model is prone to illusions in excessively long contexts, while this method strictly limits its processing scope to the highly relevant text fragments screened by the second model. This contextual constraint effectively suppresses interference from irrelevant information and significantly reduces the risk of illusions. The entire process, while ensuring efficiency, ultimately achieves accurate and reliable content extraction by optimizing context quality layer by layer. 2. Secondly, this application also establishes a standardized document meta-database through standardized data preprocessing, providing a reliable foundation for subsequent processing. Semantic vector-based similarity calculation achieves deep content matching, overcoming the limitations of traditional keyword retrieval. Through reinforcement learning training strategies, the model acquires professional discriminative capabilities, synchronously outputting inference paths during the screening process, making the model's decision-making process transparent and traceable, thus improving the system's practicality and reliability. 3. Unstructured documents are again converted into structured text that retains the original layout and content elements. Through reasonable text segmentation and context preservation strategies, the document content is meticulously organized. Combining the advantages of semantic and keyword retrieval methods, a complementary retrieval mechanism is formed, capable of understanding deep semantics while capturing specific terms, significantly improving the comprehensiveness and accuracy of the retrieval. 4. Finally, through multi-model collaboration and structured output constraints, the accuracy of content extraction is effectively improved, and errors caused by model over-interpretation are suppressed. Implementing the method as an executable program makes the entire technical solution deployable and reproducible, facilitating practical application and promotion, and providing users with efficient and reliable literature retrieval and knowledge extraction capabilities. Attached Figure Description

[0017] Figure 1 A schematic diagram illustrating the document retrieval and content extraction method provided in this application; Figure 2 A flowchart illustrating the document retrieval and content extraction methods provided for this application; Figure 3 A schematic diagram of the training method for the second artificial intelligence provided in this application for the document retrieval and content extraction method; Figure 4 A schematic diagram comparing the training method of the second artificial intelligence for the document retrieval and content extraction method provided in this application with the training methods of existing technologies. Detailed Implementation

[0018] The advantages of the present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments.

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0020] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0022] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0023] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0024] In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the convenience of the description of the invention and have no specific meaning in themselves. Therefore, "module" and "part" can be used interchangeably.

[0025] Please see Figures 1-2 , Figure 1 A schematic diagram illustrating the document retrieval and content extraction method provided in this application; Figure 2 A flowchart illustrating the document retrieval and content extraction method provided for this application.

[0026] like Figures 1-2 As shown, this application provides a method for document retrieval and content extraction, including: vectorizing the documents and storing them in a first vector database; The first artificial intelligence model is used to vectorize the search requirements, and relevant documents are initially screened from the vector database based on vector similarity. The second AI model further filters relevant literature based on domain relevance to obtain highly relevant literature; the first AI model has a higher retrieval efficiency than the second AI model; the second AI model is trained based on a specific domain. Highly relevant literature is sliced ​​and vectorized to construct a second vector database; Based on the retrieval requirements, the N most relevant text slices are retrieved from the second vector database; The third artificial intelligence model extracts and outputs results relevant to the search requirements based on N text slices.

[0027] The underlying principle needs to be explained here: This application employs a hierarchical processing architecture, using multiple AI models with clearly defined roles to sequentially perform document screening and content extraction. The first AI model performs rapid initial screening based on semantic vectors; the second AI model acquires professional knowledge through domain-adaptive training, enabling it to more accurately understand technical terms and complex concepts, thereby improving screening accuracy; the third AI model is prone to illusions in excessively long contexts, while this method strictly limits its processing scope to the highly relevant text fragments screened by the second model. This contextual constraint effectively suppresses interference from irrelevant information and significantly reduces the risk of illusions. The entire process, while ensuring efficiency, ultimately achieves accurate and reliable content extraction by optimizing the context quality layer by layer.

[0028] The above is an explanation of the basic concept of this application. The following example, "ferroelectric materials" in the field of materials science, will be used to illustrate the specific implementation of this application. The aim is to extract key information related to this topic from a large amount of literature, such as material name, performance parameters, preparation methods and related physical mechanisms.

[0029] First, it should be noted that there are no restrictions on the specific method of storing the vectorized documents in the first vector database.

[0030] In one possible implementation, storing the vectorized documents in a first vector database includes: Using DOI as a unique identifier for each document, we extract the title and abstract fields to construct metadata. Perform missing value processing, deduplication and merging on metadata, and complete field alignment and unified encoding conversion; The processed metadata is then vectorized in batches and imported into the first vector database.

[0031] Those skilled in the art will understand that the specific methods for handling missing values, deduplication, and merging are not limited. For example, handling missing values ​​specifically involves: traversing all records and removing any document entries that lack the DOI, title, or abstract field.

[0032] Deduplication and merging specifically involves using the DOI as a unique identifier for each document, merging any duplicate records that may exist in the dataset, and ensuring the uniqueness of each document in the database.

[0033] Field alignment and unified encoding specifically involve mapping field names from different data sources (e.g., "ArticleTitle", "Title") to a standard field (title). All text fields are uniformly converted to UTF-8 format to avoid garbled characters or parsing errors in subsequent processing.

[0034] A standardized document metadata database was established through standardized data preprocessing. Unified document identification and a complete cleaning process ensured data quality, providing a reliable foundation for subsequent vectorization and semantic retrieval, and guaranteeing the accuracy of the system's processing results from the source.

[0035] As those skilled in the art will understand, the specific implementation of vectorization is not limited. In one possible implementation, the vector embedding process of this embodiment can rely on the jasper_en_vision_language_v1 model. The vector embedding process employs the advanced multilingual multimodal embedding model jasper_en_vision_language_v1. This model can output vectors representing encoded text or image data, and by integrating a visual encoder and introducing a self-distillation mechanism to align visual embeddings with text embeddings, it develops multimodal retrieval capabilities. Therefore, this model has significant advantages over traditional models.

[0036] Even if the user's query differs from the original document in terms of vocabulary selection, the model can still successfully retrieve relevant information through its deep conceptual understanding capabilities. This ability effectively reduces information omissions caused by the diversity of expression methods, significantly improves the recall and accuracy of document retrieval in complex problem scenarios in professional fields, and provides a higher quality and more comprehensive data foundation for subsequent knowledge extraction and reasoning by large language models.

[0037] Specifically, for each preprocessed document, its title and abstract fields are concatenated into a single text string. This string is then input into the jasper_en_vision_language_v1 model to generate a 1024-dimensional floating-point vector representing the core semantics of the document, which is then imported into the first vector database.

[0038] It should be noted that the specific implementation of the first vector database is also not limited.

[0039] In one possible implementation, the first vector database uses the open-source vector database Milvus to store and manage document metadata and its corresponding semantic vectors. A collection is created within the database to store the document data. The schema of this collection is defined as follows: doi: String type (VARCHAR), used as the primary key to uniquely identify each record.

[0040] title: String type (VARCHAR), stores the original title of the document.

[0041] abstract: String type (VARCHAR), stores the original abstract of the document.

[0042] metadata_vector: A floating-point vector type (FLOAT_VECTOR) with dimensions of 1024, used to store semantic vectors generated by the jasper_en_vision_language_v1 model.

[0043] Create an HNSW (Hierarchical Navigable Small World) index on the metadata_vector field. HNSW is an efficient graph indexing algorithm that enables extremely low-latency approximate nearest neighbor search on large-scale datasets.

[0044] Finally, all preprocessed and vectorized literature data were imported into the Milvus collection in batches. Each imported data entry contained a DOI, title, abstract, and its corresponding 1024-dimensional semantic vector. After importation, Milvus automatically organized all vector data according to the created HNSW index, thus completing the construction of the first vector database.

[0045] Here is an example from a paper on ferromagnetism in compounds: The input document metadata is: DOI: 10.1016 / j.jmmm.2014.01.010 Title: Ferromagnetism in novel compounds of the R3Pt23Si11 series with heavy rare earth: Gd, Tb, Dy, Ho and Er Abstract: We have synthesized new compounds of the $\mathrm{R}_3 \mathrm{Pt}_{23} \mathrm{Si}_{11} $ series with $ \mathrm{R} = \mathrm{Gd} $, Tb,Dy, Ho and Er, and studied their magnetic and thermodynamic properties. X-raypowder diffraction characterization confirms that all these compounds crystalllize in the same face-centered cubic structure, space group... Aligning with the above construction process, the processing procedure for this document is as follows: Preprocessing: The DOI, title, and summary fields of this record are complete and have passed the missing value check. Its DOI is unique in the database and has passed the deduplication check. The text encoding has been converted to UTF-8.

[0046] Semantic vectorization: The title and abstract are concatenated into a long text string and input into the embedding model.

[0047] Generate vector: The model outputs a 1024-dimensional floating-point vector V_example = [0.123, -0.456,..., 0.789].

[0048] Data import: Insert the following structured entities into the Milvus document collection: doi: 10.1016 / j.jmmm.2014.01.010 title_abstract: "[TITLE]High-entropy perovskite oxides as a new platform for robust catalysis[ABSTRACT]We have synthesized new compounds ofthe..." metadata_vector: [0.123, -0.456, ..., 0.789].

[0049] Second, in one possible implementation, the retrieval requirements are vectorized using a first artificial intelligence model, and relevant literature is initially screened from a vector database based on vector similarity, including: The output dimension of the first artificial intelligence model is fixed at D, and the user query Q and the document metadata D are encoded into semantic vectors of a unified dimension h. The first AI is deployed in a memory-resident manner on the accelerated computing unit, resulting in a single batch similarity calculation latency of <1000ms, and calculating the cosine similarity between the two. Sort by cosine similarity in descending order, and select the top k articles as relevant articles.

[0050] Semantic vector-based similarity calculation enables deep content matching, overcoming the limitations of traditional keyword retrieval. This method understands the semantic relationship between queries and documents, significantly improving the coverage of initial searches and providing ample candidate documents for subsequent processing.

[0051] Specifically, this can be achieved through a pre-packaged query engine class (PaperQuery). During initialization, this engine first loads the jasper_en_vision_language_v1 embedding model from the local path. Simultaneously, it loads the document metadata embedding vectors pre-computed and persistently stored in the preceding steps. This file contains the semantic vectors of all documents, the corresponding DOI list, and the original title-summary concatenation text.

[0052] When a user enters a query_text, such as "The Curie temperature offers electric materials", the query engine performs the following operations: Query vectorization: Invoke the loaded jasper_en_vision_language_v1 model to encode the user's query text into a 1024-dimensional query vector with the same dimensions as the document vector.

[0053] Similarity calculation: Perform an ANN search using Milvus's HNSW index to calculate the cosine similarity between the query vector and all document metadata vectors in memory.

[0054] Sorting and filtering: Based on the calculated similarity scores, all documents are sorted in descending order. Then, based on a preset top_k value, the top-ranked documents are selected to form the first round of relevant document set.

[0055] Output: The candidate set of results, including DOI, similarity score, and original title abstract, is saved as a timestamped JSON file for subsequent steps and for traceability. This step enables rapid and semantically relevant preliminary screening of massive amounts of literature, outputting the first round of relevant literature results.

[0056] It should be noted that the first artificial intelligence model here can either directly search the full text of the document, thus ensuring a comprehensive search, or it can search based on the title and abstract information of the document, greatly reducing the required computing power and processing time. This application does not impose any restrictions on this.

[0057] Third: There are no restrictions on the training methods for the second artificial intelligence.

[0058] Please see Figure 3 , Figure 3 A schematic diagram of the training method for the second artificial intelligence provided in this application for the document retrieval and content extraction method.

[0059] like Figure 3 As shown, in one possible implementation, training the second artificial intelligence includes: Construct a question-and-answer dataset for a specific domain using relevant literature; Among them, T i To combine the text of the document title and abstract, A i To correspond to the full text excerpt, y i Indicate whether the document mentions content related to the search requirements; Define the reward function: in, Rewards for correct answers Rewards for effective classification To ensure strict format rewards, For lenient reward format, Rewards for XML counting; The weights w1, w2, w3, w4, and w5 are determined by a grid search of the validation set. The second artificial intelligence is trained using a group relative policy optimization algorithm: in each iteration, from Sample small batches of data, and generate predictions using the model. And calculate the reward Using the objective function Update the strategy parameters; where, for The expectation operator below represents the averaging of trajectory samples generated by the strategy; t is the time step index in the generated sequence; Indicates the state Next, select an action (generate a token). The probability is calculated until the reward converges or reaches the preset performance index, thus obtaining the trained second artificial intelligence model.

[0060] Let's take literature in the field of ferroelectric materials as an example. First, we construct a high-quality, diverse question-answering dataset D_train for the topic of "ferroelectric materials". Each sample in the dataset is a triple (x_i, c_i, y_i), where x_i is the concatenated text of the title and abstract of a document, c_i is a full-text segment of the document that is related to "ferroelectric materials" (as a positive example) or an irrelevant segment (as a negative example), and y_i is a binary classification label (e.g., "yes" or "no"), indicating whether the document is related to the topic of "ferroelectric materials".

[0061] Next, a multi-dimensional reward function R is defined to guide the optimization direction of the model during reinforcement learning. This reward function comprehensively considers five aspects: R_correct: A positive reward is given when the model's predicted classification result matches the true label y_i.

[0062] R_validity: Rewards are given when the model's output is strictly "yes" or "no" to ensure the validity of the classification.

[0063] R_strict_format: When the model's output strictly conforms to the preset XML format (e.g., ... <think> ...< / think> <answer> ...< / answer> When this is done, a reward will be given.

[0064] R_loose_format: When the output contains necessary XML tags (such as...) <think>and <answer>However, when the structure is not entirely strict, a lower reward should be given.

[0065] R_xml_count: Based on specific XML tag pairs (e.g., ... <think> ...< / think> Rewards are given for the complete occurrence of a given event to encourage models to generate structurally complete thought processes.

[0066] The total reward function is The optimal values ​​for weights w1 to w5 are determined by performing a grid search on the validation set.

[0067] Please see Figure 4 , Figure 4 A schematic diagram comparing the training method of the second artificial intelligence for the document retrieval and content extraction method provided in this application with the training methods of existing technologies.

[0068] Those skilled in the art will understand that the specific method of "constructing a question-and-answer dataset for a specific domain using relevant literature" is not limited. For example... Figure 4 As shown, in one possible implementation, the question-answering dataset can rely on labels generated by a general-purpose large language model (such as Qwen3-235B). The general-purpose large language model extracts titles and summaries from the aforementioned relevant documents, inputs them into the large language model for automatic annotation, and generates a question-answering dataset containing classification results and inference paths, which is then used to train the second artificial intelligence model. Compared to existing technologies, this method not only avoids the high cost of traditional manual annotation but also ensures the quality of training data through the powerful inference capabilities of the large language model, enabling the second model to maintain high speed while possessing high selection accuracy.

[0069] In another possible implementation, the question-and-answer dataset can also be implemented by manually selecting questions and answers, and this application does not impose any restrictions on this.

[0070] Finally, the Group Relative Policy Optimization (GRPO) algorithm is used to train the artificial intelligence model. In each iteration, a mini-batch of data is sampled from the training set D_train. The model generates text containing the reasoning process and classification results based on the input x_i, and calculates the reward value according to the aforementioned reward function. The GRPO algorithm updates the model's policy parameters by maximizing the expected reward until the model performance converges on the validation set or reaches the preset performance index. Through this process, a second artificial intelligence model with high-precision judgment and interpretability in the field of "ferroelectric materials" is obtained.

[0071] For example, the second AI model can be trained based on the Qwen3-1.7B model, an open-source language model developed by Alibaba with 1.7 billion parameters. This model was chosen because it demonstrates powerful natural language understanding, logical reasoning, and instruction-following capabilities while maintaining low computational resource requirements, making it an ideal choice for fine-tuning specific domain tasks.

[0072] Therefore, this embodiment employs the Group Relative Policy Optimization (GRPO) reinforcement learning algorithm, which, through a rigorous reward mechanism, guides the model to learn how to focus on the key information most relevant to the user's query and suppress over-interpretation of irrelevant content. Compared to directly using a general pre-trained model, the Qwen3-1.7B model, fine-tuned by GRPO, not only significantly improves the accuracy in vertical domain (such as ferroelectric materials) literature screening tasks, but also greatly enhances the interpretability and reliability of the screening results by generating a structured reasoning process, thereby contributing to the efficient and accurate processing of the literature relevance prediction task in this embodiment.

[0073] After obtaining the second artificial intelligence model for the professional field, relevant literature can be screened again based on field relevance to obtain highly relevant literature. Specifically, this can include: The titles and abstracts of relevant literature are used to construct prompts for input into a second artificial intelligence model; Generate a complete output containing the reasoning path and classification results, and extract the classification results and reasoning path from it; Highly relevant literature was obtained.

[0074] Specifically, the trained second artificial intelligence model performs high-throughput screening of literature and outputs relevant literature in an interpretable manner; this includes: extracting the title and abstract of each relevant literature generated in the aforementioned steps, concatenating them into text, and inputting them into the trained second artificial intelligence model.

[0075] The second AI model performs in-depth analysis on each document and generates a structured text containing its internal "thinking" process and final judgment. For example: <think> The abstract mentions 'perovskite oxides' and 'ferroelectric properties', which are highly relevant to the search term 'ferroelectric materials'.< / think> <answer> yes< / answer> ".

[0076] This method automatically extracts from the model's output. <answer>The classification result within the label ("yes" or "no") and <think>The reasoning path within the tag. Finally, all screened documents form the final list of highly relevant documents.

[0077] Thus, the reasoning path is output synchronously during the screening process, making the decision-making process of the model transparent and traceable. This feature enhances the credibility of the automated screening results, provides a basis for users to judge, and improves the practicality of the system.

[0078] Similar to the first artificial intelligence model, the second artificial intelligence model here can also search directly on the full text of the document, or search based on the title and abstract information of the document, which will not be repeated here. The present application also does not make any limitation here.

[0079] Those skilled in the art can understand that the highly relevant documents need to be preprocessed to obtain cleaner data sources before further processing.

[0080] Exemplarily, the document retrieval and content extraction method further includes, before slicing and vectorizing the highly relevant documents: Overall layout recognition of the highly relevant documents to distinguish titles, paragraphs, charts, and formula regions; Using an OCR pipeline for scanned pages, and introducing syntax rules for correction in formula or chemical formula regions; Calling a table structure recognition module to extract cell content and coordinates to generate standardized metadata; Through hierarchical title detection and reading order reconstruction, integrating text, formula, and table data, and converting into hierarchical structured text.

[0081] In one example, this step can be implemented by: according to the DOI in the highly relevant document generated in the foregoing step, batch reading the corresponding document full text (usually in PDF format) from a pre-set local database.

[0082] It should be noted that the specific way of parsing is also not limited. In one possible implementation, the structured content extraction pipeline of the PDF file of the embodiment can rely on the mineru open source project. The core design of this project is to develop a multi-stage processing pipeline. This pipeline deeply combines the layout analysis technology of computer vision and traditional text content extraction. Compared with traditional rule-based or pure text stream-based PDF parsing methods, mineru greatly improves the accuracy and robustness of complex layout analysis of scientific literature by introducing visual dimension information. The embodiment can rely on it to efficiently and accurately convert the PDF full text of highly relevant literature into structured text files required for subsequent vectorization steps, ensuring the integrity of the original hierarchical structure of the literature (such as chapter titles, paragraph relationships) and key elements (such as tables, formulas), and laying a solid data foundation for subsequent processing of the present example. The specific processing flow includes: Layout analysis: First, use a computer vision-based layout analysis model to identify the overall layout of the PDF page, accurately dividing the title, paragraph text, chart, formula, and other different logical areas.

[0083] Content extraction and OCR: Content extraction is performed on the text area. For scanned PDFs or image format content, a multi-language OCR engine is called to perform character recognition. In particular, in the chemical formula area, molecular syntax rules are introduced to constrain and correct the OCR results, ensuring accurate identification of chemical bonds, subscripts, and ring structures.

[0084] Table structure recognition: Call the table structure recognition sub-engine to automatically detect the horizontal and vertical lines of the table, extract the content and row and column coordinates of each cell, and convert them into a standardized metadata format (such as JSON or XML).

[0085] Structured reconstruction: Reconstruct the chapter structure of the document through a hierarchical title detection algorithm, and combine the reading order model to fuse the extracted text, corrected chemical formula, and table metadata into a hierarchical, structurally complete Markdown file in the correct logical order.

[0086] Therefore, by converting unstructured literature into structured text that preserves the original layout and content elements, it is possible to provide fine processing for subsequent processing. This conversion ensures that complex content in the literature is fully preserved, laying a foundation for accurate information extraction.

[0087] After obtaining accurate structured text, the structured text of highly relevant literature can be sliced and vectorized.

[0088] Therefore, in one possible implementation, slicing and vectorizing highly relevant literature also includes: The structured text is preprocessed to remove the reference section; The text is divided into multiple slice units according to a preset length by using a recursive character segmentation method; An extended context window containing a predetermined number of characters before and after each slice unit is generated and stored in the metadata; The text content of each slice unit is encoded into a dense vector by using an embedding model and stored in the second vector database.

[0089] Specifically, the following method can be used: first, the text content of each Markdown file is preprocessed, and the reference (References) section at the end of the text is matched and removed by using a regular expression to eliminate irrelevant information interference.

[0090] Next, a recursive character text splitter (RecursiveCharacterTextSplitter) is used to set a block size of 2000 characters and zero overlap, and the cleaned text is divided into multiple slice units with as much semantic coherence as possible.

[0091] To enhance the context information of each slice unit, the embodiment further generates a "wider window" for each slice unit. Specifically, the position of each slice unit in the original text is located, and 1000 characters before and after it are expanded to form a text segment containing more rich context information, which is stored in the metadata of the slice unit.

[0092] Then, the original text content of each slice unit is encoded into a semantic vector V_chunk by using an embedding model again. The generated vector V_chunk and its corresponding metadata (including the source file DOI, the start and end index in the file, the extended context window text, etc.) are stored in a dedicated Milvus vector database. The database uses a dual retrieval architecture of "dense vector HNSW graph index + keyword retrieval inverted BM25" to support subsequent efficient hybrid retrieval.

[0093] Through this reasonable and precise text segmentation and context preservation strategy, the fine organization of literature content is realized. This method not only maintains the integrity of the semantics, but also provides convenience for precise positioning of information and optimizes the effect of subsequent retrieval.

[0094] In one possible implementation, based on the retrieval requirements, the most relevant N text slices are recalled from the second vector database, including: The retrieval requirements are encoded into a vector, and the vector index is used to obtain the vector similarity score of each slice; Meanwhile, the search demand is segmented, and the BM25 relevance score of each slice is obtained based on keyword search; The vector similarity score and the BM25 relevance score are weighted and fused to obtain a comprehensive relevance score; The comprehensive relevance score is sorted, and the top N slices are selected.

[0095] Specifically, when the user proposes a more specific knowledge extraction target T, for example, the query query is "The Curie temperature of ferroelectric materials", the method performs a hybrid search to recall the most relevant original text paragraphs.

[0096] Query encoding: using the same embedding model (jasper_en_vision_language_v1) as the aforementioned method, the query is encoded into a 1024-dimensional dense query vector.

[0097] Construct a hybrid search request: Dense vector: create an AnnSearchRequest, and perform an approximate nearest neighbor search on the encoded dense query vector and the vector field in the database. The search parameters are as follows: nprobe: 16.

[0098] BM25 search: submit the user's original query text query_text to Elasticsearch, and perform full-text search based on the BM25 algorithm on the stored slice unit original text field to obtain the relevance score.

[0099] Perform and rearrange: the dense similarity score returned by Milvus The BM25 score returned by Elasticsearch , and calculate the comprehensive score in a linear weighting manner: . Wherein α ∈ [0, 1] is a preset weight, and norm represents a normalization operation (such as Min-Max or Z-score).

[0100] Recall the most relevant paragraphs: from the rearranged results, select a predetermined number (for example, 600) of matching items. These matching items are the most relevant original text paragraph set to the user's extraction target, and are returned together with the source DOI for the final result output.

[0101] Finally, after obtaining the slices, the results can be extracted and output by the third artificial intelligence model. Those skilled in the art can understand that, based on the foregoing second artificial intelligence model, the screening of literature in the training field can be efficiently and accurately completed. In the stage of extracting and outputting the structure, the task target has changed from "judging relevance" to high-precision, structured information extraction from a small number of highly relevant literatures. This task requires stronger ability of context understanding, logical reasoning and following complex instructions, so a larger general large language model (such as Qwen3-235B, GPT-4, etc.) can be used to realize it, thereby avoiding the efficiency bottleneck and effect compromise brought by making a single model undertake two completely different tasks at the same time, and efficiently and accurately obtaining the results.

[0102] It should be clear that, as mentioned before, both the first artificial intelligence and the second artificial intelligence can make judgments based only on the title and abstract information of the literature, without accessing the full text. The distinguishing point of the third artificial intelligence is that the third artificial intelligence must access the full text content and provide detailed context information through text slicing to ensure the accuracy and completeness of content extraction.

[0103] Therefore, in a possible implementation, the third artificial intelligence model extracts and outputs the results related to the retrieval requirement based on the N text slices, which includes: Splicing the N text slices in the reading order, calling multiple third artificial intelligence models for parallel processing, and fusing multiple outputs through a third artificial intelligence model using a voting mechanism to output the results; Specifically, when the information is relatively scattered and needs to be integrated across paragraphs, text integration and prompt construction can be performed first: the most relevant paragraph set of the text slices recalled in the foregoing step is spliced according to its reading order in the original literature to form a single and coherent text block. Based on this integrated text, a standardized prompt (Prompt) is constructed, which clearly indicates the task target, for example: "Please extract 'Curie temperature of ferroelectric material' from this text, and output structured data containing'material chemical formula' and 'Curie temperature value (unit K)'. Then, multiple models can be processed in parallel, the same prompt and integrated text are distributed to multiple heterogeneous large language models for processing, and the API interfaces of each model are called simultaneously using asynchronous requests or multi-threading mechanisms to obtain their respective independent knowledge extraction results. Exemplarily, the large language models used here can include but are not limited to Qwen3-235B, Llama3-70B, GPT4 and Claude4, etc.

[0104] Finally, voting and result fusion can be performed to collect all structured or unstructured answers returned by the large language models. A voting and fusion process is initiated, where a large language model (e.g., Qwen3-235B) acts as a "summarizer," feeding the outputs of all models as new context input to the summarizer model and issuing the following instructions: "Please comprehensively analyze the extraction results provided by multiple models, vote according to the principle of majority rule, and reach a consensus conclusion on the Curie temperature of each material. Please standardize the chemical formulas and unify the temperature units, and finally output the valid information of the final decision in a precise JSON array format." This effectively filters out accidental errors or illusions from individual models, generating more confident structured knowledge.

[0105] In cases where the core information of a document is concentrated, the context is closely related, or an accurate judgment requires reading the entire text, the coordinates of N text slices in the original document can be used as indices to extract the complete original document as context. A structured extraction tool can be built and driven by a large language model to call the tool to extract information from the context and output the results.

[0106] Specifically, the first step is to construct the context: based on the highly relevant literature generated in the previous steps, locate the processed structured Markdown full-text file corresponding to each literature.

[0107] Next, the tool interface is defined: For example, one or more precise extraction tools can be defined programmatically using LangChain. For instance, for the Curie temperature extraction task, a tool named Curi_Extract can be defined, with strictly defined parameters such as: Extract_object (required to be a simplified linear chemical formula), extract_result (required to be a temperature value in Kelvin), and source_text (the original sentence from which the information originates). This tool definition provides structured instructions and pattern specifications for the output of the large language model.

[0108] Finally, a single call is made to extract the prompt containing the full-text Markdown content and send it to the third artificial intelligence model all at once.

[0109] It should be noted that either of the two methods described above can be used, or both methods can be used simultaneously when the nature of the literature or problem is unclear. Those skilled in the art can implement these methods according to the complexity of the actual extraction task, the distribution of literature paragraphs, and the limitations of computing resources; this application does not impose any restrictions here.

[0110] A second aspect of this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the document retrieval and content extraction method described above.

[0111] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.< / think> < / answer> < / answer> < / think>

Claims

1. A method for document retrieval and content extraction, characterized in that, The document retrieval and content extraction methods include: The documents are vectorized and stored in the first vector database; The search requirements are vectorized using the first artificial intelligence model, and relevant documents are initially screened from the vector database based on vector similarity. The relevant documents are further filtered based on domain relevance using a second artificial intelligence model to obtain highly relevant documents; the retrieval efficiency of the first artificial intelligence model is greater than that of the second artificial intelligence model; the second artificial intelligence model is trained based on a specific domain. The highly relevant documents are sliced ​​and vectorized to construct a second vector database; Based on the search requirements, the N most relevant text slices are retrieved from the second vector database; Based on the N text slices, the third artificial intelligence model extracts and outputs results related to the search requirements.

2. The document retrieval and content extraction method as described in claim 1, characterized in that, The step of vectorizing the documents and storing them in the first vector database includes: Using DOI as a unique identifier for each document, we extract the title and abstract fields to construct metadata. The metadata is processed for missing values, deduplication and merging, and field alignment and unified encoding conversion are completed; The processed metadata is then batch-vectorized and imported into the first vector database.

3. The document retrieval and content extraction method as described in claim 1, characterized in that, The process of vectorizing the search requirements using a first artificial intelligence model and initially filtering relevant documents from the vector database based on vector similarity includes: The output dimension of the first artificial intelligence model is fixed at D, and the user query Q and the document metadata D are encoded into semantic vectors of a unified dimension h. The first AI is deployed in a memory-resident manner on the accelerated computing unit, resulting in a single batch similarity calculation latency of <1000ms, and calculating the cosine similarity between the two. Sort by cosine similarity in descending order, and select the top k articles as the relevant articles.

4. The document retrieval and content extraction method as described in claim 1, characterized in that, The training of the second artificial intelligence includes: Construct a question-and-answer dataset for a specific domain using the relevant literature; in, Combine the text of the document title and abstract. To correspond to the full text excerpt, Indicate whether the document mentions content related to the search requirement; Define the reward function: in, Rewards for correct answers Rewards for effective classification To ensure strict format rewards, For lenient reward format, Rewards for XML counting; The weights w1, w2, w3, w4, and w5 are determined by a grid search of the validation set. The second artificial intelligence is trained using a group relative policy optimization algorithm: in each iteration, from Sample small batches of data, and generate predictions using the model. And calculate the reward Using the objective function Update the strategy parameters; where, for The expectation operator below represents the averaging of trajectory samples generated by the policy; t is the time step index in the generated sequence; Indicates the state Next, select an action (generate a token). The probability is calculated until the reward converges or reaches the preset performance index, thus obtaining the trained second artificial intelligence model.

5. The document retrieval and content extraction method as described in claim 1, characterized in that, The second artificial intelligence model is used to further filter the relevant literature based on domain relevance, resulting in highly relevant literature including: The titles and abstracts of the relevant documents are used to construct prompts for input into the second artificial intelligence model; Generate a complete output including the reasoning path and classification results. ,from Extracting classification results with reasoning path ; The highly relevant literature was obtained.

6. The document retrieval and content extraction method as described in claim 1, characterized in that, The document retrieval and content extraction method further includes, before slicing and vectorizing the highly relevant documents: The overall layout of the highly relevant literature is identified, distinguishing between titles, paragraphs, figures, and formula areas; Use the OCR pipeline for scanned pages and introduce syntax rules for correction in formula or chemical formula areas; Call the table structure recognition module to extract cell content and coordinates to generate standardized metadata; By detecting hierarchical headings and reconstructing the reading order, text, formulas, and table data are integrated and converted into hierarchical structured text.

7. The document retrieval and content extraction method as described in claim 6, characterized in that, The step of slicing and vectorizing the highly relevant documents also includes: The structured text is preprocessed to remove the reference section; A recursive character segmentation method is used to divide the text into multiple slice units of a preset length; For each slice unit, an extended context window containing a predetermined number of characters before and after it is generated and stored in metadata; An embedding model is used to encode the text content of each slice unit into a dense vector and then store it in the second vector database.

8. The document retrieval and content extraction method as described in claim 1, characterized in that, The step of retrieving the N most relevant text slices from the second vector data based on the retrieval requirement includes: The search requirements are encoded into vectors, and the vector similarity scores of each slice are obtained using vector indexes. Simultaneously, the search requirements are segmented into words, and the BM25 relevance scores of each slice are obtained based on keyword retrieval; The vector similarity score and the BM25 correlation score are weighted and fused to obtain a comprehensive correlation score; Sort the slices according to their comprehensive relevance scores and select the top N slices.

9. The document retrieval and content extraction method as described in claim 1, characterized in that, The step of extracting and outputting results related to the search requirements based on the N text slices using a third artificial intelligence model includes: The N text slices are concatenated in reading order, multiple third artificial intelligence models are called for parallel processing, and multiple outputs are fused through a voting mechanism by one of the third artificial intelligence models to output the result; And / or, using the coordinates of the N text slices in the original document as indices, extract the complete original document as context, construct a structured extraction tool and drive the large language model to call the tool to extract information from the context and output the results.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the document retrieval and content extraction method as described in any one of claims 1 to 9.

Images