A literature traceability question-and-answer system in the field of materials science and its construction method

Abstract

This invention provides a traceable question-answering system and its construction method for materials science literature, relating to the field of artificial intelligence technology. The system involves converting PDF documents into structured Markdown text; performing a two-stage block-segmentation operation on the Markdown text to obtain the block-segmentation results; performing domain-adaptive vector encoding to construct a domain-specific training dataset for materials science and training an embedding model; converting text blocks into vector representations based on the embedding model and the block-segmentation results; storing the vector representations in a database and establishing an approximate nearest neighbor index structure; acquiring user question information and performing multi-level retrieval optimization operations in the database to generate high-confidence fragments; concatenating the high-confidence fragments with the user question information to drive a domain-specific large language model to generate answers with precise citation tags. This addresses the problems of insufficient question-answering accuracy and poor traceability in existing traceable question-answering systems.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a document traceability question-and-answer system and its construction method in the field of materials science. Background Technology

[0002] In existing technologies, keyword-based literature retrieval systems (such as Web of Science and Scopus) provide basic literature search functions, but their search results often contain a large amount of irrelevant content and cannot directly answer professional questions. Intelligent question-answering systems, which have emerged in recent years, attempt to solve this problem, but generally suffer from insufficient professionalism and poor traceability. The special nature of materials science places higher demands on question-answering systems. First, this field contains a large number of technical terms and complex concepts, such as "dislocation density" and "phase transition temperature," requiring the system to have professional semantic understanding capabilities. Second, material properties are often closely related to parameters such as fabrication processes and microstructure, requiring the system to establish correlations between multiple parameters. Furthermore, researchers need to know the source literature of the answers for verification and in-depth research.

[0003] Current intelligent question-answering systems in the field of materials science generally suffer from insufficient accuracy and a lack of citation capabilities. While the MatKG system developed by the Chinese Academy of Sciences constructs a knowledge graph, its accuracy is limited by the graph's coverage and it cannot provide precise literature sources. MatChat AI from the Songshan Lake Materials Laboratory supports answer citation, but when dealing with complex material property prediction problems, the cited literature passages often lack strong relevance to the answer. NIST's Materials Data Curation System, relying on preset templates, severely restricts the accuracy and traceability of its answers. These systems generally suffer from misunderstandings of technical terms, inaccurate reasoning about complex relationships, and low matching rates between cited literature and generated answers, making it difficult for researchers to fully trust the answers provided by the systems and their citation evidence.

[0004] Therefore, developing a traceable question-and-answer system specifically for materials science literature has significant research value and application prospects. This system needs to overcome the limitations of existing technologies and undergo targeted optimizations in areas such as understanding technical terminology and tracing literature sources to truly meet the practical needs of materials researchers. Summary of the Invention

[0005] To address the issues of insufficient accuracy and poor traceability in traceable question-answering systems, this application provides a traceable question-answering system for materials science literature and its construction method.

[0006] Firstly, the material science literature traceability question-and-answer system provided in this application adopts the following technical solution:

[0007] A document traceability question-and-answer system in the field of materials science, comprising:

[0008] The document preprocessing module is used to convert PDF documents into structured Markdown text;

[0009] The text segmentation module is used to perform a two-stage segmentation operation based on the Markdown text to obtain the segmentation operation result;

[0010] The semantic encoding module is used to construct a domain training dataset corresponding to materials science, train the embedding model, and convert text blocks into vector representations based on the trained embedding model and the results of the block segmentation operation.

[0011] An index building module is used to store the vector representation in a database and establish an approximate nearest neighbor index structure based on clustering buckets;

[0012] The intelligent retrieval module is used to obtain user query information and perform multi-level retrieval optimization operations in the database to generate high-confidence fragments;

[0013] The source generation module is used to concatenate high-confidence fragments with user query information, driving the domain-wide language model to generate answers with precise citation tags.

[0014] Optionally, the document preprocessing module includes:

[0015] The OCR restoration unit is used to identify multimodal elements in a PDF using the MinerU parsing tool and convert the multimodal elements into a Markdown-compatible format.

[0016] The structural reconstruction unit is used to restore the logical hierarchy of the text through a reading order analysis algorithm and automatically remove noise data from headers and footers.

[0017] The semantic verification unit is used to verify the semantic integrity of the converted Markdown, ensuring that technical terms are not distorted.

[0018] Optionally, the two-stage segmentation in the text segmentation module includes a first stage and a second stage;

[0019] The first stage is based on the document structure as the main thread, and performs coarse-grained segmentation according to the heading level and adds metadata. The second stage performs fine-grained segmentation through sentence boundary recognition rules, and controls the maximum segment length and overlap length to maintain semantic continuity.

[0020] Optionally, the sentence boundary recognition rules include:

[0021] Set the maximum block length to The context overlap length is Divide the paragraph content into several sub-blocks It satisfies the following constraints:

[0022]

[0023] Among them, intersection This indicates the overlapping content shared between two adjacent text blocks. By default, the overlapping part is traced back to the beginning of a complete sentence to ensure that the semantic continuity is not broken.

[0024] Perform sentence segmentation on the text, assuming the original text is split into a set of sentences. Then, the parts are assembled according to the following conditions:

[0025]

[0026] in Indicates the first The length of each sentence;

[0027] When the length of the current block approaches the maximum length, a new text block is formed;

[0028] Assuming the current block contains Each sentence, based on a preset overlap ratio Calculate the number of overlapping sentences to be retained. The calculation formula is as follows:

[0029] .

[0030] Optionally, the semantic encoding module is also used to train a domain embedding model;

[0031] The BGE-m3 model was selected as the base embedding model, and a corpus was constructed from the target domain as the domain training dataset.

[0032] During training, a full fine-tuning approach is adopted to optimize all model parameters while maintaining the original structure of the BGE-m3 model.

[0033] The training objective is to maximize the vector consistency between semantically similar samples and minimize the similarity between irrelevant samples. The loss function is:

[0034]

[0035] in This is the sample currently being processed. As a positive example, For negative samples, Represents a text block Vector representation after encoding by the Embedding model This represents the cosine similarity.

[0036] Optionally, the index building module further includes: a metadata binder, a schema template, and a bucket clustering engine module;

[0037] The metadata binder is used to associate vectors with context data, which includes: a unique identifier, the original text content, the document ID, the paragraph sequence number, the group to which it belongs, and the task identifier.

[0038] The schema template contains a primary key field. Floating-point vector field Document identifier field of string type Block-level numbering With the original content field This is used to describe the position of a text block in the semantic space and structural context. All fields are uniformly transformed and written into the Milvus collection.

[0039] The bucket clustering engine uses IVF_FLAT indexes combined with floating-point vector fields to construct an approximate nearest neighbor index, with cosine distance as the metric.

[0040] Optionally, the traceability generation module includes:

[0041] A hallucination suppressor is used to remove invalid reference markers based on semantic relevance scores combined with a preset threshold.

[0042] A dynamic splicer is used to arrange high-confidence segments in descending order according to a sequential splicing mechanism;

[0043] The domain-wide model enables a reference marking mechanism and uses the Qwen2.5-7B-Instruct model to generate answers with reference sources, ensuring a 1:1 correspondence between the number and the input block.

[0044] Secondly, this application provides a method for constructing a traceable question-and-answer system for materials science literature, including:

[0045] Convert PDF documents into structured Markdown text;

[0046] Perform a two-stage chunking operation on the Markdown text to obtain the chunking operation result;

[0047] Construct a domain training dataset corresponding to materials science, train the embedding model, and convert text blocks into vector representations based on the trained embedding model and the results of the block segmentation operation.

[0048] The vector representation is stored in the database to establish an approximate nearest neighbor index structure based on clustering buckets;

[0049] Obtain user query information and perform multi-level search optimization operations in the database to generate high-confidence fragments;

[0050] By concatenating high-confidence fragments with user query information, the domain-specific large language model is driven to generate answers with precise citation tags.

[0051] Optionally, the step of converting PDF documents into structured Markdown text includes:

[0052] Use the MinerU parsing tool to identify multimodal elements in the PDF and convert the multimodal elements into a Markdown-compatible format;

[0053] The algorithm analyzes the reading order to restore the logical hierarchy of the text and automatically removes noise data from headers and footers.

[0054] Verify the semantic integrity of the converted Markdown to ensure that technical terms are not distorted.

[0055] Optionally, the step of performing a two-stage chunking operation based on the Markdown text to obtain the chunking operation result includes:

[0056] Based on the Markdown text, in the first stage, the document structure is used as the main thread, and coarse-grained segmentation is performed according to the heading level, with additional metadata added.

[0057] In the second stage, fine-grained segmentation is performed using sentence boundary recognition rules to control the maximum segment length and overlap length in order to maintain semantic continuity.

[0058] Get the results of the block operation.

[0059] In summary, this application provides a traceable question-answering system and its construction method for materials science literature, relating to the field of artificial intelligence technology. The system involves converting PDF documents into structured Markdown text; performing a two-stage block segmentation operation on the Markdown text to obtain the segmentation results; performing domain-adaptive vector encoding to construct a domain-specific training dataset for materials science and training an embedding model; converting text blocks into vector representations based on the embedding model and the block segmentation results; storing the vector representations in a database and establishing an approximate nearest neighbor index structure based on clustering buckets; acquiring user question information and performing multi-level retrieval optimization operations in the database to generate high-confidence fragments; concatenating the high-confidence fragments with the user question information to drive a domain-specific large language model to generate answers with precise citation tags. This addresses the problems of insufficient question-answering accuracy and poor traceability in existing traceable question-answering systems. Attached Figure Description

[0060] Figure 1 This is a structural block diagram of the first embodiment of the document traceability question-and-answer system in the field of this application.

[0061] Figure 2 This is a flowchart illustrating the first embodiment of the method for constructing traceable question-and-answer databases in the field of materials presented in this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] Reference Figure 1 , Figure 1 This is a structural block diagram of the first embodiment of the document traceability question-and-answer system in the field of this application.

[0064] like Figure 1 As shown, the materials science literature traceability question-and-answer system proposed in this application includes:

[0065] Document preprocessing module 10 is used to convert PDF documents into structured Markdown text;

[0066] Text segmentation module 20 is used to perform a two-stage segmentation operation based on the Markdown text to obtain the segmentation operation result;

[0067] The semantic encoding module 30 is used to construct a domain training dataset corresponding to materials science, train the embedding model, and convert text blocks into vector representations based on the trained embedding model and the results of the block segmentation operation.

[0068] The index building module 40 is used to store the vector representation in the database and establish an approximate nearest neighbor index structure based on clustering buckets;

[0069] The intelligent retrieval module 50 is used to obtain user query information and perform multi-level retrieval optimization operations in the database to generate high-confidence fragments;

[0070] The source generation module 60 is used to concatenate high-confidence fragments with user query information, driving the domain large language model to generate answers with precise citation tags.

[0071] It should be noted that the document preprocessing module includes: an OCR restoration unit, used to identify multimodal elements in the PDF using the MinerU parsing tool and convert the multimodal elements into a Markdown-compatible format; a structure reconstruction unit, used to restore the text's logical hierarchy through a reading order analysis algorithm and automatically remove header and footer noise data; and a semantic verification unit, used to verify the semantic integrity of the converted Markdown and ensure that technical terms are not distorted.

[0072] In its implementation, this embodiment presents a PDF document parsing scheme based on the MinerU parsing tool, used to convert complex and diverse research PDF documents into a standardized Markdown text format. This method possesses a high degree of document structure restoration capability, effectively identifying and extracting multimodal content such as text, images, tables, and formulas, while automatically removing irrelevant information such as headers, footers, and page numbers, ensuring accurate restoration of semantic continuity and reading order. This method maximizes the preservation of the original PDF's semantic hierarchy and multimodal information, providing high-quality input for subsequent content slicing, knowledge extraction, and large-scale model construction.

[0073] It is understood that the two-stage segmentation in the text segmentation module includes a first stage and a second stage; the first stage is based on the document structure as the main line, and performs coarse-grained segmentation according to the title level and adds metadata; the second stage performs fine-grained segmentation through sentence boundary recognition rules, and controls the maximum segment length and overlap length to maintain semantic continuity.

[0074] It should be noted that the sentence boundary recognition rule includes: setting the maximum block length to... The context overlap length is Divide the paragraph content into several sub-blocks It satisfies the following constraints:

[0075]

[0076] Among them, intersection This indicates the overlapping content shared between two adjacent text blocks. By default, the overlapping part is traced back to the beginning of a complete sentence to ensure that the semantic continuity is not broken.

[0077] Perform sentence segmentation on the text, assuming the original text is split into a set of sentences. Then, the parts are assembled according to the following conditions:

[0078]

[0079] in Indicates the first The length of each sentence;

[0080] When the length of the current block approaches the maximum length, a new text block is formed;

[0081] Assuming the current block contains Each sentence, based on a preset overlap ratio Calculate the number of overlapping sentences to be retained. The calculation formula is as follows:

[0082] .

[0083] Compared to traditional character truncation methods, this strategy achieves block-level concatenation on a sentence-by-sentence basis, effectively avoiding the problem of semantic fragmentation and improving the information carrying capacity of segments in downstream question-answering tasks.

[0084] In practical implementation, to achieve high-quality slicing of Markdown formatted text, this embodiment proposes a two-stage segmentation algorithm that combines structural information and semantic boundaries. First, based on the document structure as the main thread, the entire text is coarse-grainedly segmented according to the heading level. Then, fine-grained segmentation at the sentence level is further performed within each paragraph, achieving both maintaining contextual coherence and facilitating the construction of knowledge units for subsequent model processing.

[0085] Specifically, in the first stage, regular expressions are used to identify heading tags (such as "#", "##", "###") in the Markdown document, and the entire document is divided into several logical paragraphs based on their hierarchical relationship. Each paragraph is segmented using a high-level heading as an anchor point, and the corresponding heading content, hierarchical information, etc., are appended to the paragraph content in the form of metadata to ensure the integrity of the content structure and the traceability of the hierarchy.

[0086] After structural segmentation, the system enters the second stage, where sentence-level semantic segmentation is used to further process each paragraph. To adapt to the semantic sentence segmentation requirements in scenarios involving mixed Chinese and English, the system has constructed cross-language sentence boundary recognition rules based on regular expressions, capable of recognizing typical boundary patterns such as Chinese "。!?" and English "。!?" followed by uppercase letters or line endings.

[0087] It is understood that the semantic encoding module is also used to train the domain embedding model;

[0088] The BGE-m3 model was selected as the base embedding model, and a corresponding corpus was constructed from the field of materials science as the domain training dataset.

[0089] During training, a full fine-tuning approach is adopted to optimize all model parameters while maintaining the original structure of the BGE-m3 model.

[0090] The training objective is to maximize the vector consistency between semantically similar samples and minimize the similarity between irrelevant samples. The loss function is:

[0091]

[0092] in The sample currently being processed serves as the "anchor" for comparative learning. As a positive example, For negative samples, Represents a text block Vector representation after encoding by the Embedding model This represents the cosine similarity.

[0093] In practice, to achieve accurate semantic representation of professional documents, this embodiment designs and trains a dedicated Embedding model for the field of materials chemistry, which is used to semantically encode structured text blocks and provide a high-quality vector foundation for subsequent retrieval and generation tasks.

[0094] This embodiment uses the open-source BGE-m3 model from the Beijing Academy of Artificial Intelligence as the base embedding model, and performs domain-adaptive fine-tuning based on this model to improve the professional semantic representation capability. Specifically, a corpus containing approximately 15,000 high-quality scientific research documents is constructed from six sub-fields of materials science, including power electronic materials, two-dimensional materials, optoelectronic materials, high-entropy alloys, energy storage materials, and electrocatalysis, as the domain training dataset.

[0095] During training, a full fine-tuning approach is adopted to optimize all model parameters while maintaining the original structure of the BGE-m3 model, enabling it to deeply adapt to the domain semantics.

[0096] The trained Embedding model not only retains the general language understanding ability of the BGE-m3 model, but also significantly enhances its semantic modeling ability for terminology, technical expressions and knowledge structures in the field of materials chemistry through continuous learning of a large amount of professional corpus.

[0097] After obtaining the optimized domain embedding model, each segmented text block is encoded sequentially. Specifically, each text block... Input the trained Embedding model Generate its corresponding vector representation:

[0098]

[0099] All text block vectors constitute a vector set. It will be used for tasks such as vector retrieval and knowledge retrieval in the future.

[0100] It should be noted that the index building module also includes: a metadata binder, a schema template, and a bucket clustering engine module; the metadata binder is used to associate vectors with context data, and the context data includes: a unique identifier, the original text content, the document ID, the paragraph sequence number, the group to which it belongs, and the task identifier; the schema template contains a primary key field. Floating-point vector field Document identifier field of string type Block-level numbering With the original content field The field is used to describe the position of text blocks in semantic space and structural context. All fields are uniformly transformed and written into Milvus collection. The bucket clustering engine uses IVF_FLAT index combined with floating-point vector fields to build an approximate nearest neighbor index, with cosine distance as the metric.

[0101] In this specific implementation, the open-source vector database Milvus is used to store and index text block vectors encoded by the domain embedding model, aiming to achieve efficient semantic retrieval capabilities for the field of materials science.

[0102] For each vector obtained Structured encapsulation is performed, and before writing to the database, complete contextual metadata is attached to each vector sample, including a unique number, original text content, document ID, paragraph sequence number, group, and task identifier.

[0103] After the data writing is complete, based on the vector field An Approximate Nearest Neighbor (ANN) index is constructed using IVF_FLAT as the index type and cosine similarity as the distance metric. To balance recall accuracy and query performance, parameters are set during the index construction process. The high-dimensional vector space was divided into 128 clustering buckets, and a reverse file structure was used to quickly match vectors within each bucket. This resulted in a hierarchical and domain-adaptive semantic vector library, which not only possesses efficient vector similarity retrieval capabilities but also provides a low-latency, high-precision semantic invocation foundation for subsequent question-answering generation tasks.

[0104] It is understood that the source generation module includes: an illusion suppressor, used to remove invalid reference markers based on semantic relevance scores and a preset threshold; a dynamic splicer, used to arrange high-confidence fragments in descending order according to a sequential splicing mechanism; and a domain-wide model, which enables a reference marking mechanism and uses the Qwen2.5-7B-Instruct model to generate answers with reference sources, ensuring that the number corresponds 1:1 with the input block.

[0105] In practice, after the text blocks are stored in the database, users can ask any questions based on natural language, and the domain embedding model is used to encode the user's questions to generate a semantic representation consistent with the text block vector space.

[0106] Specifically, the question text entered by the user After undergoing the same preprocessing steps as the text blocks, the input domain embedding model is... Generate its corresponding vector representation. :

[0107]

[0108] Text block recall; generated query vectors Compared with all text block vectors in the Milvus vector database Similarity matching is performed. The system selects pairs based on cosine similarity. The vector that is closest to the query.

[0109]

[0110] in This represents the set of all vectors that have been stored in the database. This represents the normalized cosine similarity function.

[0111] Reranking and filtering: Considering that initial vector retrieval may introduce some semantically biased segments, this invention introduces the bge-reranker-m3 model, open-sourced by the Beijing Academy of Artificial Intelligence, as a Reranking component to improve the accuracy of the recall results. This model is based on a cross-coding architecture, capable of simultaneously inputting the question and candidate text blocks, and generating a discriminant score for their semantic relevance. ,in This represents the scoring function of the Reranker model. For the first A number of candidate text blocks.

[0112] All recall candidates according to The scores are sorted in descending order, and a threshold is set. After filtering out low-relevance segments, the remaining segments ultimately meet the following conditions. High confidence paragraph:

[0113]

[0114] in, This is the set of filtered high-confidence paragraphs.

[0115] In its implementation, after vectorizing the questions and recalling relevant knowledge fragments, this embodiment further conducts generative question answering tasks based on a large-scale language model. To meet the knowledge expression and citation needs of highly specialized fields such as materials science, a training-based alignment mechanism is used to perform full-scale fine-tuning of the large model to improve its answer accuracy, professionalism, and citation consistency.

[0116] Domain-wide model fine-tuning: During the construction of training data, the dataset used for fine-tuning the domain embedding model is consistently adopted as the source of the basic corpus. When generating training samples, pre-segmented and encoded text blocks are extracted from this corpus, organized using a structured prompt word format, and preliminary answers are generated through Deepseek-R1.

[0117] When generating answers, the model enables citation tagging by default, labeling the source of each citation in the output (e.g., "[citation:1]", "[citation:2]", etc.), and achieves fine-grained supervised training through sentence-paragraph alignment. To reduce the risk of "citation illusion" in large model generation, the bge-reranker-m3 model is introduced to calculate semantic relevance scores between all output sentences with citation tags and their corresponding reference segments. When the correlation score is below a preset threshold... When this happens, the system will determine the citation as invalid and remove the citation marker from the training data, retaining the main body of the sentence as a non-citation sentence for training, thereby constructing a more robust and less illusion-prone instruction alignment corpus. Finally, the Qwen2.5-7B-Instruct model is fully fine-tuned using the aforementioned high-quality question-answering data, enabling it to possess domain language understanding capabilities, citation accuracy control capabilities, and generation stability.

[0118] Generative responses; after fine-tuning, the system uses user questions and... The knowledge fragments are used to construct a context window, which is then input into the fine-tuned large model to generate domain-specific answer text with citations. The context window is constructed using a sequential concatenation mechanism, prioritizing the incorporation of reference fragments with higher scores. Each citation marker in the generated answer corresponds to a knowledge fragment number in the input, ensuring traceable semantic sources and verifiable citation behavior.

[0119] The final output of the system not only covers the professional terminology and logical structure of the materials field, but also has high interpretability and citation consistency, meeting the needs of highly reliable language generation in complex application scenarios such as scientific research, teaching and technical consulting.

[0120] This application provides a method for constructing a traceable question-and-answer system for materials science literature, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the method for constructing traceable question-and-answer databases in the field of materials presented in this application.

[0121] In this embodiment, the method for constructing a traceable question-and-answer system for materials science literature includes the following steps:

[0122] Step S10: Convert the PDF document into structured Markdown text.

[0123] Step S20: Perform a two-stage chunking operation based on the Markdown text to obtain the chunking operation result.

[0124] Step S30: Construct a domain training dataset corresponding to materials science, train the Embedding model, and convert the text blocks into vector representations based on the trained Embedding model and the results of the block segmentation operation.

[0125] Step S40: Store the vector representation in the database and establish an approximate nearest neighbor index structure based on clustering buckets.

[0126] Step S50: Obtain user query information and perform multi-level retrieval optimization operations in the database to generate high-confidence fragments;

[0127] Step S60: Concatenate the high-confidence fragment with the user's question information to drive the domain-wide language model to generate an answer with precise citation tags.

[0128] In practice, the steps of converting PDF documents into structured Markdown text include: using the MinerU parsing tool to identify multimodal elements in the PDF and converting the multimodal elements into a Markdown-compatible format; restoring the text's logical hierarchy through a reading order analysis algorithm and automatically removing header and footer noise data; and verifying the semantic integrity of the converted Markdown to ensure that technical terms are not distorted.

[0129] It should be noted that the step of performing a two-stage chunking operation based on the Markdown text to obtain the chunking operation result includes: in the first stage, based on the document structure as the main thread, performing coarse-grained segmentation according to the heading level and attaching metadata; in the second stage, performing fine-grained chunking through sentence boundary recognition rules, controlling the maximum chunk length and overlap length to maintain semantic continuity; and obtaining the chunking operation result.

[0130] This invention provides a traceable question-answering system and its construction method for materials science literature, relating to the field of artificial intelligence. The system involves converting PDF documents into structured Markdown text; performing a two-stage block-segmentation operation on the Markdown text to obtain the block-segmentation results; performing domain-adaptive vector encoding to construct a domain-specific training dataset for materials science and training an embedding model; converting text blocks into vector representations based on the embedding model and the block-segmentation results; storing the vector representations in a database and establishing an approximate nearest neighbor index structure based on clustering buckets; acquiring user question information and performing multi-level retrieval optimization operations in the database to generate high-confidence fragments; concatenating the high-confidence fragments with the user question information to drive a domain-specific large language model to generate answers with precise citation tags. This addresses the problems of insufficient question-answering accuracy and poor traceability in existing traceable question-answering systems.

[0131] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solution of this application. In specific applications, those skilled in the art can make settings as needed, and this application does not impose any restrictions on this.

[0132] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0133] In addition, for technical details not described in detail in this embodiment, please refer to the material science literature traceability question-and-answer method provided in any embodiment of this application, which will not be repeated here.

[0134] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0135] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application. The above are only preferred embodiments of this application and do not limit the patent scope of this application. All equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A document traceability question-and-answer system in the field of materials science, characterized in that, include: The document preprocessing module is used to convert PDF documents into structured Markdown text; The text segmentation module is used to perform a two-stage segmentation operation based on the Markdown text to obtain the segmentation operation result; The semantic encoding module is used to construct a domain training dataset corresponding to materials science, train the embedding model, and convert text blocks into vector representations based on the trained embedding model and the results of the block segmentation operation. An index building module is used to store the vector representation in a database and establish an approximate nearest neighbor index structure based on clustering buckets; The intelligent retrieval module is used to obtain user query information and perform multi-level retrieval optimization operations in the database to generate high-confidence fragments; The source generation module is used to concatenate high-confidence fragments with user question information, driving the domain-wide language model to generate answers with precise citation tags; The text segmentation module includes a first stage and a second stage in its two-stage segmentation. The first stage is based on the document structure as the main line, and performs coarse-grained segmentation according to the heading level and adds metadata. The second stage performs fine-grained segmentation through sentence boundary recognition rules, and controls the maximum segment length and overlap length to maintain semantic continuity. The sentence boundary recognition rules include: Set the maximum block length to The context overlap length is Divide the paragraph content into several sub-blocks It satisfies the following constraints: Among them, intersection This indicates the overlapping content shared between two adjacent text blocks. By default, the overlapping part is traced back to the beginning of a complete sentence to ensure that the semantic continuity is not broken. Perform sentence segmentation on the text, assuming the original text is split into a set of sentences. Then, the parts are assembled according to the following conditions: in Indicates the first The length of each sentence; When the length of the current block approaches the maximum length, a new text block is formed; Assuming the current block contains Each sentence, based on a preset overlap ratio Calculate the number of overlapping sentences to be retained. The calculation formula is as follows: ； The index building module further includes: a metadata binder, a schema template, and a bucket clustering engine module; The metadata binder is used to associate vectors with context data, which includes: a unique identifier, the original text content, the document ID, the paragraph sequence number, the group to which it belongs, and the task identifier. The schema template contains a primary key field. Floating-point vector field Document identifier field of string type Block-level numbering With the original content field This is used to describe the position of a text block in the semantic space and structural context. All fields are uniformly transformed and written into the Milvus collection. The bucket clustering engine uses IVF_FLAT indexes combined with floating-point vector fields to construct an approximate nearest neighbor index, with cosine distance as the metric.

2. The system according to claim 1, characterized in that, The document preprocessing module includes: The OCR restoration unit is used to identify multimodal elements in a PDF using the MinerU parsing tool and convert the multimodal elements into a Markdown-compatible format. The structural reconstruction unit is used to restore the logical hierarchy of the text through a reading order analysis algorithm and automatically remove noise data from headers and footers. The semantic verification unit is used to verify the semantic integrity of the converted Markdown, ensuring that technical terms are not distorted.

3. The system according to claim 1, characterized in that, The semantic encoding module is also used to train a domain embedding model; The BGE-m3 model was selected as the base embedding model, and a corresponding corpus was constructed from the field of materials science as the domain training dataset. During training, a full fine-tuning approach is adopted to optimize all model parameters while maintaining the original structure of the BGE-m3 model. The training objective is to maximize the vector consistency between semantically similar samples and minimize the similarity between irrelevant samples. The loss function is: in This is the sample currently being processed. As a positive example, For negative samples, Represents a text block Vector representation after encoding by the Embedding model This represents the cosine similarity.

4. The system according to claim 1, characterized in that, The source tracing generation module includes: A hallucination suppressor is used to remove invalid reference markers based on semantic relevance scores combined with a preset threshold. A dynamic splicer is used to arrange high-confidence segments in descending order according to a sequential splicing mechanism; The domain-wide model enables a reference marking mechanism and uses the Qwen2.5-7B-Instruct model to generate answers with reference sources, ensuring a 1:1 correspondence between the number and the input block.

5. A method for constructing a traceable question-and-answer system for materials science literature, characterized in that, include: Convert PDF documents into structured Markdown text; Perform a two-stage chunking operation on the Markdown text to obtain the chunking operation result; Construct a domain training dataset corresponding to materials science, train the embedding model, and convert text blocks into vector representations based on the trained embedding model and the results of the block segmentation operation. The vector representation is stored in the database to establish an approximate nearest neighbor index structure based on clustering buckets; Obtain user query information and perform multi-level search optimization operations in the database to generate high-confidence fragments; By concatenating high-confidence fragments with user query information, the domain-specific large language model is driven to generate answers with precise citation tags; The two-stage segmentation includes a first stage and a second stage; The first stage is based on the document structure as the main line, and performs coarse-grained segmentation according to the heading level and adds metadata. The second stage performs fine-grained segmentation through sentence boundary recognition rules, and controls the maximum segment length and overlap length to maintain semantic continuity. The sentence boundary recognition rules include: Set the maximum block length to The context overlap length is Divide the paragraph content into several sub-blocks It satisfies the following constraints: Among them, intersection This indicates the overlapping content shared between two adjacent text blocks. By default, the overlapping part is traced back to the beginning of a complete sentence to ensure that the semantic continuity is not broken. Perform sentence segmentation on the text, assuming the original text is split into a set of sentences. Then, the parts are assembled according to the following conditions: in Indicates the first The length of each sentence; When the length of the current block approaches the maximum length, a new text block is formed; Assuming the current block contains Each sentence, based on a preset overlap ratio Calculate the number of overlapping sentences to be retained. The calculation formula is as follows: ； The index construction method also includes: a metadata binder, a schema template, and a bucket clustering engine module; The metadata binder associates vectors with context data, which includes: a unique identifier, the original text content, the document ID, the paragraph sequence number, the group to which it belongs, and the task identifier. The schema template contains a primary key field. Floating-point vector field Document identifier field of string type Block-level numbering With the original content field Describe the position of the text block in the semantic space and structural context. All fields are uniformly transformed and written into the Milvus collection. The bucket clustering engine uses IVF_FLAT indexes combined with floating-point vector fields to construct an approximate nearest neighbor index, with cosine distance as the metric.

6. The method according to claim 5, characterized in that, The steps for converting PDF documents into structured Markdown text include: Use the MinerU parsing tool to identify multimodal elements in the PDF and convert the multimodal elements into a Markdown-compatible format; The algorithm analyzes the reading order to restore the logical hierarchy of the text and automatically removes noise data from headers and footers. Verify the semantic integrity of the converted Markdown to ensure that technical terms are not distorted.

Images