An improved context memory method for extracting engineering property data in project documents
By constructing an automated closed-loop update mechanism for refined example datasets and fidelity assessment, the problems of insufficient semantic understanding in engineering documents and high model tuning costs are solved, achieving high-accuracy extraction of engineering features and low-cost knowledge updates.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2026-04-09
- Publication Date
- 2026-07-10
AI Technical Summary
Existing technologies suffer from insufficient semantic understanding in engineering documents, high model tuning costs and catastrophic forgetting, and a lack of contextual relevance in few-shot learning, resulting in low extraction accuracy and difficulty in adapting to enterprise-level, real-time updated domain knowledge.
We construct a refined example dataset, transform feature variables into vectors through an embedding model, dynamically retrieve semantically similar examples as small sample contexts, and introduce an automated closed-loop update mechanism for fidelity evaluation to achieve low-cost continuous accumulation of domain knowledge and improve extraction accuracy.
It significantly improves the accuracy of feature extraction from engineering documents, reduces the illusion rate, enables low-cost model updates and secure data deployment, and adapts to enterprise-level real-time knowledge updates.
Smart Images
Figure CN122366633A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computation and natural language processing technology, and specifically relates to an information extraction method for engineering documents, which can be used in AIGC, context engineering, and large language model enhancement generation scenarios. Background Technology
[0002] In the field of enterprise engineering, accurately extracting engineering characteristic data from technical documents is a core competitive advantage. However, existing technologies have significant limitations:
[0003] (1) Insufficient semantic understanding: Traditional keyword-based sparse retrieval (such as BM25) can perform keyword matching, but it cannot identify variables with similar semantics but different expressions, making it difficult to understand the deeper meaning of characteristic variables.
[0004] (2) Model tuning costs and risks: Fine-tuning or full-parameter training of LLM for a specific domain not only requires massive amounts of high-quality domain data and extremely high computational costs, but also easily leads to "catastrophic forgetting" of the model.
[0005] (3) Although existing model improvement-based technical solutions attempt to enhance the model’s use of historical knowledge by introducing a memory module, they still require modification of the model architecture and retraining, and have the following problems: (a) The improved model still relies on historical knowledge in the pre-training stage, which is difficult to adapt to the real-time updated domain knowledge at the enterprise level; (b) There is a lack of quality screening mechanism for the generated results, and low-quality extraction results may contaminate the memory module and affect the accuracy of subsequent inference; (c) The continuous updating of the memory module is essentially still a parameter-level fine-tuning, which has the risk of catastrophic forgetting.
[0006] (4) Limitations of few-sample learning: Existing few-sample learning methods mostly use fixed examples and lack contextual relevance to the variable to be extracted. In complex engineering environments, if accurate examples and context cannot be provided, LLM is prone to "illusion" and the accuracy of data extraction will decrease. Summary of the Invention
[0007] The technical problem this invention aims to solve is to address the shortcomings of current engineering document feature extraction methods, such as the lack of specific context due to the use of fixed small sample examples, the susceptibility of models to illusions, and the high cost and catastrophic risk of forgetting due to frequent model tuning required for domain knowledge updates. This invention provides a context memory method for extracting information from engineering reports. It uses semantically similar refined examples as small sample contexts by dynamically retrieving them and introduces an automated closed-loop update mechanism based on fidelity assessment to achieve low-cost continuous accumulation of domain knowledge and improve the accuracy of large language model extraction.
[0008] The technical solution adopted by this invention to solve the above-mentioned technical problems is an improved contextual memory method for extracting engineering characteristic data from project documents, comprising:
[0009] (1) Construct an initial refinement example dataset, which consists of multiple data units, each containing engineering characteristic variable information and its corresponding refinement example;
[0010] (2) Extract the characteristic variables from the project report to be processed, use the embedding model to convert the characteristic variables into vectors, and retrieve similar data units in the refined example dataset in the vector space as dynamic small sample examples;
[0011] (3) Construct prompt words by combining the retrieved dynamic small sample examples with the current document context, and input them into the generative large language model to obtain the extraction results;
[0012] (4) Calculate the fidelity index of the output results of the generative large language model using the evaluation model;
[0013] (5) Based on the fidelity index, determine whether the current extraction result meets the storage conditions. If it does, convert it into a new data unit and store it in the refined example dataset.
[0014] Specifically, the Qwen2.5 model can be used for generative large language models.
[0015] Specifically, the structure of a Data Unit (DU) includes the engineering characteristic variable query, variable value, unit, data source document, and refine_example content for use with prompt words; refine_example includes the question, context, notes, and answer.
[0016] The beneficial effects of this invention are:
[0017] (1) Improved accuracy: By dynamically retrieving the refined example that is closest to the semantics of the current variable as a small sample context, a high-quality inference reference is provided for the large language model, which significantly reduces the illusion rate and the extraction accuracy can reach 91.59%.
[0018] (2) Low-cost continuous evolution: Without the need for frequent fine-tuning or retraining of the model, the domain knowledge can be accumulated and updated in real time through an automated closed loop of "extraction-evaluation-input", avoiding the risk of catastrophic forgetting.
[0019] (3) Secure and deployable: The solution supports local deployment. Both the vector database and the generative model can run in the internal environment without relying on the external network, which meets the enterprise's requirements for data security and privacy protection. Attached Figure Description
[0020] Figure 1 The present invention provides an overall architecture flowchart of a context memory system for extracting information from engineering reports. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the following description is provided in conjunction with the embodiments and... Figure 1 The present invention will be described in further detail below.
[0022] This system is divided into two stages: offline construction and online inference.
[0023] In the offline phase, the engineering characteristic variable query in the data unit DU extracted from the historical engineering report is vectorized using the embedding model to construct the Refine_Example vector dataset.
[0024] In the online phase, engineering characteristic variables in the documents to be processed are vectorized using the same embedding model, and semantically similar dynamic few-sample examples are retrieved from the Refine_Example vector dataset. The same embedding model is used in both phases to ensure consistency in the retrieval space.
[0025] I. Offline Phase
[0026] (1) Data Construction: Data Units (DUs) are constructed by extracting engineering characteristic variables from historical engineering report documents using engineering characteristic templates. An engineering characteristic template is a predefined list of engineering characteristic variables to be extracted, containing the names of all characteristic variables that need to be extracted from the engineering documents. For example, in the hydropower field, the characteristic variable names might be "rated head," "installed capacity," and "design flow." Each DU contains the following fields:
[0027] DU={"query": str, "value": str, "unit": str, "source": str, "refine_example": str};
[0028] Here, str indicates a string type. The meanings of each field are as follows: query is the project characteristic variable, value is the variable value, unit is the unit, source is the source document, and refine_example is the refinement example.
[0029] refine_example consists of the following four parts:
[0030] Question: Based on whether the query contains a specific query statement constructed from units;
[0031] Context: The original document text description related to the variable's value;
[0032] Note: Used to record special supplementary information related to variables;
[0033] Answer: Standard extraction results generated according to the preset format.
[0034] For multi-level engineering characteristic variables, when constructing the DU, the last-level engineering characteristic variable is used as the query content for embedding and encoding to maintain the focus of semantic retrieval. For example, for the three-level engineering characteristic variable "unit section / turbine / rated head", the last-level engineering characteristic variable "rated head" is used as the query content for embedding and encoding.
[0035] (2) Model fine-tuning and storage: The multilingual text embedding model BGE-M3 was selected as the embedding model. It was trained using supervised data from specific fields such as hydropower, and the text was mapped to a 1024-dimensional vector and stored in the open-source embedded vector database ChromaDB.
[0036] During the vector database storage phase, the query in DU is used as the retrieval index, and the remaining fields (value, unit, source, refine_example) constitute metadata for associated storage, which facilitates the quick retrieval of complete example information during retrieval.
[0037] II. Online Phase
[0038] (3) Prompt Construction: A search-enhanced generation (RAG) system is used, with new documents to be extracted and engineering feature templates as input. For each feature variable in the template, the text related to the current engineering feature variable query is retrieved from the document as context. The n1=10 most similar examples to the current variable are retrieved from the vector database, and the n2=3 optimal few-shot examples are further selected based on similarity. The retrieved dynamic few-shot examples contain complete refined examples extracted in the past (including questions, context, notes, and answers). When constructing user prompts, these examples are used as references for the few-shots, guiding the large language model to output the current result according to the format of the answer in the refined_example.
[0039] (4) LLM generation: The constructed system prompts (role settings) and user prompts (including context, n2 examples and the current question) are sent to the local Qwen2.5 model to obtain the feature value answer as the output result and enter the loyalty evaluation step.
[0040] (5) Fidelity Assessment: The question constructed based on the current engineering characteristic variables, the retrieved context, and the answer generated by the generative LLM are input into the assessment model to evaluate whether the answer is completely generated based on the context and without factual fabrication. In this embodiment, the assessment model uses the same Qwen2.5 model as the generative LLM, and guides it to score fidelity through prompt word engineering. The output score ranges from 0 to 1, where 1.0 indicates that the answer is completely faithful to the context, and 0 indicates that it is completely irrelevant or contains factual fabrication. A fidelity threshold Fn=1.0 is set, and the current extraction result is considered true and valid only when the score is 1.0.
[0041] (6) Feedback learning: If the fidelity score is 1.0, the current extraction result is determined to be completely faithful to the context. The system automatically encapsulates the context and result of this extraction in DU format and stores it in the Refine_Example dataset. If the score is lower than 1.0, the extraction result is determined to have factual bias and is not included in the memory set. It can be handed over to manual review or directly discarded.
[0042] The new DU stored in the Refine_Example dataset will be retrieved and utilized in the "dynamic few-sample retrieval" step of subsequent document processing, enabling real-time expansion and evolution of the memory set.
[0043] Overall, this system is built on the Refine_Example vector dataset, which consists of multiple data units (DUs). By vectorizing the target engineering characteristic variables, it achieves accurate retrieval of dynamic small sample examples with semantic associations from the dataset. The retrieved examples are collaboratively constructed with the context of the current engineering report to guide the local large language model to complete the Few-shot generation task. Simultaneously, the evaluation model is called to perform factual fidelity detection on the generated results. Only when the evaluation score reaches a preset threshold Fn, the current extracted results and context are automatically reconstructed into a new DU and sent back to the dataset, realizing real-time accumulation and memory update of domain knowledge.
[0044] Technical effectiveness verification:
[0045] To verify the effectiveness of the context memory method and system for extracting engineering report information described in this invention, this embodiment selects engineering technical documents in the field of water conservancy and hydropower as the test dataset, and conducts extraction experiments on engineering characteristic variables unique to this field.
[0046] (1) Experimental setup
[0047] The Qwen 2.5 series model was selected as the generative LLM and evaluation model. The BGE-M3 model, fine-tuned with 1500 data points from the hydropower sector, was used, with an output vector dimension of 1024. The comparison schemes are as follows:
[0048] Option A: Traditional keyword-based sparse retrieval (BM25).
[0049] Option B: Standard RAG retrieval (excluding Refine_Example dynamic small sample memory and closed-loop update).
[0050] Solution C: The method described in this invention (including dynamic Refine_Example retrieval and closed-loop update based on Faithfulness).
[0051] (2) Results and Analysis
[0052] The feature variable extraction accuracies of the three schemes were 62.5%, 78.2%, and 91.59%, respectively. This invention employs semantic retrieval based on 1024-dimensional vectors, which can accurately identify the deep semantics of engineering feature variables, solving the problem that traditional methods cannot handle semantically close descriptions. By introducing a fidelity score and setting a threshold Fn=1.0, this invention ensures that only extraction results that are completely faithful to the context are included in the Refine_Example dataset. Experiments show that after adding dynamic few-sample examples, the LLM-generated answers are more focused on the main text logic, significantly reducing non-factual output. As the number of processed documents increases, the Refine_Example dataset is continuously expanded through a closed-loop update mechanism. The system shows a significant upward trend in the success rate of extracting rare or complex multi-level feature variables.
[0053] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.
Claims
1. A contextual memory method for extracting engineering characteristic data from project documents, characterized in that, Includes the following steps: (1) Construct an initial refinement example dataset, which consists of multiple data units, each containing engineering characteristic variable information and its corresponding refinement example; (2) Extract the characteristic variables from the project report to be processed, use the embedding model to convert the characteristic variables into vectors, and retrieve similar data units in the refined example dataset in the vector space as dynamic small sample examples; (3) Construct prompt words by combining the retrieved dynamic small sample examples with the current document context, and input them into the generative large language model to obtain the extraction results; (4) Calculate the fidelity index of the output results of the generative large language model using the evaluation model; (5) Based on the fidelity index, determine whether the current extraction result meets the storage conditions. If it does, convert it into a new data unit and store it in the refined example dataset.
2. The method as described in claim 1, characterized in that, The structure of a Data Unit (DU) is defined as follows: DU={"query": str, "value": str, "unit": str, "source": str, "refine_example": str}; Where str represents the string type, query represents the project characteristic variable, value represents the variable value, unit represents the unit, source represents the data source document, and refine_example represents the refined example content used by the prompt words; The refine_example consists of the following four parts: Question: Based on whether the query contains a specific query statement constructed from units; Context: The original document text description related to the variable's value; Note: Used to record special supplementary information related to variables; Answer: Standard extraction results generated according to the preset format.
3. The method as described in claim 1, characterized in that, In step (2), if the characteristic variable is a multi-level engineering characteristic variable, then the last level variable is taken as the query content in the data unit for embedding encoding.
4. The method as described in claim 1, characterized in that, The embedding model is a deep learning model that has been supervised and fine-tuned for a specific engineering application domain, and the vector dimension output by the embedding model is 1024-dimensional.
5. The method as described in claim 1, characterized in that, During the vector database storage phase, the query in the data unit serves as the retrieval index, and the remaining fields of the data unit constitute metadata for associated storage. The metadata includes: {"value": str, "unit": str, "source": str, "refine_example": str}.
6. The method as described in claim 1, characterized in that, The specific method for calculating the fidelity index in step (4) is as follows: input the question constructed based on the current engineering characteristic variables, the retrieved context, and the answer generated by the generative large language model into the evaluation model to evaluate whether the answer is generated entirely based on the context and without any factual fabrication. The data entry criteria in step (5) are as follows: set the fidelity threshold Fn=1.0; when the evaluation score is equal to the threshold, determine that the current question and the context constitute a high-quality data pair, and update it to the refined example dataset after encapsulation according to the data unit format.
7. The method according to any one of claims 1 to 6, characterized in that, The method is based on a refined example vector dataset consisting of multiple data units. By vectorizing the target engineering characteristic variables, it achieves accurate retrieval of dynamic small sample examples with semantic associations from the dataset. The retrieved examples are collaboratively constructed with the current engineering report context to guide the local large language model to complete the small sample generation task. Simultaneously, the evaluation model is called to perform factual fidelity detection on the generated results. It is ensured that only when the evaluation score reaches a preset threshold Fn, the current extraction result and context are automatically reconstructed into new data units and sent back to the dataset. This achieves real-time accumulation and memory update of domain knowledge during the extraction process.