Retrieval enhancement generation method and data set generation method for time-sensitive problems

Abstract

The invention discloses a retrieval enhancement generation method for a time-sensitive problem, which comprises the following steps of: mixed time perception retrieval: enhancing document retrieval by adding time constraint on the basis of semantic relevance, and guiding by a time card to ensure that the retrieved document not only conforms to the meaning of query, but also conforms to the semantic relevance; the time context is met; the progressive multi-step reflection comprises the following steps of: firstly, acquiring and evaluating an initial document set by applying mixed time perception retrieval; if a document is retrieved, generating a final answer by using a large language model; otherwise, entering a reflection stage, and summarizing useful time information in the retrieved document into a context; and merging document sets accumulated in all iterations to generate a final answer. According to the method, a new framework integrating dynamic knowledge updating and time reasoning into the retrieval and generation process is provided, and accurate and timely response can be made to time-related problems.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically to the field of data retrieval technology, and particularly to a method for enhancing the generation of retrieval data and a method for generating datasets for time-sensitive problems. Background Technology

[0002] With the rapid development of the real world, the demand for models capable of handling dynamic information is increasing. However, Large Language Models (LLMs) remain inherently static, with their knowledge fixed at training time. Retrieval-Augmented Generation (RAG) has emerged to address this need, enabling LLMs to retrieve relevant documents from external sources to improve the accuracy of their responses. While existing RAG systems are effective for static knowledge retrieval, they face difficulties in handling time-sensitive queries due to fundamental limitations in retrieval, reasoning, and evaluation benchmarks.

[0003] Lack of Temporal Benchmarks: Despite recent progress, current RAG benchmarks primarily evaluate static knowledge retrieval and lack systematic evaluation of time-sensitive reasoning. While some datasets attempt to incorporate temporal elements, their temporal reasoning remains overly simplistic and fails to capture complex event dependencies. Furthermore, maintaining these benchmarks is costly and inefficient, as manually curated temporal questions quickly become outdated. This underscores the need to build automated benchmarks capable of dynamically generating and updating time-sensitive queries.

[0004] Retrieval Limitations: Traditional Retrieval Augmentation (RAG) models heavily rely on semantic similarity and keyword matching to retrieve relevant documents. While these methods are effective in static question answering, they often fail to capture the temporal constraints of queries, retrieving outdated or mismatched information. For example, a native RAG system, when answering a question about events in 2024, retrieved a document from 2021, resulting in a factual error in the output. Therefore, retrieval outputs often lack temporal consistency, reducing their reliability in time-sensitive tasks.

[0005] Limitations of Generation: Beyond retrieval, the generation process of RAG systems also faces challenges in handling complex multi-step temporal reasoning. Retrieving relevant documents does not guarantee success. The retrieved information may be temporally inconsistent or semantically insufficient, leading the model to fabricate missing details. Existing techniques, such as query rewriting and question decomposition, attempt to improve retrieval performance, but they are inherently static and cannot dynamically adjust the retrieval process across multiple inference steps. This necessitates a more adaptive iterative approach to ensure that the retrieved knowledge remains consistent with the query in both time and context. Summary of the Invention

[0006] To address these challenges, this application provides an automated benchmark for evaluating the temporal reasoning capabilities of RAG systems. Unlike previous datasets, this application systematically constructs multi-step, time-aware question-answer pairs by leveraging Large Language Models (LLMs) to generate simple single-hop time questions. These questions are then aggregated into complex multi-hop queries through structured composition (serial, parallel, and hybrid) to ensure scalability and adaptability to evolving real-world time challenges.

[0007] Building upon this foundation, this application proposes Chrono-O1, a novel framework for enhancing RAG by combining hybrid time-aware retrieval with Progressive Multi-Step Reflection (PMSR). Unlike traditional RAG methods that rely solely on semantic relevance, this application's hybrid retrieval mechanism incorporates temporal constraints, ensuring that retrieved documents are consistent with the query in both context and time. Furthermore, PMSR implements an iterative reasoning process in which dynamic interactions are retrieved and generated, enabling the model to optimize retrieval results, bridge information gaps, and iteratively construct temporally coherent responses. By embedding temporal reasoning at each stage, Chrono-O1 significantly improves the performance of time-sensitive queries by approximately 15%, setting a new benchmark for time-aware question answering.

[0008] To achieve the above objectives, the first aspect of this application discloses a method for enhancing the generation of time-sensitive questions, comprising the following steps:

[0009] Hybrid time-aware retrieval: This method enhances document retrieval by incorporating time constraints on top of semantic relevance, guided by time cards to ensure that retrieved documents conform to both the meaning of the query and its temporal context. The retrieval process first uses a restated query for semantic retrieval, generating a set of candidate documents. Then, time cards are used to reorder and filter these documents, evaluating their relevance to the time constraint t. ref The degree of fit; the final retrieved set D t The choice aims to maximize semantic relevance and temporal consistency;

[0010] Progressive multi-step reflection: First, apply hybrid time-aware retrieval to acquire and evaluate the initial document set D. t If the retrieved document is used, the final answer A is generated using a large language model. t Otherwise, proceed to the reflection phase, summarizing the useful time information from the retrieved documents into a context C; this context is used to optimize the query to Q. t ′ The identified information gaps are explicitly included, and then additional retrieval steps are performed using the optimized query to obtain a new document set D.t ′ Merge the retrieved documents with C, and repeat this process until the retrieved information meets the requirements or the predefined maximum number of iterations is reached.

[0011] Combine all the documents accumulated in the iterations to generate the final answer.

[0012] Preferably, the time card comprises:

[0013] Time constraints, including explicit timestamps and time ranges;

[0014] Time keywords extracted from the question;

[0015] To retrieve optimized, restated queries, semantic and temporal elements are integrated.

[0016] The method for generating the time card includes: for a given time problem Q t =(q sem ,t ref ), where q sem Represents semantic content, t ref To indicate time constraints, time cards are generated using a large language model.

[0017] Preferably, the hybrid time-aware retrieval step is specifically defined as follows:

[0018]

[0019] in It is a knowledge base; Retrieve and Filter represent semantic retrieval and time filtering, respectively. ref It is a time constraint, D t It is the final set retrieved.

[0020] Preferably, the initial document set D is evaluated based on two criteria. t :

[0021] Semantic sufficiency: Does the retrieved document provide enough information to answer the question?

[0022] Temporal consistency: Whether the retrieved documents meet the time constraint t ref ;

[0023] If the retrieved document meets both criteria, the final answer A is generated using a large language model. t ;

[0024] Optimized query Q′ t The definition is as follows:

[0025] Q′ t =f(q)sem ,t ref C)

[0026] Where f is the query optimization function.

[0027] Preferably, the final answer is A. t It is a comprehensive context used to generate answers, formed by merging the collection of documents accumulated throughout all iterations.

[0028]

[0029] in This represents the union of the document sets retrieved in k iterations.

[0030] The method for generating datasets for time-sensitive problems disclosed in the second aspect of this application includes the following steps:

[0031] S1. Source article preparation: Data is collected from news websites using web crawlers and preprocessed; the preprocessing includes cleaning and deduplication of the data to remove redundant or irrelevant content; the data is summarized using a large language model, and the time and fact elements in the data are extracted into paragraphs, which constitute the basic input for question generation;

[0032] S2. Single-time question-answer generation: Systematically generate time-based question-answer pairs from the processed source text using a large language model;

[0033] S3 Multi-Time Question-Answer Combination: Extends single-document question-answer pairs into more complex multi-document question-answer pairs through parallel, serial, and hybrid methods.

[0034] Preferably, in the single-time question-answer generation step, each paragraph containing time information serves as the basis for generating independent question-answer pairs; given a source paragraph P, the large language model is prompted to generate a set of time-based question-answer pairs. Q i Let A represent the i-th question. i This indicates the corresponding answer; to improve diversity and relevance, different prompt templates were developed for different time-related question types. After generation, the question-answer pairs were filtered to remove duplicates and ensure uniqueness.

[0035] Preferably, in parallel processing, the sub-questions are logically independent but together constitute the necessary conditions for answering the main question; each sub-question provides unique information, and only by solving all sub-questions can a complete answer be obtained; in order to construct parallel questions programmatically, semantically aligned individual question-answer pairs are identified and aggregated into a unified query; the model summarizes the independent answers to comprehensively answer the combined question.

[0036] In serial questions, sub-questions are interdependent, forming a sequential chain of reasoning, with the answer to one sub-question serving as the input or context for the next. Serial question combinations are constructed by identifying shared temporal or event-based references between question-answer pairs, which act as bridges connecting the sub-questions.

[0037] Hybrid problem combinations combine parallel and serial patterns, and include both independent subproblems and interdependent subproblems.

[0038] Preferably, to construct a hybrid problem combination, first create parallel and series problem combinations separately, and then combine them;

[0039] Combinatorial problem Q c Represented as subproblems Q1, Q2, ..., Q n Combinations:

[0040] Q c =g(Q1,Q2,…,Q) n ),

[0041] Where g is a combination function used to encode the relationships between subproblems;

[0042] For parallel problems, g is the union of independent problems:

[0043] Q c =Q1∪Q2∪…∪Q n

[0044] For the chaining problem, g is a chain of sequential dependencies:

[0045] Q c =Q1→Q2→…→Q n

[0046] For mixed problems, g combines two operations:

[0047] Q c =Q1→(Q2∪…∪Q n ).

[0048] In summary, the beneficial effects of this application are as follows:

[0049] This application introduces a benchmark dataset for evaluating time-sensitive RAG systems, created through a novel automated process. ChronoQA contains a variety of time queries from dynamic knowledge domains, providing a foundational resource for evaluating the time reasoning capabilities of RAG models.

[0050] This application proposes a novel framework that integrates dynamic knowledge updates and temporal reasoning into the retrieval and generation process, enabling accurate and timely responses to time-related issues. Attached Figure Description

[0051] Figure 1 A framework diagram of this application. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0053] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0054] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0055] The technical solutions provided in this application involve technologies such as machine learning and natural language processing in artificial intelligence, which are specifically described and illustrated through the following embodiments.

[0056] Before introducing the embodiments of this application, some terms involved in this application will be explained.

[0057] 1. Retrieval Augmentation (RAG) Benchmarks: Retrieval Augmentation (RAG) datasets play a crucial role in evaluating retrieval augmentation methods, but most existing benchmarks focus on static knowledge retrieval and lack systematic approaches to temporal reasoning. Early question-answering datasets, such as Natural Questions (NQ), TriviaQA, and MS MARCO, primarily evaluate open-domain retrieval, relying on web documents and knowledge graphs. More advanced retrieval augmentation benchmarks, such as HotpotQA, introduce multi-hop retrieval, requiring models to synthesize information from multiple sources. However, these datasets assume knowledge is static, ignoring how answers change over time, a key limitation for time-sensitive applications. To bridge this gap, recent work has focused on time-aware benchmarks. FreshQA evaluates whether models can retrieve the most temporally relevant evidence, while Context-Aware Retrieval Augmentation Datasets (CRAG) and Domain Retrieval Augmentation Datasets (DomainRAG) incorporate time-sensitive document updates. Nevertheless, these datasets remain limited in complexity and lack multi-hop reasoning on event sequences and implicit temporal relationships.

[0058] 2. Retrieval Augmentation (RAG) Approaches: Retrieval Augmentation (RAG) frameworks enhance Large Language Models (LLMs) by introducing external knowledge retrieval, improving factual accuracy and reducing illusions. Early methods like REALM employed dual-encoder retrieval, dynamically retrieving knowledge during answer generation. Methods such as Fusion-in-Decoder improve retrieval augmentation by directly integrating retrieved documents into the input sequence, enabling the model to jointly process these documents. Self-Retrieval Augmentation (Self-RAG) further improves this by introducing a self-reflective mechanism, iteratively optimizing the retrieval process. However, a key challenge of time-sensitive retrieval augmentation is that external knowledge is often fragmented and outdated. Recent methods attempt to systematize retrieval through structured representations. Graph Retrieval Augmentation (GraphRAG) constructs knowledge graphs, providing a global perspective for retrieval and improving response quality. Furthermore, search agents utilizing Chain-of-Thought (CoT) reasoning improve the reasoning of large language models for structured knowledge during inference.

[0059] In one embodiment, this application proposes ChronoQA, an automated benchmark that dynamically generates multi-step time-related questions by aggregating various real-world knowledge sources. The dataset in this application explicitly models temporal reasoning dependencies to ensure reliable evaluation of time-sensitive retrieval and generation.

[0060] In one embodiment, this application proposes Chrono-O1, which directly integrates time-aware retrieval and iterative multi-step reflection to ensure that the retrieved documents not only match semantically but also are temporally consistent with the query.

[0061] Time-related questions Q t A time-related question is a natural language query whose answer must be based not only on semantic relevance but also on a specific temporal context. Unlike questions based on static facts, time-related questions require reasoning about time-sensitive information, and the validity or relevance of the answer depends on time constraints. Formally, a time-related question is defined as:

[0062] Q t =(q sem ,t ref ,t query ), (1)

[0063] Where q sem This represents the semantic content of the question, including factual or conceptual queries. ref This indicates a time reference, specifying a time limit relevant to the question. This limit can be explicit, directly mentioning a specific date or time period (e.g., "What happened in 2023?"), or implicit, requiring inference from the context to determine the relevant time range (e.g., "What is the current status of the program?"). Furthermore, t query This indicates the query time, referring to the moment the question was asked.

[0064] Time-Sensitive Retrieval Augmentation (TS-RAG) is a retrieval augmentation system specifically designed to address time-sensitive question-answering challenges. Unlike traditional retrieval augmentation systems that prioritize semantic relevance, TS-RAG integrates temporal reasoning to ensure that the retrieved information matches the time sensitivity of the query.

[0065] Given a time-related problem Q t =(q sem ,t ref ), TS-RAG will be based on the document {d1,d2,…,d k} and query q sem The semantic relevance and their relationship with reference time t ref The time alignment is used to retrieve relevant documents. After the retrieval is complete, TS-RAG incorporates time reasoning into the answer generation process, using the query, retrieved documents, and time references to generate a semantically accurate and temporally up-to-date answer.

[0066] A t=Generate(q sem ,{d1,d2,…,d k},t ref (2)

[0067] This ensures that the answers are not only relevant in content, but also reflect the latest time context.

[0068] In one embodiment, the ChronoQA dataset sets a new standard for enhanced time retrieval datasets by combining diverse time reasoning tasks, multi-document support, and an automated construction process. Figure 1 As shown, the construction process of this dataset mainly consists of three steps: source article preparation, time-related question generation, and verification.

[0069] S1. Source Article Preparation

[0070] To generate timely questions, a rapidly evolving data source containing explicit time information is needed. News articles, due to their high update frequency and built-in timestamps, are well-suited for this purpose. This application used a web crawler to collect articles from well-known news websites from January 1, 2019 to August 30, 2024. An average of 171.8 articles were collected per day, with an initial dataset of 350,000 articles. Given the large number of original articles and the presence of noise, preprocessing was necessary to ensure data quality. First, the articles were cleaned and deduplicated to remove redundant or irrelevant content. Next, Large Language Models (LLMs) were used to summarize each article, extracting its key time and factual elements into concise paragraphs. These summaries, referred to as "dense time paragraphs," constituted the basic input for question generation. A total of 294,696 dense time paragraphs were generated, providing a high-quality and manageable source for further processing.

[0071] Table 1: Representative examples of (time-based questions and answers).

[0072]

[0073] S2. Single-time question and answer generation

[0074] This application utilizes Large Language Models (LLMs) to systematically generate time-based question-and-answer pairs from processed source text. Each paragraph rich in time information serves as the basis for generating independent question-and-answer pairs. Given a source paragraph P, the LLM is prompted to generate a set of time-based question-and-answer pairs. Q i Let A represent the i-th question. iThis indicates the corresponding answer. To improve diversity and relevance, this application developed different prompt templates for different types of time-related questions, such as queries based on explicit time (e.g., "When did event X occur?") and queries related to implicit time (e.g., "What event happened before event Y?"). After generation, the question-answer pairs are filtered to remove duplicates and ensure uniqueness. This step resulted in a library containing over 10,000 high-quality time-related question-answer pairs.

[0075] S3 Multi-Time Question and Answer Combination

[0076] To evaluate the model's ability to handle multi-document reasoning, single-document question-answering pairs were extended to more complex multi-document question-answering pairs. These questions require aggregating information from multiple documents to produce coherent and accurate answers. This combination was divided into three modes: parallel, serial, and hybrid, each presenting a different reasoning challenge.

[0077] In parallel processing, sub-problems are logically independent, but together they constitute the necessary conditions for answering the main question. Each sub-problem provides unique information, and all sub-problems must be solved to arrive at the complete answer. For example, the question "What will the weather be like in location A, location B, and location C on September 15, 2023?" can be broken down into three independent sub-problems:

[0078] (1) "What will the weather be like in location A on September 15, 2023?"

[0079] (2) "What will the weather be like in location B on September 15, 2023?"

[0080] (3) "What will the weather be like in location C on September 15, 2023?"

[0081] To programmatically construct parallel questions, identify semantically aligned individual question-answer pairs (e.g., based on topic or temporal proximity) and aggregate them into a unified query. The model must summarize the independent answers to comprehensively address the combined question.

[0082] In a chain problem, subproblems are interdependent, forming a sequential reasoning chain, where the answer to one subproblem serves as the input or context for the next subproblem.

[0083] Series circuits are constructed by identifying shared temporal or event-based references between question-answer pairs, which act as bridges connecting sub-questions. This approach challenges models to perform sequential reasoning, requiring intermediate answers to be used in subsequent queries.

[0084] Hybrid problem combinations combine parallel and series patterns to form more complex hybrid structures. These problems contain both independent subproblems and interdependent subproblems. They integrate:

[0085] Parallel subproblems and serial dependencies.

[0086] To construct hybrid problem combinations, first create parallel and serial problem combinations separately, and then combine them. This reflects the complexity of the real world, where multiple reasoning strategies often intertwine.

[0087] To formalize the generation process, the combinatorial problem Q is... c Represented as subproblems Q1, Q2, ..., Q n Combinations:

[0088] Q c =g(Q1,Q2,…,Q) n (3)

[0089] Where g is a combination function, which uses a large model (LLM) to generate a combination problem based on information, and is used to encode the relationships between subproblems.

[0090] For parallel problems, g is the union of independent problems:

[0091] Q c =Q1∪Q2∪…∪Q n (4)

[0092] Table 2: Comparison of Temporal Retrieval Augmentation (TAG) datasets.

[0093]

[0094] Table 3: Statistical data on the problem categories in this application.

[0095]

[0096]

[0097] For the chaining problem, g is a chain of sequential dependencies:

[0098] Q c =Q1→Q2→…→Q n (5)

[0099] For mixed problems, g combines two operations:

[0100] Q c =Q1→(Q2∪…∪Q n (6)

[0101] By employing these three strategies, a diverse and challenging set of multi-document question-answering pairs was created. These questions tested the model's ability to aggregate independent information, perform sequential reasoning, and handle mixed reasoning tasks. This diversity ensures that the dataset serves as a reliable benchmark for evaluating temporal multi-document reasoning.

[0102] To ensure the quality of the ChronoQA dataset, this application implemented a multi-step validation process combining rule-based filtering, Large Language Model (LLM) evaluation, and human validation. Rule-based filtering validated structural and logical consistency, such as ensuring that multi-document questions cited at least two documents. The LLM evaluated fluency, temporal relevance, and semantic coherence, filtering out poorly constructed or inconsistent question-answer pairs. Finally, human evaluation of approximately 6000 samples confirmed that over 95% of the samples met the quality standards, validating the effectiveness of the process. This rigorous process ensures that ChronoQA is a comprehensive and reliable dataset suitable for evaluating and benchmarking models for time-sensitive tasks.

[0103] This application's ChronoQA provides comprehensive coverage of temporal reasoning challenges, as shown in Table 3. Compared to existing datasets (Table 2), ChronoQA demonstrates significant advantages in scale, diversity, and temporal complexity. Unlike previous datasets that were typically limited to static or direct temporal reasoning, ChronoQA integrates multiple time types (absolute, aggregate, and relative) and time representations (explicit and implicit). Furthermore, it emphasizes multi-document reasoning, with a large portion of the problems requiring the synthesis of information from multiple sources, filling a key gap in benchmarks like TriviaQA and FreshQA. Its automated build process further ensures scalability and timeliness, overcoming the limitations of manual maintenance in existing benchmarks.

[0104] In one embodiment, to address the challenges posed by time-sensitive question answering, this application proposes the time-sensitive retrieval augmentation generation (RAG) framework Chrono-O1, whose components and integration methods are as follows:

[0105] The Chrono-O1 framework consists of two main modules: Hybrid Time-Aware Retrieval and Progressive Multi-Step Reflection (PMSR). The hybrid retrieval module enhances document retrieval by adding time constraints to semantic relevance, ensuring that retrieved documents conform to both the meaning of the query and its temporal context. PMSR complements this by introducing an iterative reasoning mechanism that optimizes the retrieval and reasoning process to effectively handle multi-hop and temporally complex queries.

[0106] Traditional retrieval augmentation (RAG) systems prioritize semantic relevance but often neglect time constraints that are crucial for timeliness. To overcome this limitation, in one embodiment, this application employs a hybrid retrieval method that combines semantic retrieval with time filtering and is guided by a temporal card.

[0107] Time card generation: For a given time problem Q t =(q sem ,t ref ), where q sem Represents semantic content, t ref To represent time constraints, time cards are generated using a Large Language Model (LLM). A time card contains:

[0108] Time constraints, including explicit timestamps and time ranges.

[0109] Time keywords extracted from the question.

[0110] To retrieve optimized, restated queries, semantic and temporal elements are integrated.

[0111] Hybrid retrieval process. The retrieval process first performs semantic retrieval using a restated query, generating a set of candidate documents D = {d1, d2, ..., d...}. n Then, these documents are reordered and filtered using time cards, which evaluate their relevance to the time constraint t. ref The degree of fit. The final retrieved set D t The choice aims to maximize semantic relevance and temporal consistency. Formally, the retrieval process is defined as follows:

[0112]

[0113] in It is a knowledge base, and Retrieve and Filter represent semantic retrieval and time filtering, respectively.

[0114] In one embodiment, to address the limitations of single-step retrieval, particularly for multi-hop and time-complex queries, this application introduces a Progressive Multi-Step Reflection (PMSR) mechanism. PMSR integrates temporal reasoning into the iterative retrieval and reasoning process, ensuring that information gaps are gradually filled while maintaining temporal consistency. The iterative reasoning process begins with PMSR applying hybrid time-aware retrieval to obtain an initial document set D. t These documents were evaluated based on two criteria:

[0115] Semantic sufficiency: Does the retrieved document provide enough information to answer the question?

[0116] Temporal consistency: Whether the retrieved documents meet the time constraint t ref .

[0117] If the retrieved document meets both criteria, the system will use a Large Language Model (LLM) to generate the final answer A. t Otherwise, PMSR enters the reflection phase, where it analyzes the retrieved documents to identify missing information and optimizes the query accordingly.

[0118] Query optimization and iterative retrieval: During the replay process, the system summarizes useful time information from the retrieved documents into a context C. This context is used to optimize the query to Q. t ′ Clearly include identified information gaps:

[0119] Q′ t =f(q) sem ,t ref ,C) (8)

[0120] Here, f is a query optimization function, which uses a Large LLM (LLM) to generate an optimized query based on the information. Then, additional retrieval steps are performed using the optimized query to obtain a new document set D. t ′ The retrieved documents are merged with C, and this process is repeated until sufficient information is retrieved, or the predefined maximum number of iterations N is reached.

[0121] Final answer generation: Merge the collection of documents accumulated from all iterations to form a comprehensive context for generating the answer.

[0122]

[0123] in This represents the union of the document sets retrieved in k iterations.

[0124] The retrieval performance metrics used in this application include recall, mean average precision (MAP), and normalized depreciation cumulative gain (NDCG) across overall, absolute, aggregate, relative, multiple, and single categories. The model used in this application is highlighted in blue, and the best results are shown in bold. Among the baseline models, the best-performing model is marked in orange. This application combines hybrid time-aware retrieval with PMSR (Probabilistic Multi-Step Reasoning), offering numerous advantages. By explicitly defining inference time constraints and iteratively optimizing queries, this application effectively handles multi-hop and time-complex queries. For simpler queries, the system can complete processing in one step, avoiding unnecessary computational overhead. This adaptability ensures a balance between efficiency and robustness, setting a new benchmark for time-sensitive question answering.

[0125] To evaluate the validity of this application, experiments were conducted in a controlled retrieval environment using a locally constructed database containing 300,000 documents. These documents were encoded and stored in a Chroma vector database. This setup ensured the reproducibility of the experimental results.

[0126] This application was evaluated against several baseline models to ensure a comprehensive comparison: Direct LLM Answering, where the large language model answers the question without retrieval augmentation; Native RAG, which retrieves documents based on the original query without time adjustment; Temporal Filter, which applies time filtering to the retrieved documents based on time constraints extracted from the large language model; Query Rewrite, which reformulates the query with time constraints before retrieval and answering; Query Decomposition, which breaks the query down into subquestions, retrieves answers for each subquestion, and synthesizes them into the final response; and Iterate RAG, which employs an iterative retrieval process where the system evaluates whether the initially retrieved documents are sufficient to answer the query. If not, it optimizes the query and performs additional retrieval iterations. These baseline models cover a range of retrieval and inference strategies, providing a robust evaluation framework.

[0127] This application evaluates model performance across two key dimensions: retrieval quality and end-to-end answer accuracy. For retrieval, recall, mean average precision (MAP), and normalized discounted cumulative gain (NDCG) are used to assess the relevance and ranking of retrieved documents. For end-to-end evaluation, answer correctness is measured and automated using GPT-4o. A manual review of 600 randomly selected evaluation results confirmed an error rate below 1%, ensuring the reliability of this approach. This evaluation comprehensively measures the effectiveness of each model in handling time-sensitive issues.

[0128] This application's hybrid temporal retrieval method achieves state-of-the-art performance in recall and ranking metrics, outperforming traditional retrieval-augmented generation (RAG) methods such as native RAG, query rewriting, and query decomposition. It is worth noting that temporal filtering methods that apply time constraints to retrieved documents face difficulties in multi-hop inference. By utilizing hybrid temporal-aware retrieval, this application ensures that retrieved documents are temporally relevant and semantically rich.

[0129] Among baseline methods, query decomposition and iterative RAG perform relatively well because they can decompose complex queries or iteratively optimize retrieval. However, these methods still face challenges in multi-document synthesis and inference consistency. While time-based filtering methods are effective at removing irrelevant documents, they lack the adaptive optimization capabilities required to handle complex queries. This application demonstrates a significant advantage in handling complex time-based queries, particularly achieving the highest accuracy in multi-hop inference. The performance improvement is attributed to Progressive Multi-Step Retrieval (PMSR), which uses iterative optimization based on time-based inference constraints to ensure a high degree of match between retrieved evidence and the query's time frame.

[0130] Increasing the maximum number of inference iterations in this application (from a maximum of 2 to a maximum of 4) progressively improves accuracy. At a maximum of 2 iterations, this application already outperforms most baseline models, and further iterations improve the correctness of the answers, particularly for implicit and multi-hop inference tasks. However, the improvement diminishes beyond the maximum of 4 iterations, suggesting that excessive iterations may introduce redundancy without significantly improving accuracy.

[0131] The results show that Progressive Multi-Step Reflection (PMSR) excels in refining and enhancing retrieved information through iterative reflection, while Temporal Cards play a fundamental role in ensuring the semantic and temporal relevance of the initial retrieval.

[0132] In experiments, the computational efficiency of this application was evaluated. Its hybrid retrieval introduces only one additional Large Language Model (LLM) call for generating time cards, while time reordering is applied to the narrowed candidate set after retrieval, significantly improving time relevance while incurring minimal overhead (~10%). During generation, PMSR dynamically adapts to query complexity, using one LLM call for reflection, selection, and refinement in each iteration. The subset processed in each iteration gradually decreases, maintaining efficiency without sacrificing accuracy. On average, PMSR completes within 1-2 iterations, increasing processing time by 40-50% compared to a single-pass RAG; the additional cost primarily stems from extra token consumption rather than redundant computation.

[0133] This application proposes a time-sensitive retrieval augmented generation (RAG) framework that enhances both retrieval and inference capabilities to address the challenges of temporal question answering. By combining hybrid temporality-aware retrieval with progressive multi-step reflection, this application effectively matches retrieved information to temporal constraints, providing accurate and context-relevant answers even for complex multi-hop queries. Furthermore, this application introduces ChronoQA, an automated benchmark for evaluating temporal inference capabilities in RAG systems, providing a scalable and dynamic dataset for assessing model performance across evolving knowledge domains. Experimental results demonstrate that this application significantly outperforms existing baseline models, bridging the gap between static retrieval and dynamic temporal understanding.

[0134] The beneficial effects of this application are as follows:

[0135] This application introduces a benchmark dataset for evaluating time-sensitive RAG systems, created through a novel automated process. ChronoQA contains a variety of time queries from dynamic knowledge domains, providing a foundational resource for evaluating the time reasoning capabilities of RAG models.

[0136] This application proposes a novel framework that integrates dynamic knowledge updates and temporal reasoning into the retrieval and generation process, enabling accurate and timely responses to time-related issues.

[0137] As used herein, the term "preferred" is meant as an example, illustration, or illustration. Any aspect or design described herein as "preferred" need not be construed as being more advantageous than other aspects or designs. Rather, the use of the term "preferred" is intended to present the concept in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusionary "or." That is, unless otherwise specified or clear from the context, "X uses A or B" naturally includes either of the permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0138] Furthermore, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components (e.g., elements, etc.), the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure performing the functions in the exemplary implementations of this disclosure shown herein. Moreover, although specific features of this disclosure have been disclosed with respect to only one of several implementations, such features may be combined with one or more features of other implementations that may be desirable and advantageous for a given or particular application. Furthermore, with regard to the use of the terms “comprising,” “having,” “containing,” or variations thereof in the Detailed Description or claims, such terms are intended to be included in a manner similar to the term “including.”

[0139] The functional units in this invention embodiment can be integrated into a processing module, or each unit can exist physically separately, or multiple units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. The aforementioned devices or systems can execute the storage methods in the corresponding method embodiments.

[0140] In summary, the above embodiments are one implementation of the present invention, but the implementation of the present invention is not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made that deviate from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A method for enhancing the generation of time-sensitive questions, characterized in that, Includes the following steps: Input: Enter multiple documents and questions; Hybrid time-aware retrieval: This method enhances document retrieval by adding time constraints on top of semantic relevance, and is guided by time cards to ensure that the retrieved documents not only match the meaning of the query but also their temporal context. The retrieval process first uses a restated query for semantic retrieval, generating a set of candidate documents. Then, time cards are used to reorder and filter these documents, evaluating their relevance to the time constraint t. ref The degree of fit; the final retrieved set D t To maximize semantic relevance and temporal consistency; Progressive multi-step reflection: First, apply hybrid time-aware retrieval to acquire and evaluate the initial document set D. t If the document is retrieved, use the large language model to generate the final answer A. t Otherwise, proceed to the reflection phase, summarizing the useful time information from the retrieved documents into a context C; this context optimizes the query to Q′. t The identified information gaps are explicitly included, and then additional retrieval steps are performed using the optimized query to obtain a new document set D. ' t Merge the retrieved documents with C, and repeat this process until the retrieved information meets the requirements or the predefined maximum number of iterations is reached. Combine all the documents accumulated in the iterations to generate the final answer.

2. The retrieval enhancement generation method for time-sensitive questions according to claim 1, characterized in that, The time card contains: Time constraints, including explicit timestamps and time ranges; Time keywords extracted from the question; To retrieve optimized, restated queries, semantic and temporal elements are integrated. The method for generating the time card includes: for a given time problem Q t =(q sem ,t ref ), where q sem Represents semantic content, t ref To indicate time constraints, time cards are generated using a large language model.

3. The retrieval enhancement generation method for time-sensitive questions according to claim 2, characterized in that, The hybrid time-aware retrieval steps are specifically defined as follows: in It is a knowledge base; Retrieve and Filter represent semantic retrieval and time filtering, respectively. ref It is a time constraint, D t It is the final set retrieved.

4. The retrieval enhancement generation method for time-sensitive questions according to claim 3, characterized in that, The initial document set D was evaluated based on two criteria. t : Semantic sufficiency: Does the retrieved document provide enough information to answer the question? Temporal consistency: Whether the retrieved documents meet the time constraint t ref ; If the retrieved document meets both criteria, the final answer A is generated using a large language model. t ; Optimized query Q ' t The definition is as follows: Q't=f(q sem ,t ref ,C) Where f is the query optimization function.

5. The retrieval enhancement generation method for time-sensitive questions according to claim 4, characterized in that, Final answer A t It is a comprehensive context used to generate answers, formed by merging the collection of documents accumulated throughout all iterations. in This represents the union of the document sets retrieved in k iterations.

6. A method for generating datasets for time-sensitive problems, characterized in that, Includes the following steps: S1. Source article preparation: Data is collected from news websites using web crawlers and preprocessed; the preprocessing includes cleaning and deduplication of the data to remove redundant or irrelevant content; The data is summarized using a large language model, and the time and fact elements in the data are extracted into paragraphs, which constitute the basic input for question generation. S2. Single-time question-answer generation: Systematically generate time-based question-answer pairs from the processed source text using a large language model; S3 Multi-Time Question-Answer Combination: Extends single-document question-answer pairs into more complex multi-document question-answer pairs through parallel, serial, and hybrid methods.

7. The dataset generation method for time-sensitive problems according to claim 6, characterized in that, In the single-time question-answer generation step, each paragraph containing time information serves as the basis for generating independent question-answer pairs; given a source paragraph P, the large language model is prompted to generate a set of time-based question-answer pairs. Q i Let A represent the i-th question. i This indicates the corresponding answer; to improve diversity and relevance, different prompt templates are developed for different time-related question types. After generation, the question-answer pairs are filtered to remove duplicates and ensure uniqueness.

8. The dataset generation method for time-sensitive problems according to claim 7, characterized in that, In parallel processing, sub-questions are logically independent but together constitute the necessary conditions for answering the main question; each sub-question provides unique information, and only by solving all sub-questions can a complete answer be obtained; in order to construct parallel questions programmatically, semantically aligned individual question-answer pairs are identified and aggregated into a unified query; the model summarizes independent answers to comprehensively answer the combined question. In serial questions, sub-questions are interdependent, forming a sequential chain of reasoning, with the answer to one sub-question serving as the input or context for the next. Serial question combinations are constructed by identifying shared temporal or event-based references between question-answer pairs, which act as bridges connecting the sub-questions. Hybrid problem combinations combine parallel and serial patterns, and include both independent subproblems and interdependent subproblems.

9. The dataset generation method for time-sensitive problems according to claim 8, characterized in that, To construct hybrid problem combinations, create parallel and serial problem combinations separately, and then combine them. Combinatorial problem Q c Represented as subproblems Q1, Q2, ..., Q n Combinations: Q c =g(Q1,Q2,…,Q n ), Where g is a combination function used to encode the relationships between subproblems; For parallel problems, g is the union of independent problems: Q c =Q1∪Q2∪…∪Q n For the chaining problem, g is a chain of sequential dependencies: Q c =Q1→Q2→…→Q n For mixed problems, g combines two operations: Q c =Q1→(Q2∪…∪Q n )。

Images