A method and system for accelerating multi-level semantic caching in intelligent question answering systems

Abstract

This invention discloses a multi-level semantic caching acceleration method and system for intelligent question answering systems, relating to the field of natural language processing technology. The method includes: constructing a four-level caching architecture, accessing each level of cache sequentially according to priority; if a precise cache hit occurs, the result is returned directly; if a miss occurs, semantic similarity matching is performed; if a semantic cache hit occurs, the result is written back and returned; if a miss occurs, subsequent caches are checked sequentially; if all misses occur, the RAG process is executed and the result is written back. The system includes caching modules at each level, an access scheduling module, and a write-back module. This invention, through its multi-level caching architecture, improves cache hit rate, reduces the amount of large model calls, significantly improves question answering response speed, adapts to high-concurrency intelligent question answering scenarios involving multiple languages ​​along the Belt and Road Initiative, and reduces the cost of large model calls.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and natural language processing technology, specifically relating to a multi-level semantic caching acceleration method and system for a retrieval-enhanced generative (RAG) AI question-answering system. It is adapted to the high-concurrency application scenarios of multilingual intelligent question answering along the Belt and Road Initiative, and can effectively reduce the cost of calling large models and improve the efficiency of question-answering response. Background Technology

[0002] The AI-based RAG intelligent question-answering system is widely used in Belt and Road policy consultation and multilingual government affairs question-answering scenarios. The traditional RAG question-answering process usually includes four core steps: query vectorization, vector retrieval, prompt word construction, and large language model generation. The complete end-to-end response time is usually 5-10 seconds, which cannot meet the user's real-time query needs.

[0003] Currently, most caching solutions for intelligent question-answering systems in the industry are single-precision caching models, which have several technical shortcomings: First, the caching level is single, only supporting exact text matching, and it cannot effectively hit multilingual queries with the same semantics but different expressions, with an overall cache hit rate of only about 35%; Second, the cache update mechanism is lagging behind. When the Belt and Road Initiative policy knowledge base documents are updated, the existing cache cannot be updated synchronously, which can easily lead to the system returning outdated answers, resulting in a question-answering accuracy of only 80%; Third, there is no hierarchical access scheduling mechanism. All query requests are processed through a single cache channel, making it impossible to match the optimal caching strategy based on query characteristics, resulting in poor resource scheduling rationality; Fourth, there is the problem of repeated vector calculation. The same query text needs to be recalculated every time a question-answering is triggered, which consumes a lot of BGE-M3 model inference resources. The inference time for a single query can reach 100ms, resulting in a high system load.

[0004] In summary, existing intelligent question-answering caching technologies suffer from problems such as low hit rate, high resource consumption, high response latency, and poor adaptability, making it difficult to meet the needs of multilingual, high-concurrency, and low-latency government question-answering scenarios under the Belt and Road Initiative. Therefore, a new multi-level semantic caching acceleration solution is urgently needed to address the aforementioned technical shortcomings. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-level semantic caching acceleration method and system for intelligent question answering systems. This method solves the technical problems of existing RAG intelligent question answering caching technology, such as single-level caching, poor semantic reusability, low cache hit rate, high resource consumption of large models, and high response latency. It is adapted to the high-concurrency intelligent question answering scenarios of multilingual systems along the Belt and Road Initiative, and achieves the effects of faster question answering response, reduced resource consumption, and improved accuracy.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a multi-level semantic caching acceleration method for intelligent question answering systems, which constructs a four-level progressive caching acceleration architecture. According to the order of hit latency from low to high and hit accuracy from high to low, the caching architecture is divided into an L1 precise caching layer, an L2 semantic caching layer, an L3 hot data eviction (LRU) caching layer, and an L4 embedding vector caching layer. The caching layers cooperate and are scheduled in a hierarchical manner to achieve high-speed response of intelligent question answering.

[0007] The specific processing flow is as follows: After a user query request is received by the system, the L1 precise cache layer is accessed first. If an exact match is found, the cached result is returned directly with a response time of less than 1 millisecond. If the precise cache layer is not found, the fine-tuned BGE-M3 embedding model is called to generate a 1024-dimensional query vector. The cosine similarity is calculated with the historical query vector in the semantic cache layer. When the similarity is ≥ 0.92, the semantic match is determined to be successful. The corresponding cached answer is returned, and the current query result is written back to the L1 precise cache layer. If the semantic cache layer is not found, the L3 hot data eviction cache layer and the L4 embedding vector cache layer are accessed sequentially. If any cache layer is found, the result is returned directly, and the data in the high-priority cache layer is updated synchronously according to the write-back rules. If none of the four cache layers are found, the system executes the complete retrieval enhancement generation process to obtain the question and answer result, and synchronously writes the result back to each level of cache layer to complete the cache data update.

[0008] This invention establishes a multi-level cache write-back and hot data promotion mechanism. When the semantic cache layer is hit, the query key-value pair is automatically written to the precise cache layer. When the hot data eviction cache layer or the embedding vector cache layer is hit, the result is automatically migrated to a higher-priority cache layer, realizing dynamic promotion of high-frequency hot data and continuously optimizing cache response efficiency. At the same time, the embedding model of the semantic cache layer adopts a lazy loading strategy, loading it into memory only during the first semantic matching verification, effectively reducing the peak memory usage at system startup.

[0009] The second aspect of this invention provides a multi-level semantic caching acceleration system for an intelligent question-answering system, used to implement the above-mentioned multi-level semantic caching acceleration method, including a precise caching module, a semantic caching module, a hot data eviction caching module, an embedded vector caching module, a cache access scheduling module, and a cache write-back module; each caching module corresponds to a four-level caching architecture, the cache access scheduling module is used to schedule cache access level by level according to a preset priority order, and the cache write-back module is used to realize the hot data upgrade and cache synchronous update after the low-priority cache hit.

[0010] Compared with the prior art, the present invention has the following beneficial effects: 1. Extremely low latency response: the hit latency of L1 precise caching layer, L3 hot data eviction caching layer and L4 embedded vector caching layer is less than 1 millisecond, and the hit latency of L2 semantic caching layer is less than 50 milliseconds. Compared with the traditional 5-10 second complete RAG process response speed, the response speed is improved by two orders of magnitude, which fully meets the real-time question answering requirements.

[0011] 2. Significantly improve cache hit rate: Breaking through the limitations of traditional single exact matching cache, it relies on semantic similarity matching to achieve effective hits for queries with the same but different expressions, increasing the cache hit rate from 35% to 72%, and greatly reducing redundant calculations and model calls.

[0012] 3. Save computing power and operating costs: The embedded vector caching layer avoids the resource consumption of repeated vector calculations. The multi-level caching mechanism greatly reduces the number of calls to large language models, reducing the number of large model calls by 60% and reducing monthly operating costs by 58%.

[0013] 4. Dynamically optimize cache efficiency: Through the hot data automatic upgrade mechanism, high-frequency query data is dynamically migrated to the high-priority, low-latency cache layer, which further improves the response speed of hot query by 99% and continuously optimizes the rationality of cache scheduling.

[0014] 5. Reduce system memory consumption: The embedded model lazy loading strategy effectively reduces the peak system startup memory by 40%, and the hot data eviction strategy enables controllable cache capacity, ensuring long-term stable operation of the system. Attached Figure Description

[0015] Figure 1 This is a block diagram of the multi-level semantic caching acceleration system of the intelligent question-answering system of the present invention. Detailed Implementation

[0016] The present invention will be further described in detail and completely below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited to the following embodiments.

[0017] This invention discloses a multi-level semantic caching acceleration method for intelligent question-answering systems, adapted to high-concurrency question-answering scenarios in multilingual government affairs related to the Belt and Road Initiative. The overall process is shown in the attached figure. Figure 1 As shown, by using a four-level hierarchical cache for step-by-step verification, matching, and write-back, question-and-answer response is accelerated and computing power is reduced.

[0018] In practice, the system defaults to enabling a four-level caching collaborative working mechanism, with each level of cache parameters preset to adapt to government affairs Q&A scenarios: L1 precise cache is set to a 1-hour TTL expiration period, L3 hot data eviction cache capacity is configured to 2000 entries with a TTL of 1800 seconds, L4 embedded vector cache capacity is configured to 8000 entries, and the semantic cache similarity preset threshold is fixed at 0.92, adapting to the matching requirements of Belt and Road professional terms and multilingual synonym expressions.

[0019] When a user terminal initiates a multilingual policy inquiry, government affairs consultation, or other Q&A request, the system initiates a hierarchical cache verification process according to preset priorities. First, it calls the L1 precise cache for full-text exact matching. If the query text is completely consistent with the local cached data, the system directly retrieves the cached result and returns it to the user. The overall response latency is controlled within 1 millisecond, achieving extremely fast response.

[0020] If the L1 precise cache does not find a matching data, the system automatically calls the BGE-M3 embedding model, fine-tuned for the Belt and Road multilingual corpus, to convert the current user query text into a 1024-dimensional standard vector. It then iterates through all historical query vectors stored in the L2 semantic cache layer, calculating the vector matching degree one by one using the cosine similarity formula. When the similarity between any historical query vector and the current query vector is greater than or equal to a preset threshold of 0.92, a semantic match is considered successful. The system retrieves the corresponding historical question-and-answer cache result and sends it back to the user, with a response time not exceeding 50 milliseconds. Simultaneously, a cache write-back mechanism is triggered, automatically writing the current user query and corresponding response data into the L1 precise cache layer, completing the hot data upgrade. Subsequent user queries with the same or highly similar information can be responded to quickly through the L1 precise cache.

[0021] If the L2 semantic cache fails to match, the system automatically checks the L3 hot data eviction cache layer and the L4 embedded vector cache layer level by level. The L3 hot data eviction cache stores frequently accessed question-and-answer hot data, which can directly match recent similar queries. The L4 embedded vector cache stores vector data from historical queries, allowing direct reuse of vector calculation results and avoiding repeated model inference. When an L3 or L4 cache hits, the system quickly returns the corresponding question-and-answer result and, according to the hot data upgrade rules, synchronously migrates the hot data from the lower-priority cache to a higher-priority cache layer, optimizing the response efficiency of subsequent queries.

[0022] When all four levels of cache (L1, L2, L3, and L4) fail to find the query, the system determines it to be a completely new query request and automatically initiates a full retrieval enhancement generation process. This process involves knowledge base retrieval, prompt word construction, and large-scale model inference to generate standardized question-and-answer results. After completing the user's response feedback, the query text, corresponding vector data, and question-and-answer results are synchronously written to the four-level cache layers to complete a full-level cache update, providing cache reuse support for subsequent similar queries.

[0023] Meanwhile, this invention adopts an embedded model lazy loading strategy, loading the model into system memory only when L2 semantic cache matching is triggered for the first time, effectively reducing the peak memory usage during system startup. Combined with the automatic eviction mechanism of L3 hot data eviction cache, it can maintain system memory stability for a long time, adapting to high-concurrency and uninterrupted government Q&A service scenarios, and effectively solving the industry pain points of low hit rate, high computing power consumption and high response latency of traditional Q&A cache.

Claims

1. A multi-level semantic cache acceleration method and system for intelligent question and answer systems, characterized in that, Includes the following steps: A four-level caching architecture is constructed, consisting of an exact caching layer, a semantic caching layer, a hot data eviction caching layer, and an embedded vector caching layer, ordered from highest to lowest access priority. Upon receiving a user query request, each level of cache is accessed sequentially according to the aforementioned priority order. If the exact cache layer is hit, return the cached result directly; If the exact cache layer is not hit, calculate the vector representation of the current query and calculate the cosine similarity with each of the historical query vectors stored in the semantic cache layer; If there are historical queries in the semantic cache layer with a similarity exceeding the preset threshold of 0.92, return the cached answer corresponding to the historical query and write the current query result back to the precise cache layer; if the semantic cache layer does not hit, check the hot data eviction cache layer and the embedding vector cache layer in turn. If all cache layers fail to find the target, execute the complete search enhancement generation process and write the final result back to each level of cache layer.

2. The method as described in claim 1, characterized in that, The cosine similarity calculation formula is S = (A・B) / (||A|| × ||B||), where A is the current query vector and B is the historical query vector in the semantic cache layer.

3. The method as described in claim 1, characterized in that, The vector is represented as a 1024-dimensional vector.

4. The method as described in claim 1, characterized in that, The write-back mechanism is as follows: when the semantic cache layer is hit, the query key-value pair is automatically written to the precise cache layer; when the hot data eviction cache layer or the embedded vector cache layer is hit, the result is automatically written to the higher priority cache layer.

5. A multi-level semantic caching acceleration system for an intelligent question-answering system, characterized in that, include: The precise caching module, semantic caching module, hot data eviction caching module, and embedded vector caching module correspond to the four-level caching architecture. The cache access scheduling module is used to send access requests to each level of cache module in sequence according to a preset priority order; the cache write-back module is used to automatically write the query results to a higher priority cache module when a low-priority cache module hits the cache.

Images