A Prompt optimization and caching system, method, apparatus and medium
By building a closed loop of "collection-optimization-caching-routing" on the front end, combined with multi-level caching and a two-layer optimization mechanism, the problem of Prompt optimization relying on manual experience and complex deployment is solved, achieving low-cost and efficient Prompt optimization and caching, and reducing API call volume and response latency.
Patent Information
- Application Number
- CN202611309836.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-27
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, Prompt optimization heavily relies on human experience, is complex and costly to deploy, cannot be easily integrated into the front end, results in significant resource waste, and has high response latency. Existing caching technologies cannot utilize front end behavioral data for precise optimization.
By employing the Prompt effect tracking module, dynamic model routing module, and acquisition module, a closed loop of "collection-optimization-caching-routing" is built on the front end. L1 memory caching, L2 local caching, and L3 similarity caching are utilized, combined with the L1 fast optimization rule base and the L2 intelligent optimization mechanism, to dynamically select execution nodes to reduce API call volume and response latency.
It achieves reduced API call volume and response latency in a pure front-end environment, lowers deployment costs, improves Prompt optimization efficiency, utilizes front-end behavioral data for precise caching, and reduces manual reliance.
Smart Images

Figure CN123114590A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of artificial intelligence front-end application optimization technology, and specifically relates to a Prompt optimization and caching system, method, device and medium. Background Technology
[0002] With the rapid development of Large Language Model (LLM) technology, front-end applications are increasingly integrating AI capabilities to enhance user experience. However, existing technologies suffer from the following problems: High manual costs: Prompt optimization heavily relies on human experience, requiring specialized engineers for repeated debugging, and cannot adapt to real-time changes in user intent. Heavy backend dependencies: Most existing prompt optimization solutions require training dedicated models on the backend, resulting in high deployment costs and hindering lightweight integration on the frontend. Significant resource waste: Repeated calls to the same prompt for the same business scenario lead to significant token waste and API call costs. High response latency: Each prompt requires calling a large cloud model, and network latency severely impacts user experience.
[0003] Existing technologies, such as Ping An Technology's Prompt generation patent (a dedicated model for backend heavy-mode training) and Huawei's data processing patent (batch-stream heterogeneous development), suffer from complex deployment and high costs. Meanwhile, existing Prompt caching technologies primarily focus on the backend API layer, with high caching thresholds (≥1024 tokens), uncontrollable strategies, and an inability to leverage frontend behavioral data for precise optimization.
[0004] Therefore, a Prompt optimization and caching system that is easy to deploy and low in cost is needed. Summary of the Invention
[0005] To address the aforementioned problems, this application proposes a Prompt optimization and caching system, including: The Prompt effect tracking module saves the original function and builds a wrapper function. After intercepting the call, it passes the original input parameters to the saved original function for execution and collects Prompt data for reporting. The dynamic model routing module evaluates the complexity of the Prompt data, selects the execution node based on the complexity to obtain a routing score, and executes AI calls based on the routing score. The module retrieves the current number of frames per second. If the number of frames per second is less than a, then the Prompt data is cached. Cache caching of the Prompt includes querying the L1 memory cache, L2 local cache, and L3 similarity cache in sequence. If a cached response is found, it is returned directly. If a cached response is not found, the AI model is called and the result is written back to each level of cache.
[0006] If the number of frames per second is greater than or equal to a, then the Prompt is optimized. The optimization of the Prompt adopts a two-layer optimization mechanism consisting of L1 fast optimization rule base fast optimization and L2 intelligent optimization.
[0007] Furthermore, the Prompt effect tracking module saves the original function, constructs a wrapper function, and after intercepting the call, passes the original input parameters to the saved original function for execution, and collects and reports Prompt data, including the following steps: Construct a wrapper function and save the original AI SDK into the wrapper function; The business code calls the wrapper function, the wrapper function's interceptor records the start time, and the wrapper function calls the original AISDK; After the original AI SDK processes the code and returns a wrapper function, the wrapper function's interceptor records the end time, obtains the total time based on the start and end times, and retrieves the behavioral metrics. The wrapper function compresses the total time and behavioral metrics, writes them to the reporting queue, and returns the corresponding results to the business code.
[0008] Furthermore, the dynamic model routing module evaluates the complexity of the Prompt data, selects the execution node based on the complexity to obtain a routing score, and executes the AI call based on the routing score, including the following steps: Receive Prompt data and perform complexity analysis on the Prompt data to obtain the number of words and logical layers; A complexity scoring model is constructed based on the number of words and the number of logical layers. Select the processing model based on the complexity scoring model; Obtain the routing score model and execute AI calls based on the routing score model.
[0009] Furthermore, Complexity scoring models include, Score_c = number of words × 0.3 + number of logical layers × 20 + technology density × 50 + task type weight; Routing scoring models include, Score = α × Speed + β × Quality γ×Cost, where Speed is the normalized value of the estimated response speed, Quality is the normalized value of the estimated output quality, Cost is the normalized value of the estimated call cost, and α, β, and γ are configurable weight coefficients.
[0010] Furthermore, if the number of frames per second is ≥ a, then the Prompt is optimized; the optimization of the Prompt adopts a two-layer optimization mechanism consisting of L1 fast optimization rule base fast optimization and L2 intelligent optimization, including the following steps, If 20 ≤ frames per second < 30, then perform L1 fast optimization on the Prompt. If the frame rate is ≥30 frames per second, then L1 fast optimization and L2 intelligent optimization will be performed on the Prompt.
[0011] Furthermore, if the number of frames per second is less than 'a', then caching is performed on the Prompt. Caching the Prompt involves sequentially querying the L1 memory cache, L2 local cache, and L3 similarity cache. If a match is found, a cached response is returned directly; otherwise, the AI model is invoked and the result is written back to each level of cache. This includes the following steps: Get the current frame rate per second. If the frame rate per second is less than 20, cache the Prompt. Receive a Prompt query request and calculate or read the Prompt fingerprint; Query the L1 memory cache. If a match is found, return the corresponding response directly, and the process ends. If the L1 memory cache is not hit, query the L2 local cache. If it is hit, return the corresponding response and synchronously write back to the L1 memory cache so that subsequent requests can hit the layer faster. If the L2 local cache is not hit, query the L3 similarity cache; if the similarity is >95%, return the response corresponding to the most similar entry. If no match is found, the AI model is invoked to generate a new corresponding response, which is then written sequentially to the L3 similarity cache, L2 local cache, and L1 memory cache before finally returning the result.
[0012] Furthermore, the L3 similarity cache generates a 64-bit fingerprint by sequentially performing word segmentation, word weight calculation, weighted vector accumulation, and dimensionality reduction, and judges the similarity by Hamming distance, with the similarity threshold set to 95%.
[0013] A Prompt optimization and caching method, including, Save the original function and build a wrapper function. After intercepting the call, pass the original input parameters to the saved original function for execution and collect the Prompt data for reporting. The complexity of the Prompt data is evaluated, and the execution node is selected based on the complexity to obtain the routing score. The AI call is then executed based on the routing score. Get the current frames per second. If the number of frames per second is less than a, then the Prompt data is cached. Cache caching of the Prompt includes querying the L1 memory cache, L2 local cache, and L3 similarity cache in sequence. If a cached response is found, it is returned directly. If a cached response is not found, the AI model is called and the result is written back to each level of cache.
[0014] If the number of frames per second is greater than or equal to a, then the Prompt is optimized. The optimization of the Prompt adopts a two-layer optimization mechanism consisting of L1 fast optimization rule base fast optimization and L2 intelligent optimization.
[0015] A Prompt optimization and caching device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.
[0016] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.
[0017] Beneficial effects: 1. The Prompt optimization and caching system of this invention includes a Prompt effect tracking module, a dynamic model routing module, and an acquisition module. The module acquires the current frame rate per second (fps). If the fps is less than 'a', the Prompt data is cached. Cache execution includes sequentially querying the L1 memory cache, L2 local cache, and L3 similarity cache. If a match is found, a cache response is returned directly; otherwise, the AI model is invoked and the result is written back to each level of cache. If the fps is greater than or equal to 'a', the Prompt is optimized. Prompt optimization employs a two-layer optimization mechanism consisting of L1 fast optimization rule base optimization and L2 intelligent optimization. This two-layer optimization mechanism, with the page detection unit monitoring the page frame rate per second and dynamically determining the optimization level based on a preset threshold, constructs a "collection-optimization-caching-routing" closed loop in a pure front-end environment, reducing API call volume and response latency, while also being simple to deploy and low-cost.
[0018] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description and the accompanying drawings. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a schematic diagram of the architecture of the Prompt optimization and caching system according to an embodiment of the present invention.
[0021] Figure 2 This is a timing diagram illustrating the interception of AI SDK calls by the monitoring agent in Embodiment 1 of the present invention.
[0022] Figure 3 This is a schematic diagram of the L1 fast optimization / L2 intelligent optimization dual-layer Prompt optimization sub-process in Embodiment 2 of the present invention.
[0023] Figure 4 This is a schematic diagram of the three-level cache query subprocess in Embodiment 3 of the present invention.
[0024] Figure 5 This is a schematic diagram of the dynamic model routing process in Embodiment 4 of the present invention.
[0025] Figure 6 This is a schematic diagram of the L3 similarity cache calculation process in Embodiment 3 of the present invention.
[0026] Figure 7 This is a schematic diagram of the overall performance budget control process in Embodiment 2 of the present invention.
[0027] Figure 8 This is a schematic diagram of the hardware structure of the device according to an embodiment of the present invention.
[0028] Figure 9 This is a schematic diagram of the data reporting timing in Embodiment 1 of the present invention.
[0029] Figure 10 This is a flowchart illustrating the Prompt optimization and caching method according to an embodiment of the present invention.
[0030] Figure 11 This is a flowchart illustrating the Prompt optimization and caching device according to an embodiment of the present invention. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0032] Example 1 Non-intrusive Prompt (prompt word or input command) effect tracking, see reference. Figure 2 , Figure 3 and Figure 9 This embodiment addresses the problem that existing technologies cannot evaluate the effectiveness of prompts at a fine-grained level on the front end and can only rely on backend logs.
[0033] Specifically, refer to Figure 3 The monitoring proxy injection mechanism is as follows: S11, Initialization Phase: Save the original AI SDK (Software Development Kit) chat completion request to the original method variable; construct a wrapper function using the proxy object ( Figure 3 The monitoring agent in the SDK is replaced with a wrapper function, which replaces the methods exposed by the SDK. Business code subsequently calls only the wrapper function; the original methods are not deleted and remain held by the original methods. The functions of the original AI SDK are saved in a secure location, and wrapper functions are built. Business code calls the wrapper functions, which in turn can call the functions of the original AI SDK.
[0034] S12, Invocation Trigger: When the business code executes a function from the original AI SDK, it actually enters the interceptor of the proxy object. Here, "parameters and context" refers to the request object passed in from the business side, including fields such as the dialogue message array, model name, temperature, and streaming information, as well as the bound object at the time of the invocation. The business code then calls the wrapper function; S13, Timing and Forwarding: The interceptor records the start time and passes the input parameters as is to the original method, which then initiates the actual request to the AI server. It's important to note that the first step replaces the SDK's external interface, not destroys the original AI SDK; the proxy object's role is to add data collection logic before and after the call, while the core inference is still performed by the original method. The wrapper function records the start time through the interceptor and executes the function that calls the original AI SDK.
[0035] S14, Response Processing: After the original method returns, the end time is recorded, the time for the first token (the smallest unit of measurement for text processing in a large model) and the total time are calculated, and behavioral metrics such as the number of response tokens and the number of subsequent edits by the user are collected, compressed, and written to the reporting queue. The function return response of the original AI SDK is sent to the interceptor, which records the termination time, calculates the time interval between the start and end times, and collects behavioral metrics such as the number of response tokens and the number of subsequent edits by the user, compressed, and written to the reporting queue.
[0036] S15, Result Return: Return the business code without rewriting the original response object, ensuring zero intrusion into existing business logic.
[0037] S16 Data Reporting Strategy Figure 9 The data collector writes the raw monitoring data into a message queue after dictionary encoding and zlib compression. When the queue meets either a cumulative total of 10 messages or an interval of 5 seconds, it sends the data in batches to the analysis platform's server through the reporting interface. Forced refresh is triggered when the page is unloaded to avoid data loss.
[0038] Furthermore, the raw monitoring data collected from multiple dimensions includes: Prompt features (template ID, number of placeholders, text length), response quality (first token time, total time, token consumption, adoption rate), user behavior (number of secondary edits, satisfaction level), and business context (page URL, role, device level, network type).
[0039] Example 2 Lightweight online optimization engine, for reference Figure 4 and Figure 8 This embodiment includes the overall performance budget control process ( Figure 8 ) and L1 rapid optimization / L2 intelligent optimization sub-processes ( Figure 4 There are two levels, and they are in a hierarchical relationship: Figure 8 Decide "whether to optimize, and to what level"; Figure 4 Describe the specific execution order of L1 fast optimization and L2 intelligent optimization when optimization is allowed.
[0040] Performance budget control, reference Figure 8 Once the page has finished loading, the page detection unit is activated to monitor the page's frame rate per second in real time and switch working modes according to the following rules: Frames per second ≥ 30: Full-featured optimization is enabled; L1 fast optimization and L2 intelligent optimization can participate in parallel (enter...). Figure 4 Sub-process); 20 ≤ Frames per second < 30: Downgrade to L1 fast optimization rule optimization only (enter) Figure 4 (but does not trigger L2 intelligent optimization branch). Frames per second < 20: Pause all online optimization calculations, switch to cache mode, and only allow [these operations]. Figure 5 The results are returned using a three-level cache to ensure smooth page interaction.
[0041] It's important to note that the frames per second (fps) threshold is used to control the computational budget, not to directly determine the choice between L1 fast optimization and L2 intelligent optimization. Even if the fps is ≥30, if L1 fast optimization fails to meet the performance verification, L2 intelligent optimization may still be implemented for deeper optimization; even if the fps is in the 20~30 range, L1 fast optimization will still be performed instead of being completely skipped. Figure 4 The "Frames per second ≥ 30 / Frames per second < 30" branches indicate whether, given the current performance budget, the conditions for enabling L2 intelligent optimization are met. Figure 8 The global mode switching is seamless and the two are not contradictory.
[0042] Further, refer to Figure 4 The optimization process involves two layers: L1 fast optimization and L2 intelligent optimization. This includes the following steps: The L1 fast optimization unit maintains 58 expert rules, covering categories such as redundancy removal, format completion instructions, structured constraints, and injection prevention. It uses string matching and regular expression replacement, with a single operation taking approximately 2-8 ms. The L2 intelligent optimization unit loads the quantization model, specifically Qwen-2-0.5B-webllm, and performs semantic polishing and logical completion, with a maximum of 5 inference steps, taking approximately 60-90 ms.
[0043] Figure 4 Execution order: Receive Prompt → Performance budget check → L1 quick optimization and rule base matching → Regular expression rewriting → Optimization effect verification; If the effect is good, return; If deep optimization is required and the current performance budget allows, enter L2 intelligent optimization, then enter the local large language model semantic rewriting and return the optimization result.
[0044] Example 3 For tiered caching methods, please refer to [reference needed]. Figure 5 and Figure 6 . Figure 5 for Figure 8 The next level of detailed process in the "Cache Mode" branch; when the frame rate per second is <20 or the service explicitly enables the cache priority strategy, the system proceeds according to... Figure 5 Perform a three-level progressive query.
[0045] Cache hit and miss definitions: "Hit": Using the current Prompt (L1 memory cache / L2 local cache) or its L3 similarity cache as the key, find the corresponding historical response that is semantically equivalent and has not expired in the corresponding cache layer, and return it directly without calling the AI model; "Missed": There is no reusable entry in the current cache layer, and the next layer needs to be queried. If the L1 memory cache, L2 local cache, and L3 similarity cache are all missed, the AI model (artificial intelligence model) is called to obtain new results.
[0046] refer to Figure 5 It includes the following steps, Step S1: Receive a Prompt query request and calculate or read the Prompt fingerprint; Step S2: Query the L1 memory cache. If a match is found, return the corresponding response directly, and the process ends. Step S3: If the L1 memory cache is not hit, query the L2 local cache. If it is hit, return a response and synchronously write back the L1 memory cache so that subsequent requests can hit the faster layer; Step S4: L2 local cache miss, query L3 similarity cache (see calculation process) Figure 6 If the similarity is >95%, return the response corresponding to the most similar entry; Step S5: If no match is found at all three levels, call the AI model to generate a new answer and write it sequentially to the L3 similarity cache, L2 local cache, and L1 memory cache, and finally return the result.
[0047] refer to Figure 7 Similarity calculation includes word segmentation of the prompt text, calculation of word weights, generation of word vectors, weighted accumulation, dimensionality reduction to generate fingerprints, and 64-bit similarity hash values; calculation of Hamming distance, the smaller the distance, the higher the similarity, and in this embodiment, 95% is used as the usable threshold.
[0048] Furthermore, to verify the effectiveness of this invention, a comparative test was conducted on a front-end widget of a corporate customer service assistant. Test environment: Chrome 125 browser, MacBook Pro M2 hardware, 100Mbps network; the control group directly called the OpenAI API (nakedly tuning the benchmark performance of a large model), while the experimental group enabled all modules of this invention. Each scenario was repeated 100 times, and the average was taken. The results are as follows.
[0049] Table 1 Performance comparison test data (100 times / request)
[0050] Table 2. Hit statistics for the three-level cache (1000 real user requests, 7-day rolling window)
[0051] Table 3. Response time of tiered caching (hit scenario, 100 samples)
[0052] As shown in Tables 1-3, the overall hit rate of the three-level cache was 80%, and the API call volume decreased to 34%. The hit path time for L1 memory cache, L2 local cache, and L3 similarity cache was significantly lower than that for cloud calls. The L3 similarity cache had 157 hits, and 153 of the manually sampled hits were semantically acceptable, with an accuracy of 97.5%, meeting the 95% threshold setting. The peak usage of the L1 memory cache was 4.2MB, and it ran stably for 7×24 hours without any leaks.
[0053] Example 4 Dynamic model routing, see reference Figure 6 This embodiment selects the execution node based on the complexity of the Prompt, between a local WebLLM (Large Language Model), edge Workers AI (Cloudflare Workers AI, running an open-source large model on the edge node closest to the user), and a cloud-based large model. The steps include: S41, Receive the Prompt and perform a complexity analysis on the Prompt; Specifically, complexity calculation includes counting the number of Prompt words; parsing the number of nested JSON layers and the number of conditional statements; calculating the proportion of technical terms; and identifying the task type (question answering / generation / classification / reasoning), etc.
[0054] The comprehensive scoring formula is as follows: Score_c = number of words × 0.3 + number of logical layers × 20 + technology density × 50 + task type weight; S42, routing tiers: Score_c<30 selects local WebLLM; 30≤Score_c≤70 selects edge WorkersAI; Score_c>70 selects cloud-based large model.
[0055] S43, after the candidate nodes are determined, the final routing score is further calculated using a cost-speed-quality tradeoff formula; the tradeoff formula includes: Score = α × Speed + β × Quality γ × Cost; where: Speed is the normalized value of the estimated response speed (0~1, the larger the better); Quality is the normalized value of the estimated output quality (0~1, the larger the better); Cost is the normalized value of the estimated cost per call (0~1, the larger the more expensive); α, β, and γ are weighting coefficients, which default to 0.4, 0.4, and 0.2 respectively, and can be configured according to business scenarios. The system selects the node with the highest Score to execute the AI call. Figure 6 (Last-level execution AI call).
[0056] Example 5 Combination Figure 1 The present invention is divided into the following stages from SDK initialization to result return, and corresponding embodiments and figures are indicated: S51: SDK Initialization ( Figure 2 (Left entrance) When the page starts, the system loads and reads the API key, the capacity of the three-level cache (L1 memory cache = 100 entries, L2 local cache = 1000 entries, L3 similarity cache threshold 95%), the frame rate per second degradation threshold, and other configurations; it pre-initializes the cache module of Example 3, the optimization engine of Example 2, the acquisition module of Example 1, and the routing algorithm of Example 4, and generates an instance.
[0057] S52: Business invocation and collection, for details please refer to Example 1; The business logic executes functions from the original AI SDK. The SDK doesn't directly allow the original interface; instead, it enters a proxy object interception process: recording the start time, passing the monitoring proxy to the original method, collecting the first token (the smallest unit of measurement for text processing in a large model) time and behavior metrics, and asynchronously reporting (corresponding to...). Figure 3 (Full timeline process), reporting steps reference Figure 9 .
[0058] S53: Prompt Complexity Evaluation (Example 4) Figure 6 (First half) After request interception and before routing decision, perform computational complexity calculations: calculate word count, logical layer count, technical density, and task type, and output Score_c. Figure 6 The nodes are "Receive Prompt → Complexity Analysis → Calculate Word Count and Logical Layers → Complexity Score".
[0059] S54: Dynamic Model Routing (Example 4) Figure 6 (Second half) After selecting candidate nodes based on Score_c, the Score is calculated by substituting Speed / Quality / Cost into the values, thus determining the final execution node. Figure 5 The process involves the following nodes: "Cost-Speed-Quality Trade-off → Calculate Final Score → Execute AI Call".
[0060] S55: Performance Budget Judgment (Example 2) Figure 8 ) Before performing optimization, query the current frames per second to determine whether to perform full optimization, L1 memory caching only, or caching mode. If caching mode is selected, proceed to stage 6; otherwise, proceed to stage 7.
[0061] S56: Three-level cache lookup (Example 3) Figure 5 , Figure 8 ) The query proceeds in the following order: L1 memory cache → L2 local cache → L3 similarity cache. If a match is found, the result is returned directly. If no match is found, the AI model on the selected node in stage 4 is invoked, and the new result is written back to each level of cache. This stage is... Figure 8 The next level of the process in the cached mode branch.
[0062] S57: Prompt Online Optimization (Example 2) Figure 4 ) When the performance budget allows, perform L1 fast optimization rule optimization and optional L2 intelligent optimization. The optimized Prompt then enters stage 6 for cached queries or direct model invocation. Figure 4 for Figure 7 The next level of the process in the "Enable full-featured optimization / downgrade to L1 quick optimization" branch.
[0063] Example 6 refer to Figure 1 A Prompt optimization and caching system, comprising, The Prompt effect tracking module saves the functions of the original AI SDK and builds a wrapper function. After intercepting the call, it passes the original input parameters to the saved original AI SDK function for execution and collects Prompt data for reporting. Specifically, it uses a proxy object to initiate a chat completion request to the wrapped AI SDK. After intercepting the call, it passes the original input parameters to the saved original method for execution and collects Prompt structure features, response quality indicators and business context. The dynamic model routing module evaluates the complexity of the Prompt data, selects the execution node based on the complexity to obtain a routing score, and executes AI calls based on the routing score. The dynamic model routing steps are as follows: based on the Prompt complexity score, the execution node is selected among the local language large model, the edge large model, and the cloud large model.
[0064] The module retrieves the current number of frames per second. If the number of frames per second is less than a, then the Prompt data is cached. Cache caching of the Prompt includes querying the L1 memory cache, L2 local cache, and L3 similarity cache in sequence. If a cached response is found, it is returned directly. If a cached response is not found, the AI model is called and the result is written back to each level of cache.
[0065] If the frame rate per second is ≥a, then the Prompt is optimized. The Prompt optimization employs a two-layer optimization mechanism consisting of L1 fast optimization rule base fast optimization and L2 intelligent optimization. The page detection unit monitors the page frame rate per second and dynamically determines the optimization level based on a preset threshold. The purpose of this invention is to provide a Prompt optimization and caching method that builds a closed loop of "collection-optimization-caching-routing" in a pure front-end environment, thereby reducing API call volume and response latency.
[0066] Furthermore, the Prompt effect tracking module saves the functions of the original AI SDK, builds wrapper functions, and after intercepting the calls, passes the original input parameters to the saved original AI SDK functions for execution, and collects and reports Prompt data, including the following steps: Construct a wrapper function and save the original AI SDK into the wrapper function; The business code calls the wrapper function, the wrapper function's interceptor records the start time, and the wrapper function calls the original AISDK; After the original AI SDK processes the code and returns a wrapper function, the wrapper function's interceptor records the end time, obtains the total time based on the start and end times, and retrieves the behavioral metrics. The wrapper function compresses the total time and behavioral metrics, writes them to the reporting queue, and returns the corresponding results to the business code.
[0067] Furthermore, the wrapped proxy object method records the request start time in the apply hook, forwards the request parameters such as the dialog message array, model name, and stream passed in by the business code to the original create method saved in the initialization phase as is, records the end time after the original method returns the response and asynchronously reports the performance data, and then returns the original response to the business code.
[0068] Furthermore, the dynamic model routing module evaluates the complexity of the Prompt data, selects the execution node based on the complexity to obtain a routing score, and executes the AI call based on the routing score, including the following steps: Receive Prompt data and perform complexity analysis on the Prompt data to obtain the number of words and logical layers; A complexity scoring model is constructed based on the number of words and the number of logical layers. Select the processing model based on the complexity scoring model; Obtain the routing score model and execute AI calls based on the routing score model.
[0069] Furthermore, the complexity scoring model includes, Score_c = number of words × 0.3 + number of logical layers × 20 + technology density × 50 + task type weight; Routing scoring models include, Score = α × Speed + β × Quality γ×Cost, where Speed is the normalized value of the estimated response speed, Quality is the normalized value of the estimated output quality, Cost is the normalized value of the estimated call cost, and α, β, and γ are configurable weight coefficients.
[0070] Furthermore, if the number of frames per second is ≥ a, then the Prompt is optimized; the optimization of the Prompt adopts a two-layer optimization mechanism consisting of L1 fast optimization rule base fast optimization and L2 intelligent optimization, including the following steps, If 20 ≤ frames per second < 30, then perform L1 fast optimization on the Prompt. If the frame rate is ≥30 frames per second, then the Prompt will be optimized using L1 fast optimization and L2 intelligent optimization.
[0071] Furthermore, in the lightweight online optimization step, L1 fast optimization and L2 intelligent optimization are enabled when the frame rate per second is ≥30; L1 fast optimization rule optimization is enabled only when the frame rate per second is ≤30; optimization is paused and the system enters cache mode when the frame rate per second is <20, and results are returned only through the three-level cache.
[0072] Furthermore, if the number of frames per second is less than 'a', then caching is performed on the Prompt. Caching the Prompt involves sequentially querying the L1 memory cache, L2 local cache, and L3 similarity cache. If a match is found, a cached response is returned directly; otherwise, the AI model is invoked and the result is written back to each level of cache. This includes the following steps: Get the current frame rate per second. If the frame rate per second is less than 20, cache the Prompt. Receive a Prompt query request and calculate or read the Prompt fingerprint; Query the L1 memory cache. If a match is found, return the corresponding response directly, and the process ends. If the L1 memory cache is not hit, query the L2 local cache. If it is hit, return the corresponding response and synchronously write back to the L1 memory cache so that subsequent requests can hit the layer faster. If the L2 local cache is not hit, query the L3 similarity cache; if the similarity is >95%, return the response corresponding to the most similar entry. If no match is found, the AI model is invoked to generate a new corresponding response, which is then written sequentially to the L3 similarity cache, L2 local cache, and L1 memory cache before finally returning the result.
[0073] Furthermore, the L3 similarity cache generates a 64-bit fingerprint by sequentially performing word segmentation, word weight calculation, weighted vector accumulation, and dimensionality reduction, and judges the similarity by Hamming distance, with the similarity threshold set to 95%.
[0074] Example 7 refer to Figure 10 A Prompt optimization and caching method, including, Save the functions of the original AI SDK and build a wrapper function. After intercepting the call, pass the original input parameters to the saved original AI SDK function for execution and collect the Prompt data for reporting. The complexity of the Prompt data is evaluated, and the execution node is selected based on the complexity to obtain the routing score. The AI call is then executed based on the routing score. Get the current frames per second. If the number of frames per second is less than a, then the Prompt data is cached. Cache caching of the Prompt includes querying the L1 memory cache, L2 local cache, and L3 similarity cache in sequence. If a cached response is found, it is returned directly. If a cached response is not found, the AI model is called and the result is written back to each level of cache.
[0075] If the number of frames per second is greater than or equal to a, then the Prompt is optimized. The optimization of the Prompt adopts a two-layer optimization mechanism consisting of L1 fast optimization rule base fast optimization and L2 intelligent optimization.
[0076] Example 8 refer to Figure 11 A Prompt optimization and caching device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in Embodiment 1 or 2. Specifically, the Prompt optimization and caching device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The Prompt optimization and caching device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the schematic diagram is merely an example of a Prompt optimization and caching device and does not constitute a limitation on the Prompt optimization and caching device. It may include more or fewer components than illustrated, or combine certain or different components. For example, the Prompt optimization and caching device may also include a power supply component, input / output interfaces, network access devices, a bus, etc.
[0077] Example 9 A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in Embodiment 1 or 2.
[0078] Specifically, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the Prompt optimization and caching device.
[0079] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A Prompt optimization and caching system, characterized in that, include, The Prompt effect tracking module saves the original function and builds a wrapper function. After intercepting the call, it passes the original input parameters to the saved original function for execution and collects Prompt data for reporting. The dynamic model routing module evaluates the complexity of the Prompt data, selects the execution node based on the complexity to obtain a routing score, and executes AI calls based on the routing score. The module retrieves the current number of frames per second. If the number of frames per second is less than a, then cache the Prompt data. Cache the Prompt data by querying the L1 memory cache, L2 local cache and L3 similarity cache in sequence. If a cached response is found, the cached response is returned directly. If a cached response is not found, the AI model is called and the result is written back to each level of cache. If the number of frames per second is greater than or equal to a, then the Prompt is optimized; the optimization of the Prompt adopts a two-layer optimization mechanism consisting of L1 fast optimization and L2 intelligent optimization.
2. The Prompt optimization and caching system according to claim 1, characterized in that, The Prompt effect tracking module saves the original function, builds a wrapper function, intercepts the call, passes the original input parameters to the saved original function for execution, and collects and reports Prompt data, including the following steps: Construct a wrapper function and save the original AI SDK into the wrapper function; The business code calls the wrapper function, the wrapper function's interceptor records the start time, and the wrapper function calls the original AI SDK; After the original AI SDK processes the code and returns a wrapper function, the wrapper function's interceptor records the end time, obtains the total time based on the start and end times, and retrieves the behavioral metrics. The wrapper function compresses the total time and behavioral metrics, writes them to the reporting queue, and returns the corresponding results to the business code.
3. The Prompt optimization and caching system according to claim 1, characterized in that, The dynamic model routing module evaluates the complexity of the Prompt data, selects the execution node based on the complexity to obtain a routing score, and executes AI calls based on the routing score, including the following steps. Receive Prompt data and perform complexity analysis on the Prompt data to obtain the number of words and logical layers; A complexity scoring model is constructed based on the number of words and the number of logical layers. Select the processing model based on the complexity scoring model; Obtain the routing score model and execute AI calls based on the routing score model.
4. The Prompt optimization and caching system according to claim 3, characterized in that, Complexity scoring models include, Score_c = number of words × 0.3 + number of logical layers × 20 + technology density × 50 + task type weight; Routing scoring models include, Score = α × Speed + β × Quality γ×Cost, where Speed is the normalized value of the estimated response speed, Quality is the normalized value of the estimated output quality, Cost is the normalized value of the estimated call cost, and α, β, and γ are configurable weight coefficients.
5. The Prompt optimization and caching system according to claim 1, characterized in that, If the number of frames per second is ≥ a, then the Prompt is optimized. The optimization of the Prompt employs a two-layer optimization mechanism consisting of L1 fast optimization and L2 intelligent optimization, including the following steps: If 20 ≤ frames per second < 30, then perform L1 fast optimization on the Prompt. If the frame rate is ≥30 frames per second, then L1 fast optimization and L2 intelligent optimization will be performed on the Prompt.
6. The Prompt optimization and caching system according to claim 1, characterized in that, If the number of frames per second is less than 'a', then caching is performed on the Prompt. Caching the Prompt involves sequentially querying the L1 memory cache, L2 local cache, and L3 similarity cache. If a match is found, the cached response is returned directly; otherwise, the AI model is invoked and the result is written back to each level of cache. This includes the following steps. Get the current frame rate per second. If the frame rate per second is less than 20, cache the Prompt. Receive a Prompt query request and calculate or read the Prompt fingerprint; Query the L1 memory cache. If a match is found, return the corresponding response directly, and the process ends. If the L1 memory cache is not hit, query the L2 local cache. If it is hit, return the corresponding response and synchronously write it to the L1 memory cache so that subsequent requests can hit the layer faster. If the L2 local cache is not found, query the L3 similarity cache; if the similarity is >95%, return a similar response. If no match is found, the AI model is invoked to generate a new corresponding response, which is then written sequentially to the L3 similarity cache, L2 local cache, and L1 memory cache before finally returning the result.
7. A Prompt optimization and caching system according to claim 6, characterized in that, The L3 similarity cache generates a 64-bit fingerprint by sequentially performing word segmentation, word weight calculation, weighted vector accumulation, and dimensionality reduction. The similarity is judged by Hamming distance, and the similarity threshold is set to 95%.
8. A Prompt optimization and caching method, characterized in that, include, Save the original function and build a wrapper function. After intercepting the call, pass the original input parameters to the saved original function for execution and collect the Prompt data for reporting. The complexity of the Prompt data is evaluated, and the execution node is selected based on the complexity to obtain the routing score. The AI call is then executed based on the routing score. Get the current frames per second. If the number of frames per second is less than a, then cache the Prompt data. Cache the Prompt data by querying the L1 memory cache, L2 local cache and L3 similarity cache in sequence. If a cached response is found, the cached response is returned directly. If a cached response is not found, the AI model is called and the result is written back to each level of cache. If the number of frames per second is greater than or equal to a, then the Prompt is optimized. The optimization of the Prompt adopts a two-layer optimization mechanism consisting of L1 fast optimization rule base fast optimization and L2 intelligent optimization.
9. A Prompt optimization and caching apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.
