Cache optimization method and device for large model API gateway

Abstract

This invention relates to the field of cache optimization technology for large language model API gateways, and particularly to a cache optimization method and apparatus for large language model API gateways. The method includes: receiving an original request sent by a client to a large language model API gateway; identifying and removing dynamic noise fields from the original request to obtain a static request context; performing multi-dimensional semantic standardization processing on the static request context to obtain a standardized intermediate semantic representation; extracting intent, entity, and constraint triples from the intermediate semantic representation and combining them with tenant isolation salts to generate an IEC-structured semantic fingerprint; progressively matching the semantic fingerprint through multi-level caches; and obtaining the cached result if a match is found. This invention solves the problems in existing technologies, such as low hit rates due to the inability to reuse caches, high computational power consumption and latency in vector retrieval caching due to GPU dependence, and the inability to effectively handle dynamic request metadata, which easily leads to cache penetration.

Description

Technical Field

[0001] This invention relates to the field of caching optimization technology for large language model API gateways, and in particular to a caching optimization method and apparatus for large model API gateways. Background Technology

[0002] Large Language Model (LLM) technology is rapidly being deployed, and enterprise-level Application Programming Interface (API) gateways have become the core hub for various businesses to access large model services. The demand for massive, high-concurrency model calls is growing daily. However, large model inference suffers from common industry problems such as high token billing costs, large delays in first character generation, and resource waste caused by repeated requests. Caching technology has become a key means to optimize the performance of large model services and reduce operating costs, and is widely used in the performance tuning of large model API gateways.

[0003] Currently, caching technologies related to large model services can be mainly divided into three categories: first, traditional exact matching caching, which uses algorithms such as SHA256 to perform full-text hashing of the request body to generate cache keys, commonly found in Content Delivery Networks (CDNs) and traditional web gateways; second, semantic vector retrieval caching, which converts requests into vectors and calculates cosine similarity to achieve semantic matching, and has experimental applications in frameworks such as Lang Chain; and third, prefix caching, which optimizes the latency of model first character generation based on KV Cache reuse, and is mostly used in inference frameworks such as vLLM.

[0004] However, full-text hash caching is highly sensitive to differences at the character level in text. Semantically equivalent requests cannot reuse the cache due to slight differences in format, resulting in an extremely low hit rate. Vector retrieval caching requires a lot of computing power for embedding calculations, introducing additional latency and relying on graphics processing unit (GPU) hardware, making it difficult to adapt to latency-sensitive scenarios. All of these solutions lack the ability to effectively process dynamic metadata of requests, which can easily lead to cache penetration. Summary of the Invention

[0005] This invention provides a cache optimization method and apparatus for large-model API gateways to solve the problems in the prior art, such as low hit rate due to the inability to reuse cache, high computational power consumption and high latency of vector retrieval cache and dependence on GPU, and easy cache penetration due to the inability to effectively handle dynamic metadata of requests.

[0006] In a first aspect, embodiments of the present invention provide a caching optimization method for large-model API gateways, including:

[0007] Receive the original request sent by the client to the large model API gateway, identify and remove the dynamic noise fields in the original request to obtain the static request context;

[0008] The static request context is subjected to multi-dimensional semantic standardization processing to obtain a standardized intermediate semantic representation;

[0009] The intent, entity, and constraint triples are extracted from the intermediate semantic representation and combined with the tenant isolation salt value to generate a unique IEC structured semantic fingerprint;

[0010] The semantic fingerprint is progressively matched through a multi-level cache. If a match is found, the cached result is obtained.

[0011] In one possible implementation, after progressively matching the semantic fingerprint through multi-level caching, the method further includes:

[0012] If no match is found, the semantic fingerprint is forwarded to the large model for inference, and the inference result is synchronized to the corresponding cache.

[0013] In one possible implementation, after receiving the original request sent by the client to the large model API gateway, identifying and removing dynamic noise fields from the original request to obtain the static request context, the method further includes:

[0014] Store the dynamic noise field in the request-independent context;

[0015] If a match is found, the cached result is retrieved; if no match is found, the semantic fingerprint is forwarded to the large model for inference, and the inference result is synchronized to the corresponding cache. The process also includes:

[0016] The dynamic noise field in the request-independent context is injected back into the cached result or the inference result to form a complete response and return it to the client.

[0017] In one possible implementation, the system receives the original request sent by the client to the large model API gateway, identifies and removes dynamic noise fields from the original request, and obtains the static request context, including:

[0018] Within a preset time window, the inbound traffic to the API gateway is sampled in real time to obtain each request field and its corresponding field value;

[0019] Calculate the normalized information entropy of each request field;

[0020] If the normalized information entropy of the current request field is higher than a preset threshold, and the field value corresponding to the current request field matches the high-entropy noise regularization feature library built into the system, then the current request field is marked as a dynamic noise field.

[0021] Based on the above method of determining that the current request field is marked as a dynamic noise field, all request fields are marked accordingly;

[0022] When a new request arrives, all marked dynamic noise fields are filtered out to obtain the static request context.

[0023] In one possible implementation, the static request context undergoes multi-dimensional semantic normalization processing to obtain a standardized intermediate semantic representation, including:

[0024] Basic character processing is performed on the static request context to eliminate format differences;

[0025] Based on the system's built-in industry domain dictionary, entities with non-standard representations in the static request context after basic character processing are mapped to standard entities.

[0026] Syntactic pruning is performed on the natural language in the static request context after mapping to obtain a standardized intermediate semantic representation.

[0027] In one possible implementation, an intent, entity, and constraint triple is extracted from the intermediate semantic representation and combined with a tenant isolation salt to generate a unique Intent-Entity-Constraint (IEC) structured semantic fingerprint, including:

[0028] The intermediate semantic representation is mapped into discrete intent instructions;

[0029] Semantic entities are extracted from the intermediate semantic representation, and all extracted entities are sorted to obtain a standardized entity set;

[0030] Extract the conditions that constrain the model reasoning from the intermediate semantic representation, and sort all the extracted constraints to obtain a set of constraints.

[0031] The intent instruction, the standardized entity set, the constraint set, and the tenant isolation salt value are concatenated according to a preset format to obtain a concatenated string;

[0032] A hash algorithm is used to calculate the concatenated string to generate a unique IEC structured semantic fingerprint.

[0033] In one possible implementation, the semantic fingerprint is progressively matched sequentially through a multi-level cache, including:

[0034] The semantic fingerprint is matched against the caches sorted by priority from high to low in sequence. If the previous level cache is not hit, the matching process proceeds to the next level cache.

[0035] One possible implementation also includes:

[0036] Maintain the version compatibility matrix of the system Prompt;

[0037] After progressively matching the semantic fingerprint through multiple levels of caching, and before forwarding the semantic fingerprint to the large model for inference, the process further includes:

[0038] If no match is found, the version compatibility matrix is ​​queried. If the current version is logically compatible with the historical version, the cache of the historical version is searched and the cache result is obtained. At the same time, the cache of the current version is asynchronously refreshed.

[0039] One possible implementation also includes:

[0040] Randomly select a preset proportion of cache hit requests to perform origin verification, compare the cache results with the real-time inference results of the large model, and verify the accuracy of the cached content.

[0041] Secondly, embodiments of the present invention provide a cache optimization device for large-model API gateways, comprising:

[0042] The parameter stripping module is used to receive the original request sent by the client to the large model API gateway, identify and strip the dynamic noise fields in the original request to obtain the static request context;

[0043] The semantic normalization module is used to perform multi-dimensional semantic normalization processing on the static request context to obtain a standardized intermediate semantic representation;

[0044] The fingerprint generation module is used to extract intent, entity, and constraint triples from the intermediate semantic representation and combine them with tenant isolation salt values ​​to generate a unique IEC structured semantic fingerprint.

[0045] The hierarchical matching module is used to progressively match the semantic fingerprint through multi-level caches. If a match is found, the cached result is obtained.

[0046] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0047] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect or any possible implementation thereof.

[0048] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect or any possible implementation thereof.

[0049] This invention provides a caching optimization method and apparatus for large model API gateways. By receiving the original request sent by the client to the large model API gateway, identifying and removing dynamic noise fields in the original request, a static request context is obtained. This technical feature automatically removes dynamic metadata that has no impact on model inference, avoids cache penetration caused by dynamic parameters changing the cache key, and improves the effectiveness and stability of the caching mechanism in the production environment.

[0050] Multi-dimensional semantic standardization processing is performed on the static request context to obtain a standardized intermediate semantic representation, thereby eliminating character-level differences such as text spaces, punctuation, capitalization, expression differences, and JSON field order. This enables semantically equivalent requests to be cached reusably, significantly improving the cache hit rate and solving the problem that traditional full-text hash caching is too sensitive to text differences.

[0051] This technology extracts intent, entity, and constraint triples from intermediate semantic representations and combines them with tenant isolation salts to generate unique IEC structured semantic fingerprints. This feature transforms unstructured natural language into structured, unique semantic identifiers, achieving lightweight CPU computation and millisecond-level semantic matching, eliminating GPU dependence, and reducing the overhead of fuzzy matching embedding and the risk of semantic drift. At the same time, it achieves multi-tenant secure isolation through tenant isolation salts, avoiding cross-tenant cache leaks.

[0052] Semantic fingerprints are progressively matched through multi-level caching. If a match is found, the cached result is retrieved, and the search is performed in stages according to speed and accuracy. This ensures a high hit rate while minimizing overall response latency. If the cache is hit, the result can be returned directly without calling large model inference, which significantly reduces the cost of the computing unit token and the latency of the first character response, and reduces the waste of resources caused by repeated requests. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating the implementation of a caching optimization method for large-model API gateways provided in this embodiment of the invention.

[0055] Figure 2 This is a schematic diagram illustrating the process of identifying and stripping dynamic noise fields provided in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram illustrating the acquisition of IEC structured semantic fingerprints provided in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the caching optimization device for large-model API gateways provided in an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0059] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0060] Example 1

[0061] See Figure 1 The document illustrates a flowchart of a caching optimization method for large-model API gateways provided by an embodiment of the present invention, detailed below:

[0062] Step 101: Receive the original request sent by the client to the large model API gateway, identify and remove the dynamic noise fields in the original request to obtain the static request context.

[0063] This step is used to receive the original request sent by the client to the large model API gateway, automatically identify and remove dynamic noise fields in the original request that have no impact on model inference, obtain a static request context that can be used for subsequent semantic processing, and at the same time realize dynamic parameter management that is transparent and imperceptible to the business layer.

[0064] In one embodiment, see Figure 2 As shown, the system receives the original request sent by the client to the large model API gateway, identifies and removes the dynamic noise fields from the original request to obtain the static request context, which may include:

[0065] Within a preset time window, the inbound traffic to the API gateway is sampled in real time to obtain each request field and its corresponding field value;

[0066] Calculate the normalized information entropy of each request field;

[0067] If the normalized information entropy of the current request field is higher than the preset threshold, and the field value corresponding to the current request field matches the high-entropy noise regularity feature library built into the system, then the current request field will be marked as a dynamic noise field.

[0068] Based on the above method of determining the current request field as a dynamic noise field, all request fields are marked accordingly;

[0069] When a new request arrives, all marked dynamic noise fields are filtered out to obtain the static request context.

[0070] The gateway first receives raw API requests from clients. These requests typically carry dynamic metadata fields unrelated to the business logic, including but not limited to request_id, trace_id, timestamp, and req_trace. The system processes the raw API requests using an adaptive parameter stripping algorithm based on information entropy: a sliding time window is maintained in the API gateway's memory to monitor the discrete distribution of JSON field values ​​in the incoming requests in real time, and the normalized information entropy is calculated for each field. Optionally, the preset time window can be set to a specific duration as needed; for example, a preset time window could be set to 5 seconds.

[0071] In this embodiment, the normalized information entropy is calculated using existing methods, which will not be described in detail here.

[0072] Optionally, the preset threshold is adaptively configured. The system periodically analyzes historical traffic logs through offline computing jobs, calculates the P99 quantile of the information entropy distribution curve for each field, and dynamically distributes the preset threshold configuration to the gateway memory. The default initial preset threshold is generally set to 0.78.

[0073] Optionally, the high-entropy noise regularity feature library is the core discrimination rule in the entire dynamic parameter stripping process. High entropy refers to the field value changing extremely quickly and without any pattern, which is useless for semantic understanding. Noise refers to content that is meaningless for large model inference and cache matching. Keeping it will only interfere with the generation of cache keys. Therefore, the high-entropy noise regularity feature library is a "list of meaningless dynamic value formats" defined by regular expressions and built into the system.

[0074] The high-entropy noise regularization feature library contains format features of typical dynamic parameters such as Universally Unique Identifiers (UUIDs) and Unix timestamps. Understandably, when determining whether a currently requested field is a dynamic noise field, in addition to checking if its normalized entropy exceeds a preset threshold, a "format validation" layer is needed. This involves checking whether the specific value of the field conforms to the predefined format rules for "high-entropy noise" in the system. The purpose of this step in determining whether a currently requested field is a dynamic noise field is to accurately remove the real noise and avoid mistakenly deleting semantically meaningful high-entropy fields.

[0075] like Figure 2 As shown, the entropy analyzer returns the marking results, which are labeled as dynamic noise fields, to the API gateway. Based on the marking results, the API gateway interceptor automatically removes the dynamic noise fields before generating cache keys when it receives a new request. The remaining valid business content after removal is combined into a static request context for subsequent semantic normalization and fingerprint generation.

[0076] Understandably, if the normalized information entropy of the current requested field is not higher than the preset threshold, then the current requested field is determined to be business content and needs to be retained.

[0077] This step eliminates the need for manual maintenance of complex blacklist and whitelist configurations. It can adaptively identify dynamic parameters in unknown protocols, thereby preventing cache penetration caused by dynamic noise from the root and improving the anti-interference capability and operational efficiency of the caching system.

[0078] In one embodiment, after receiving the original request sent by the client to the large model API gateway, identifying and removing dynamic noise fields from the original request, and obtaining the static request context, the process may further include:

[0079] Store the dynamic noise field in the request-independent context;

[0080] The API gateway temporarily stores the stripped dynamic noise fields in the independent context space of the current request so that a transparent back injection operation can be performed after a cache hit, ensuring the integrity of business fields.

[0081] Step 102: Perform multi-dimensional semantic standardization processing on the static request context to obtain a standardized intermediate semantic representation.

[0082] This step is used to perform multi-dimensional and multi-granular semantic standardization processing on the static request context obtained in step 101, eliminate non-semantic differences caused by text format, expression mode and syntactic structure, and output a unified and standardized intermediate semantic representation to provide stable input for subsequent IEC structured fingerprint generation.

[0083] In one embodiment, performing multi-dimensional semantic normalization processing on the static request context to obtain a standardized intermediate semantic representation may include:

[0084] Perform basic character processing on the static request context to eliminate format differences;

[0085] Based on the system's built-in industry domain dictionary, entities with non-standard representations in the static request context after basic character processing are mapped to standard entities.

[0086] Syntactic pruning is performed on the natural language in the static request context after mapping to obtain a standardized intermediate semantic representation.

[0087] The system performs three levels of normalization processing sequentially through its built-in hybrid lightweight semantic normalization engine:

[0088] First, perform character-level regular expression replacement, which means performing basic character processing on the static request context to eliminate character-level differences such as extra spaces, different punctuation marks, case differences, and different JSON field orders in the static request context, and unify the basic text format;

[0089] Secondly, the domain dictionary synonym mapping is performed. Using the system's built-in industry domain dictionary and entity attribute graph, entities with non-standard expressions are mapped to standard entities in real time to achieve entity normalization. For example, "Py3" is uniformly mapped to "Python".

[0090] Finally, syntactic pruning is performed to remove modal particles, redundant modifiers, and other content that does not affect the core semantics, simplifying the syntactic structure while retaining the true intent and key information of the request.

[0091] After the above three-level continuous processing, the static request context is transformed into a standardized intermediate semantic representation with uniform format, standard entities, and pure semantics. This enables semantically equivalent but differently expressed requests to obtain consistent semantic expression, solving the problem that traditional full-text hash caching cannot reuse the cache due to minor differences in text, and significantly improving the accuracy of semantic matching and cache hit rate.

[0092] Step 103: Extract the intent, entity, and constraint triples from the intermediate semantic representation and combine them with the tenant isolation salt value to generate a unique IEC structured semantic fingerprint.

[0093] This step is used to extract intent, entity, and constraint triplet information from the standardized intermediate semantic representation obtained in step 102, and combine it with the tenant isolation salt value for hash encoding to generate a unique, fixed-length IEC structured semantic fingerprint, which serves as the core index key for subsequent cache matching.

[0094] In one embodiment, such as Figure 3As shown, extracting intent, entity, and constraint triples from the intermediate semantic representation and combining them with tenant isolation salt values ​​to generate a unique IEC structured semantic fingerprint can include:

[0095] Map intermediate semantic representations to discrete intent commands;

[0096] Extract semantic entities from the intermediate semantic representation and sort all extracted entities to obtain a standardized entity set;

[0097] Extract the conditions that constrain the model reasoning from the intermediate semantic representation, and sort all the extracted constraints to obtain the constraint set.

[0098] The intent command, standardized entity set, constraint set, and tenant isolation salt value are concatenated according to a preset format to obtain the concatenated string;

[0099] A hash algorithm is used to calculate the concatenated string and generate a unique IEC structured semantic fingerprint.

[0100] Optionally, triplet extraction is performed first: using a lightweight intent classifier based on FastText, the intermediate semantic representation is mapped to discretized action instructions to obtain the intent; using an entity extractor based on the AC automaton, key entities are extracted from the text of the intermediate semantic representation and sorted, for example, by lexicographical order, while non-standard expressions are aligned to standard nodes with the help of an industry domain dictionary and entity attribute graph to obtain a set of standardized entities; conditions that limit the model's inference results are extracted from the text of the intermediate semantic representation, and all extracted constraints are sorted by lexicographical order to form a set of constraints.

[0101] The preset format used during concatenation can be a fixed-format binary serialization concatenation. Optionally, the concatenated string can be... ,in, This indicates string concatenation. Indicates intentional instructions, Represents a standardized set of entities. Represents the sorted, standardized set of entities. Represents a set of constraints. This represents the sorted set of constraints. This indicates a string concatenation operation.

[0102] After extraction, the intent command, standardized entity set, and constraint set are serialized and concatenated in a fixed format and mixed with tenant isolation salt value to achieve multi-tenant data isolation.

[0103] The concatenated string is then encoded using the SHA-256 hash algorithm to generate a 256-bit fixed-length hash value, which is the IEC structured semantic fingerprint. This IEC structured semantic fingerprint is generated based on lightweight CPU computation, which can transform complex semantic matching into deterministic hash lookup. While ensuring matching accuracy, it controls the retrieval latency to the millisecond level, and does not rely on GPU hardware, avoiding the high overhead and semantic drift problems caused by traditional vector semantic caching.

[0104] Step 104: Semantic fingerprints are progressively matched through multi-level caches. If a match is found, the cached result is obtained.

[0105] This step is used to send the IEC structured semantic fingerprint generated in step 103 into a multi-level cache structure according to a preset priority to perform progressive lookup. When a match is found, the corresponding cached result is directly obtained without forwarding it to the large model for inference, thereby reducing response latency and calling costs.

[0106] In one embodiment, progressively matching semantic fingerprints through multi-level caching may include:

[0107] The semantic fingerprint is matched against the caches sorted by priority from high to low in turn. If the previous level cache is not matched, the next level cache is matched.

[0108] Optionally, the system can adopt a four-level hierarchical progressive cache matching strategy from L1 to L4. Each level of cache is executed sequentially according to the response speed from high to low and the matching accuracy from precise to fuzzy. If the previous level cache is not hit, the system will automatically enter the next level cache for search.

[0109] The L1 level cache is a full-text hash exact match cache, which is used to directly retrieve the cached result if a match is found, handle completely duplicate requests, and has a matching latency of less than 1ms. If a match is not found, the cache will be searched in the next level cache.

[0110] The L2 cache is an IEC structured fingerprint matching cache. It performs deterministic hash lookup based on the IEC structured semantic fingerprint, that is, it queries the cache value corresponding to the IEC structured semantic fingerprint in Redis. If a match is found, the cache result is retrieved; if no match is found, the next level cache is searched. This is used to handle synonym rewrite requests. The matching latency is about 5ms-10ms.

[0111] Level 3 cache is a vector similarity matching cache that uses vector cosine similarity to achieve fuzzy semantic matching. It is used to handle long-tail fuzzy requests, with a matching latency of approximately 50ms. This level of cache is optional to enable. If a match is found, the cached result is retrieved; if no match is found or this level of cache is not enabled, the search proceeds to the next level of cache.

[0112] The L4 level cache is a prefix cache based on a sliding window. For very long documents, the System Prompt is used as a common prefix for block fingerprint matching. Only the part of the user's new query is calculated, and most of the pre-calculated results can be reused. If a match is found, part of the cached results are obtained. If no match is found, the semantic fingerprint is forwarded to the large model for inference, and the inference results are synchronized to the corresponding cache.

[0113] In one embodiment, during the cache matching process, the system also includes a built-in consistency verification module. This module randomly selects a preset proportion of cache hit requests to perform origin-based verification, comparing the cached results with the real-time inference results of the large model to verify the accuracy of the cached content. Optionally, the preset proportion can be set according to requirements; for example, it can be set to 1%.

[0114] Meanwhile, the system is equipped with a version compatibility manager to maintain the version compatibility matrix of the system Prompt;

[0115] After progressively matching the semantic fingerprint through multiple levels of caching, and before forwarding the semantic fingerprint to the large model for inference, the following steps are also included:

[0116] If no match is found, the version compatibility matrix is ​​queried. If the current version is logically compatible with the historical version, the cache of the historical version is searched and the cache result is obtained. At the same time, the cache of the current version is asynchronously refreshed to avoid cache avalanche and traffic penetration caused by the Prompt iteration update.

[0117] When the IEC structured semantic fingerprint is matched and hit in any level of cache, the corresponding cache result is immediately obtained and the subsequent dynamic parameter injection process begins.

[0118] In one embodiment, if a match is found, the cached result is obtained; if a match is not found, the semantic fingerprint is forwarded to the large model for inference, and the inference result is synchronized to the corresponding cache. The method further includes:

[0119] The dynamic noise fields in the request's independent context are injected back into the cached or inference results to restore the complete business parameters of the request, forming a complete response and returning it to the client. The cached results with completed dynamic parameter injection are returned to the client, and a custom flag (such as x-smart-iec-hit: true, x-cache-degraded: true) is added to the response header to indicate the cache processing status.

[0120] The following sections will further explain this application through different application scenarios.

[0121] Example 2

[0122] The application scenario involves an AI programming assistant user initiating a code generation request carrying a random tracking number. The original input is:

[0123] {

[0124] "trace_id": "a1b2-c3d4-e5f6", / / Dynamic noise

[0125] "model": "gpt-4",

[0126] "messages": [

[0127] {"role": "user", "content": "Help me write a Python

[0128] quicksort, no comments please."} ]

[0130] }

[0131] The system identifies that the normalized information entropy of trace_id is higher than the preset threshold, and the field value corresponding to the current request field matches the system's built-in high-entropy noise regularity feature library. It then extracts the value and stores it in Context["trace_id"] ="a1b2...".

[0132] Static request context: Retains the model and messages content for subsequent fingerprint calculation.

[0133] Static request context: "help me write python quicksort no comments" -> "writequicksort in python" (via synonym mapping).

[0134] Extracting IEC triples:

[0135] Intent: CODE_GEN (confidence level 0.98);

[0136] Entity: ["python", "quicksort"] (sorted lexicographically, Py3 mapped to python);

[0137] Constraint: ["no_comment"];

[0138] IEC structured semantic fingerprint generation:

[0139] Concatenate: "CODE_GEN|python,quicksort|no_comment|TenantID:888"

[0140] Calculate the hash value: Generate the SHA-256 digest 0x7f8a...3b1c.

[0141] Cache matching and response:

[0142] L2 hit: The corresponding cached value for this hash exists in Redis.

[0143] Note: Read Context["trace_id"] and insert it back into the response header or body.

[0144] Final output: The client receives a complete code response with the correct trace_id.

[0145] Example 3

[0146] Perform noise resistance testing under high concurrency. The application scenario is: JMeter simulates 10,000 QPS requests, with each request carrying a random request_id.

[0147] Execution process: The entropy analyzer identifies request_id as a high-entropy field within 1 second. Subsequent requests automatically remove this field from the key calculation, and the cache hit rate recovers from 0% to 46% (baseline repetition rate).

[0148] Example 4

[0149] System observability and feature verification. The application scenario is: operations and maintenance personnel use external black-box probing to verify whether the system correctly performs semantic normalization and dynamic parameter caching.

[0150] Response header signature verification: Send a request containing a random noise parameter k=random_uuid. The system response header should contain custom flags such as x-smart-iec-hit: true or x-cache-degraded: true, indicating that the internal dynamic parameter stripping is normal operation.

[0151] Semantic delay feature verification: Construct queries with the same semantics but vastly different text structures (such as out-of-order entities). If the system can return the same results with a millisecond delay, it indicates that its structured fingerprint mechanism has taken effect, rather than relying on high-latency embeddings or simple full-text hashes.

[0152] Example 5

[0153] Verification of the core algorithm implementation details.

[0154] Asynchronous negative feedback link: The system adopts a bypass analysis architecture. The gateway writes events that are cache hits but subsequently returned as errors by the user into a non-blocking circular buffer, which is then consumed asynchronously by a background coroutine. When the false positive rate of a certain field exceeds the safety threshold, the field is automatically added to a temporary whitelist and an alarm is triggered. This does not block the main cache link, ensuring high availability of the system and preventing it from being blocked by complex statistical logic.

[0155] The embodiments of the present invention can be applied in the following scenarios:

[0156] Enterprise-grade LLM transit gateway: Provides unified LLM access and billing optimization within the enterprise;

[0157] AI programming assistant (Copilot): Handles massive amounts of highly repetitive code completion requests;

[0158] Intelligent customer service system: efficiently reuses answers to frequently asked questions (FAQs).

[0159] This application compares and verifies the performance of traditional hash caching, vector semantic caching, and the proposed solution (Smart-IEC) across multiple dimensions. The results show that traditional hash caching has the worst noise resistance, is highly sensitive to character-level differences, only supports literal semantic matching, and has a computation latency of less than 1ms but lacks a version management mechanism. Vector semantic caching, while possessing some noise resistance and fuzzy semantic matching granularity, suffers from high computation latency (20-100ms), reliance on GPU / NPU hardware, and lack of effective version management. In contrast, the proposed Smart-IEC solution, through an entropy-adaptive stripping mechanism, exhibits extremely strong noise resistance, supports high-precision semantic matching at the intent-entity structure level, has an inference latency of approximately 10ms in a pure CPU environment, and innovatively introduces a version chain compatibility matrix for version management. Overall, it achieves multiple advantages including low latency, low hardware dependency, high noise resistance, and efficient version management.

[0160] This invention provides a caching optimization method for large model API gateways. By receiving the original request sent by the client to the large model API gateway, identifying and removing dynamic noise fields in the original request, a static request context is obtained. This technical feature automatically removes dynamic metadata that has no impact on model inference, avoids cache penetration caused by dynamic parameters changing the cache key, and improves the effectiveness and stability of the caching mechanism in the production environment.

[0161] Multi-dimensional semantic standardization processing is performed on the static request context to obtain a standardized intermediate semantic representation, thereby eliminating character-level differences such as text spaces, punctuation, capitalization, expression differences, and JSON field order. This enables semantically equivalent requests to be cached reusably, significantly improving the cache hit rate and solving the problem that traditional full-text hash caching is too sensitive to text differences.

[0162] This technology extracts intent, entity, and constraint triples from intermediate semantic representations and combines them with tenant isolation salts to generate unique IEC structured semantic fingerprints. This feature transforms unstructured natural language into structured, unique semantic identifiers, achieving lightweight CPU computation and millisecond-level semantic matching, eliminating GPU dependence, and reducing embedding overhead and semantic drift risk. At the same time, it achieves multi-tenant secure isolation through tenant isolation salts, avoiding cross-tenant cache leaks.

[0163] Semantic fingerprints are progressively matched through multi-level caching. If a match is found, the cached result is retrieved and searched in stages according to speed and accuracy. This ensures a high hit rate while minimizing overall response latency. Cache hits can directly return results without calling large model inference, significantly reducing token costs and first-character response latency, and minimizing resource waste caused by duplicate requests.

[0164] In this embodiment of the invention, by maintaining the System Prompt version compatibility matrix, when the current version cache misses, the cache result of the logically compatible historical version can be downgraded and reused. This avoids the full cache from becoming invalid and traffic from directly penetrating to the large model inference layer due to the iteration of the Prompt version, thus ensuring the stability and availability of the system during iterative updates.

[0165] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0166] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0167] Figure 4 The diagram shows a schematic of a cache optimization device for large-model API gateways according to an embodiment of the present invention. For ease of explanation, only the parts relevant to the embodiment of the present invention are shown, and are described in detail below:

[0168] like Figure 4 As shown, the cache optimization device for large model API gateway includes: parameter stripping module 41, semantic normalization module 42, fingerprint generation module 43, and hierarchical matching module 44.

[0169] The parameter stripping module 41 is used to receive the original request sent by the client to the large model API gateway, identify and strip the dynamic noise fields in the original request to obtain the static request context.

[0170] Semantic normalization module 42 is used to perform multi-dimensional semantic normalization processing on the static request context to obtain a standardized intermediate semantic representation;

[0171] The fingerprint generation module 43 is used to extract the intent, entity and constraint triples from the intermediate semantic representation and combine them with the tenant isolation salt value to generate a unique IEC structured semantic fingerprint.

[0172] The hierarchical matching module 44 is used to progressively match semantic fingerprints through multi-level caches. If a match is found, the cached result is obtained.

[0173] In one possible implementation, after progressively matching the semantic fingerprint through multi-level caching, the hierarchical matching module 44 is further used for:

[0174] If no match is found, the semantic fingerprint is forwarded to the large model for inference, and the inference result is synchronized to the corresponding cache.

[0175] In one possible implementation, after receiving the original request sent by the client to the large model API gateway, identifying and removing dynamic noise fields from the original request to obtain the static request context, the parameter stripping module 41 is also used for:

[0176] Store the dynamic noise field in the request-independent context;

[0177] The hierarchical matching module 44 retrieves the cached result if a match is found, or forwards the semantic fingerprint to the large model for inference if a match is not found, and synchronizes the inference result to the corresponding cache. It is also used for:

[0178] The dynamic noise field in the request-independent context is injected back into the cached result or inference result to form a complete response and return it to the client.

[0179] In one possible implementation, the parameter stripping module 41 receives the original request sent by the client to the large model API gateway, identifies and strips the dynamic noise fields from the original request, and obtains the static request context, which is then used for:

[0180] Within a preset time window, the inbound traffic to the API gateway is sampled in real time to obtain each request field and its corresponding field value;

[0181] Calculate the normalized information entropy of each request field;

[0182] If the normalized information entropy of the current request field is higher than the preset threshold, and the field value corresponding to the current request field matches the high-entropy noise regularity feature library built into the system, then the current request field will be marked as a dynamic noise field.

[0183] Based on the above method of determining the current request field as a dynamic noise field, all request fields are marked accordingly;

[0184] When a new request arrives, all marked dynamic noise fields are filtered out to obtain the static request context.

[0185] In one possible implementation, the semantic normalization module 42 performs multi-dimensional semantic normalization processing on the static request context to obtain a standardized intermediate semantic representation, which is then used for:

[0186] Perform basic character processing on the static request context to eliminate format differences;

[0187] Based on the system's built-in industry domain dictionary, entities with non-standard representations in the static request context after basic character processing are mapped to standard entities.

[0188] Syntactic pruning is performed on the natural language in the static request context after mapping to obtain a standardized intermediate semantic representation.

[0189] In one possible implementation, when the fingerprint generation module 43 extracts the intent, entity, and constraint triples from the intermediate semantic representation and combines them with the tenant isolation salt to generate a unique IEC structured semantic fingerprint, it is used for:

[0190] Map intermediate semantic representations to discrete intent commands;

[0191] Extract semantic entities from the intermediate semantic representation and sort all extracted entities to obtain a standardized entity set;

[0192] Extract the conditions that constrain the model reasoning from the intermediate semantic representation, and sort all the extracted constraints to obtain the constraint set.

[0193] The intent command, standardized entity set, constraint set, and tenant isolation salt value are concatenated according to a preset format to obtain the concatenated string;

[0194] A hash algorithm is used to calculate the concatenated string and generate a unique IEC structured semantic fingerprint.

[0195] In one possible implementation, when the hierarchical matching module 44 performs progressive matching of semantic fingerprints through multi-level caches, it is used for:

[0196] The semantic fingerprint is matched against the caches sorted by priority from high to low in turn. If the previous level cache is not matched, the next level cache is matched.

[0197] In one possible implementation, the hierarchical matching module 44 is also used for:

[0198] Maintain the version compatibility matrix of the system Prompt;

[0199] After progressively matching the semantic fingerprint through multi-level caching, the hierarchical matching module 44 is also used before forwarding the semantic fingerprint to the large model for inference:

[0200] If no match is found, query the version compatibility matrix. If the current version is logically compatible with the historical version, search the cache of the historical version and obtain the cache result, while asynchronously triggering a cache refresh for the current version.

[0201] In one possible implementation, the hierarchical matching module 44 is also used for:

[0202] Randomly select a preset proportion of cache hit requests to perform origin verification, compare the cache results with the real-time inference results of the large model, and verify the accuracy of the cached content.

[0203] The above embodiments provide a cache optimization device for large model API gateways. The device receives the original request sent by the client to the large model API gateway through a parameter stripping module, identifies and strips the dynamic noise fields in the original request to obtain the static request context. This technical feature automatically removes dynamic metadata that has no impact on model inference, avoids cache penetration caused by dynamic parameters changing the cache key, and improves the effectiveness and stability of the caching mechanism in the production environment.

[0204] The semantic normalization module performs multi-dimensional semantic standardization processing on the static request context to obtain a standardized intermediate semantic representation, thereby eliminating character-level differences such as text spaces, punctuation, capitalization, expression differences, and JSON field order. This enables semantically equivalent requests to be cached reusably, significantly improving the cache hit rate and solving the problem that traditional full-text hash caching is too sensitive to text differences.

[0205] The fingerprint generation module extracts intent, entity, and constraint triples from the intermediate semantic representation and combines them with tenant isolation salt to generate a unique IEC structured semantic fingerprint. This technology transforms unstructured natural language into a structured and unique semantic identifier, achieving lightweight CPU computation and millisecond-level semantic matching, eliminating GPU dependence, and reducing embedding overhead and semantic drift risk. At the same time, it achieves multi-tenant security isolation through tenant isolation salt to avoid cross-tenant cache leakage.

[0206] The hierarchical matching module performs progressive matching of semantic fingerprints through multi-level caches. If a match is found, the cached result is retrieved, and the search is performed hierarchically according to speed and accuracy. This ensures a high hit rate while minimizing the overall response latency. If the cache is hit, the result can be returned directly without calling large model inference, which significantly reduces token cost and first-character response latency, and reduces resource waste caused by repeated requests.

[0207] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. For example... Figure 5 As shown, the electronic device 5 of this embodiment includes a processor 50 and a memory 51. The memory 51 stores a computer program 52. When the processor 50 executes the computer program 52, it implements the steps in the various method embodiments described above. Alternatively, when the processor 50 executes the computer program 52, it implements the functions of each module / unit in the various device embodiments described above.

[0208] For example, computer program 52 may be divided into one or more modules / units, which are stored in memory 51 and executed by processor 50 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 52 in electronic device 5.

[0209] Electronic device 5 may include, but is not limited to, processor 50 and memory 51. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 5 and does not constitute a limitation on electronic device 5. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 5 may also include input / output devices, network access devices, buses, etc.

[0210] The processor 50 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0211] The memory 51 can be an internal storage unit of the electronic device 5, such as a hard disk or RAM. The memory 51 can also be an external storage device of the electronic device 5, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 51 can include both internal and external storage units of the electronic device 5. The memory 51 is used to store the computer program 52 and other programs and data required by the electronic device 5. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0212] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0213] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0214] This invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0215] Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0216] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0217] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A caching optimization method for API gateways targeting large-scale application programming interfaces (APIs), characterized in that, include: Receive the original request sent by the client to the large model API gateway, identify and remove the dynamic noise fields in the original request to obtain the static request context; The static request context is subjected to multi-dimensional semantic standardization processing to obtain a standardized intermediate semantic representation; Extract the intent, entity, and constraint triples from the intermediate semantic representation and combine them with the tenant isolation salt value to generate a unique intent-entity-constraint IEC structured semantic fingerprint; The semantic fingerprint is progressively matched through a multi-level cache. If a match is found, the cached result is obtained.

2. The caching optimization method for large-model API gateways according to claim 1, characterized in that, After progressively matching the semantic fingerprint through multiple levels of caching, the process further includes: If no match is found, the semantic fingerprint is forwarded to the large model for inference, and the inference result is synchronized to the corresponding cache.

3. The caching optimization method for large-model API gateways according to claim 2, characterized in that, After receiving the original request sent by the client to the large model API gateway, identifying and removing the dynamic noise fields from the original request to obtain the static request context, the process also includes: Store the dynamic noise field in the request-independent context; If a match is found, the cached result is retrieved; if no match is found, the semantic fingerprint is forwarded to the large model for inference, and the inference result is synchronized to the corresponding cache. The process also includes: The dynamic noise field in the request-independent context is injected back into the cached result or the inference result to form a complete response and return it to the client.

4. The caching optimization method for large-model API gateways according to any one of claims 1-3, characterized in that, Receive the original request sent by the client to the large model API gateway, identify and remove the dynamic noise fields in the original request to obtain the static request context, including: Within a preset time window, the inbound traffic to the API gateway is sampled in real time to obtain each request field and its corresponding field value; Calculate the normalized information entropy of each request field; If the normalized information entropy of the current request field is higher than a preset threshold, and the field value corresponding to the current request field matches the high-entropy noise regularization feature library built into the system, then the current request field is marked as a dynamic noise field. Based on the above method of determining that the current request field is marked as a dynamic noise field, all request fields are marked accordingly; When a new request arrives, all marked dynamic noise fields are filtered out to obtain the static request context.

5. The caching optimization method for large-model API gateways according to claim 4, characterized in that, The static request context is subjected to multi-dimensional semantic standardization processing to obtain a standardized intermediate semantic representation, including: Basic character processing is performed on the static request context to eliminate format differences; Based on the system's built-in industry domain dictionary, entities with non-standard representations in the static request context after basic character processing are mapped to standard entities. Syntactic pruning is performed on the natural language in the static request context after mapping to obtain a standardized intermediate semantic representation.

6. The caching optimization method for large-model API gateways according to claim 5, characterized in that, The intent, entity, and constraint triples are extracted from the intermediate semantic representation, and combined with the tenant isolation salt value to generate a unique IEC structured semantic fingerprint, including: The intermediate semantic representation is mapped into discrete intent instructions; Semantic entities are extracted from the intermediate semantic representation, and all extracted entities are sorted to obtain a standardized entity set; Extract the conditions that constrain the model reasoning from the intermediate semantic representation, and sort all the extracted constraints to obtain a set of constraints. The intent instruction, the standardized entity set, the constraint set, and the tenant isolation salt value are concatenated according to a preset format to obtain a concatenated string; A hash algorithm is used to calculate the concatenated string to generate a unique IEC structured semantic fingerprint.

7. The caching optimization method for large-model API gateways according to claim 6, characterized in that, The semantic fingerprint is progressively matched sequentially through multi-level caching, including: The semantic fingerprint is matched against the caches sorted by priority from high to low in sequence. If the previous level cache is not hit, the matching process proceeds to the next level cache.

8. The caching optimization method for large-model API gateways according to claim 7, characterized in that, Also includes: Maintain the version compatibility matrix for the system prompt word; After progressively matching the semantic fingerprint through multiple levels of caching, and before forwarding the semantic fingerprint to the large model for inference, the process further includes: If no match is found, the version compatibility matrix is ​​queried. If the current version is logically compatible with the historical version, the cache of the historical version is searched and the cache result is obtained. At the same time, the cache of the current version is asynchronously refreshed.

9. The caching optimization method for large-model API gateways according to claim 8, characterized in that, Also includes: Randomly select a preset proportion of cache hit requests to perform origin verification, compare the cache results with the real-time inference results of the large model, and verify the accuracy of the cached content.

10. A caching optimization device for large-model API gateways, characterized in that, include: The parameter stripping module is used to receive the original request sent by the client to the large model API gateway, identify and strip the dynamic noise fields in the original request to obtain the static request context; The semantic normalization module is used to perform multi-dimensional semantic normalization processing on the static request context to obtain a standardized intermediate semantic representation; The fingerprint generation module is used to extract intent, entity, and constraint triples from the intermediate semantic representation and combine them with tenant isolation salt values ​​to generate a unique IEC structured semantic fingerprint. The hierarchical matching module is used to progressively match the semantic fingerprint through multi-level caches. If a match is found, the cached result is obtained.

Images