A prompt word cache optimization method and device, equipment and storage medium

Abstract

The application discloses a prompt word cache optimization method and device, equipment and a storage medium. The method comprises the following steps: acquiring skill subscription data associated with a current workspace; the skill subscription data contains index information of a skill package; constructing a stable prefix of a prompt word according to the index information of each skill package and performing hash calculation to obtain a stable prefix hash value; matching the stable prefix hash value with a pre-stored local hash value; if the matching is successful, acquiring a reference identifier associated with the stable prefix hash value and sending the reference identifier to a large language model to instruct the large language model to reuse a corresponding cached context representation; if the matching fails, sending the stable prefix to the large language model to enable the large language model to calculate the index information of each skill package to obtain a context representation and cache the context representation; and generating a response text according to the context representation and a session request. The method realizes the effects of improving cache hit rate, reducing token consumption and response delay.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence intelligent agent platform technology, and more specifically, to a method, apparatus, device and storage medium for optimizing prompt word caching. Background Technology

[0002] Existing AI agent platforms typically employ a full-load strategy in skill package management and prompt word assembly. This involves loading the complete skill files (including tool parameter definitions, call examples, error handling specifications, etc.) of all skill packages subscribed to by the current workspace into the prompt word context at the start of each agent session. This full-load strategy leads to a continuous expansion of prompt word length, reaching tens of thousands of tokens in large-scale, multi-skill scenarios. Simultaneously, because the complete skill files contain dynamically changing content such as timestamps, random example values, and description changes due to version iterations, the prompt word prefix content frequently changes, making it difficult to effectively match with the caching mechanism of the underlying large language model service, resulting in generally low cache hit rates. Furthermore, the full-load method exposes all the internal implementation details of the skill packages, such as MCP (Model Context Protocol) service configuration, local device control protocols, and credential referencing methods, to the model context, expanding the information leakage surface. When a skill package version is updated, the entire complete file changes, causing all cache entries in workspaces subscribed to that skill package to become invalid simultaneously, triggering a cascading cache breakdown.

[0003] Therefore, existing technologies suffer from technical problems such as: redundancy of prompt word context leading to high token consumption and response latency; dynamic changes in prompt word prefixes leading to low cache hit rate; excessive disclosure of internal skill details leading to increased security risks; and version iterations causing large-scale cache invalidation. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, device, and storage medium for optimizing prompt word caching. This addresses the technical problems in existing technologies, such as the redundant expansion of prompt word context, low cache hit rate, excessive disclosure of internal skill details leading to increased information leakage, and large-scale cascading cache failures caused by skill package version iterations. The proposed method achieves the technical effect of decoupling version iterations and cache stability by using a separate loading method for skill indexes and complete skill files, while ensuring the integrity of the agent's functionality. This method uses stable and unchanging skill summary information as the basis for calculating cache hashes, and loads dynamically complete skill files on demand.

[0005] To achieve the above objectives, the first aspect of this application proposes a prompt word caching optimization method, applied to a server, comprising: In response to a session request from the user, the skill subscription data associated with the current workspace is obtained; the skill subscription data includes index information of skill packages, which is a summary description of the skill packages and is used to characterize the identifier and available functional boundaries of the skill packages. Based on the index information of each skill pack, a stable prefix for the prompt words is constructed, and the stable prefix is ​​hashed to obtain the stable prefix hash value; The stable prefix hash value is matched with the pre-stored local hash value; If a match is successful, the reference identifier associated with the stable prefix hash value is obtained and sent to the large language model to instruct the large language model to reuse the corresponding cached context representation; If the match fails, the stable prefix is ​​sent to the large language model so that the large language model can calculate the context representation of the index information of each skill pack and cache it. The response text is generated based on the context representation and the session request and returned to the user client.

[0006] In one or more possible embodiments, after sending the stable prefix to the large language model so that the large language model calculates and caches the context representation based on the index information of each skill pack, the method further includes: Receive the reference identifier generated by the large language model based on the context representation; The reference identifier is associated with the stable prefix hash value and then stored locally.

[0007] In one or more possible embodiments, the index information includes at least one of the following: skill identifier, skill name, skill description, applicable scenarios, list of allowed Model Context Protocol (MCP) method identifiers, list of allowed local tool identifiers, data access level requirements, and risk level cap.

[0008] In one or more possible embodiments, it also includes: The session request is subjected to intent recognition to determine the target skill corresponding to the session request; Obtain the skill file corresponding to the target skill, and send the session request and the skill file to the large language model; wherein, the skill file includes the metadata of the target skill, the definition information of at least one tool included in the target skill, the MCP configuration information associated with the target skill, the local tool configuration information associated with the target skill, and the security policy information of the target skill.

[0009] In one or more possible embodiments, generating response text based on the context representation and the session request and returning it to the user terminal includes: Receive the tool invocation instruction returned by the large language model; wherein the tool invocation instruction includes the target tool identifier and invocation parameters; Based on the target tool identifier and the calling parameters, the corresponding business interface is invoked to perform data query or device control operations to obtain the operation results; The operation result is sent to the large language model, and the response text generated by the large language model based on the operation result is received and returned to the user terminal.

[0010] In one or more possible embodiments, it also includes: Obtain the hash value matching status of this session, the total number of tokens consumed in this session, and the delay time of the first byte of the response text returned by the large language model; When the percentage of misses in the hash value matching state within a preset time window exceeds a preset proportion, a cache optimization prompt is generated; An alarm is generated when the total number of tokens exceeds a preset token threshold. An alarm is generated when the delay time of the first byte exceeds a preset delay threshold.

[0011] In one or more possible embodiments, it also includes: Obtain the version reference list associated with the current workspace, wherein each version reference in the version reference list contains a skill pack identifier and a corresponding version identifier; In response to the release command for a new version of the skill pack, the contents of the new version of the skill pack are written into the immutable storage area of ​​the object storage, and a new version identifier corresponding to the new version is generated; Receive the version upgrade instruction corresponding to the current workspace, switch the version reference of the current workspace from the old version identifier to the new version identifier, and update the skill subscription data corresponding to the current workspace.

[0012] Secondly, this application provides a prompt word caching optimization device, comprising: The request receiving module is used to respond to the user's session request and obtain the skill subscription data associated with the current workspace; the skill subscription data includes the index information of the skill package, and the index information is the summary description information of the skill package, which is used to characterize the identifier and available functional boundaries of the skill package; The hash calculation module is used to construct a stable prefix for the prompt words based on the index information of each skill pack, and to perform hash calculation on the stable prefix to obtain the stable prefix hash value; A cache matching module is used to match the stable prefix hash value with a pre-stored local hash value; If a match is successful, the hit processing module obtains the reference identifier associated with the stable prefix hash value and sends it to the large language model to instruct the large language model to reuse the corresponding cached context representation. If a match fails, the missing prefix is ​​sent to the large language model so that the large language model can calculate and cache the context representation of the index information of each skill pack. The response generation module generates response text based on the context representation and the session request and returns it to the user client.

[0013] Thirdly, this application provides an electronic device, the device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in any of the first aspects.

[0014] Fourthly, this application provides a storage medium storing a computer program for causing a computer to perform the method described in any of the first aspects.

[0015] The technical solutions provided by the embodiments of this application may include the following beneficial effects: In this application, a two-layer structure of index information and complete skill files is split into skill packages. The index information is used as a stable prefix for hash calculation and cache matching. The corresponding complete skill files are loaded on demand. At the same time, version references are used instead of content copying to achieve version immutability governance of skill packages. This constructs a prompt word caching optimization method with progressive disclosure of skill indexes, stable prefix cache hash reuse, performance indicator monitoring closed loop, and version immutability governance. This achieves a significant improvement in cache hit rate, a significant reduction in token consumption and response latency, an effective reduction in the leakage surface of internal skill details, and complete decoupling of skill package version iteration and cache stability in the large language model intelligent agent platform, thus achieving the dual goals of performance optimization and security control. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings: Figure 1 A flowchart of the prompt word caching optimization method provided in this application; Figure 2 A schematic diagram of the prompt word caching optimization device provided in this application Figure 3 A schematic diagram of an electronic device provided in this application; Figure 4 A schematic diagram of a storage medium provided in this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] Existing AI agent platforms typically employ a full-load strategy in skill package management and prompt word assembly. This involves loading the complete skill files (including tool parameter definitions, call examples, error handling specifications, etc.) of all subscribed skill packages into the prompt word context at the start of each agent session. This full-load strategy leads to a continuous expansion of prompt word length, reaching tens of thousands of tokens in large-scale, multi-skill scenarios. Simultaneously, because the complete skill files contain dynamically changing content such as timestamps, random example values, and description changes due to version iterations, the prompt word prefixes frequently change, making it difficult to effectively match with the caching mechanism of the underlying large language model service, resulting in generally low cache hit rates. Furthermore, the full-load method exposes all internal implementation details of the skill packages, such as MCP service configuration, local device control protocols, and credential referencing methods, to the model context, expanding the information leakage surface. When a skill package version is updated, the entire complete file changes, causing all cache entries in workspaces subscribed to that skill package to become invalid simultaneously, triggering a cascading cache breakdown.

[0020] Therefore, existing technologies suffer from technical problems such as: redundancy of prompt word context leading to high token consumption and response latency; dynamic changes in prompt word prefixes leading to low cache hit rate; excessive disclosure of internal skill details leading to increased security risks; and version iterations causing large-scale cache invalidation.

[0021] Based on the above problems, this application proposes a prompt word caching optimization method, applied to a server, such as... Figure 1 As shown, it includes: Step 101: In response to the user's session request, obtain the skill subscription data associated with the current workspace; the skill subscription data includes index information of skill packages, and the index information is a summary description of the skill package, used to characterize the identifier and available functional boundaries of the skill package; In one or more possible embodiments, upon receiving a session request, the skill factory is queried for the skill subscription list of the current workspace, the version reference list of all subscribed skill packages is obtained, and the skill subscription data associated with the current workspace is obtained. For each subscribed skill package, the index information of the corresponding version of each skill package is obtained. The index information is automatically generated by the skill factory when the skill package is published and stored in object storage.

[0022] In one or more possible embodiments, each skill pack corresponds to an index, which is structured data and includes the following fields: (1) Skill identifier is a unique identifier in the global scope. The naming format in this application can be: skill namespace / skill name / version number, such as ops / analytics / v2.3.1, where ops represents the major field of "operation and maintenance" or "business operation" to distinguish skill packages of different departments. For example, there may be billing skills under the finance namespace, and there may be gate control skills under the device namespace. analytics represents "analysis" or "data analysis", specifically indicating that this skill package is for data analysis, not monitoring or control. v2.3.1 represents the version number, where the major version number is 2 (major version update, may not be compatible with the old version), the minor version number is 3 (new features, backward compatible), and the patch number is 1 (bug fix).

[0023] (2) Skill name, which is a short and readable name used to identify the general name of the skill package, such as "parking space status query".

[0024] (3) Skill description, which is used to briefly summarize the function of the skill package, and can also be set to not exceed a preset number of characters.

[0025] (4) Applicable scenarios, which are used to characterize the business scenario categories to which the skill package is applicable, such as daily operations, financial reconciliation, equipment maintenance, customer service, etc.

[0026] (5) The list of allowed MCP method identifiers is a list of global identifiers containing the Model Context Protocol methods that can be called by the skill pack.

[0027] (6) A list of allowed local tool identifiers, used to record the local tools accessible by the skill pack.

[0028] (7) Data access level requirements, which are used to characterize the data access permission level required for the operation of the skill package, such as general knowledge level, cloud data access level, local device access level, etc.

[0029] (8) Risk level cap, used to characterize the security risk level of the skill package.

[0030] The above fields together constitute the structured index information of the skill package, which is extracted and serialized by the server when building a stable prefix. The index information is not the complete content of the skill package, but a simplified summary data. It only contains the minimum set of fields necessary to locate and identify skills, such as skill identifiers and available function boundaries, without carrying detailed implementation information such as tool parameter definitions, calling examples, and error codes.

[0031] Step 102: Construct stable prefixes for prompt words based on the index information of each skill pack, and perform hash calculation on the stable prefixes to obtain stable prefix hash values; In one or more possible embodiments, the index information is sorted in lexicographical order of skill identifiers to ensure that the same skill subscription combination always generates the same sorting order. The sorted index information is serialized into a standardized text format, for example, a canonical format with fixed field order, fixed delimiters, and no redundant whitespace characters, to form a stable prefix text.

[0032] In one or more possible embodiments, if the workspace has not subscribed to any skill packs, the stable prefix contains only empty skill index placeholders; if the file reading of the index information of a skill pack fails (e.g., object storage is temporarily unavailable), the runtime adopts a default rejection policy, marks the skill pack as unloadable, and generates an audit event to record the failure; if the data reach level requirement in the skill index information is higher than the current data reach level determination result of the workspace, the local tool-related entries of the skill are removed at runtime, and only the cloud-reachable method identifier is retained.

[0033] In one or more possible embodiments, the process of building a stable prefix for the index information prompt includes state transitions at different stages. When the construction begins, it is in the initialization state. After initiating a query to the skill factory, it transitions to the skill subscription query state. After the skill subscription query is successful, it transitions to the index information file reading state. If any index information file reading fails, it transitions to the fault handling state, records the audit event, and continues to process the remaining skills. After all index information files are read, it transitions to the sorting and assembly state. After the sorting and assembly is completed, it transitions to the construction completion state.

[0034] In one or more possible embodiments, when the server initiates a skill subscription list query request to the skill factory and does not receive a response within a preset time, it will perform a fixed number of retries, such as three retries. If all retries fail, the most recently successfully cached skill subscription list will be used as the downgraded data, and the cached skill subscription list will be marked as potentially expiring. When the server reads the index information file of a skill package from object storage, if the read fails due to temporary unavailability of the storage service or network instability, a default rejection policy will be adopted for the skill package. The skill package will not be included in the construction scope of the current stable prefix, and a corresponding... The corresponding audit event is used to record the failure. When the server successfully reads the index information file, if the data format does not conform to the preset structured data model specification during parsing, the skill package will be skipped and not included in the construction scope of the stable prefix. At the same time, an audit event is generated and the creator of the skill package is notified to fix it. When the server detects that the total number of subscribed skill packages in the current workspace exceeds the preset upper limit threshold, the skill packages can be truncated according to business priority. The skill packages most relevant to the current session request are retained first. The truncated skill packages are not included in the construction scope of the stable prefix. At the same time, an audit event is generated to record the truncation operation.

[0035] In one or more possible embodiments, after obtaining the stable prefix, a preset hash algorithm, such as Secure Hash Algorithm 2, is performed on the stable prefix text to obtain a stable prefix hash value of fixed length.

[0036] Step 103: Match the stable prefix hash value with the pre-stored local hash value; if the match is successful, proceed to step 104; if the match fails, proceed to step 105. In one or more possible embodiments, the pre-stored local hash value is a cached hash table. The hash table adopts a key-value pair storage mode, where the key is the hash value generated by performing a hash algorithm on stable prefix text, and the value is a cache reference identifier assigned by the underlying model service (large language model). The hash table supports sharding storage by workspace identifier to ensure that cached data in different workspaces are isolated from each other. At the same time, each cache key-value pair also stores the following metadata: creation timestamp, last access timestamp, access count, associated workspace identifier, and associated skill pack version reference list (or associated skill pack version reference list hash), which is used to verify whether the cached key-value pair is still usable.

[0037] During the matching process, a constant-time comparison algorithm is first used to compare the calculated hash value with the historical hash value (local hash value) stored in the cache shard, which can effectively prevent time-series attacks. Second, the last access time of the cached key-value pairs (cache entries) is checked. If the interval between the last access time and the current time exceeds the preset expiration time threshold (the default threshold is seven days), the cache entry is determined to have expired, and a recalculation process is forcibly triggered. Then, the hash of the skill package version reference list stored in the cache entry is compared with the actual skill subscription version list of the current workspace. If the two are inconsistent, it means that the skill subscription content of the workspace has changed, the corresponding cache entry is invalid, and recalculation is required. Finally, when a cache reuse request is initiated, if the model service returns a message indicating insufficient cache space, the runtime will evict old cache entries according to the least recently used strategy to release storage space.

[0038] In one or more possible embodiments, the state machine of the cache matching phase includes the following states: hash calculation state, cache query state, cache hit state, cache miss state, cache write state, and cache eviction state. The state transitions follow these rules: after hash calculation is completed, the state transitions to cache query state; if the cache query hits, the state transitions to cache hit state, indicating a successful match and that the cache can be reused; if the cache query misses, the state transitions to cache miss state, at which point a request needs to be made to the model service to recalculate the context representation; after the model service returns the cache reference identifier, the state transitions to cache write state, where the new cache entry is written to storage; if insufficient storage space is detected during the write process, the state transitions to cache eviction state first, old entries are removed according to the least recently used strategy, and then the state transitions back to cache write state to complete the write operation.

[0039] In one or more possible embodiments, when a hash calculation fails, such as due to insufficient server memory resources, the server switches to a non-caching mode, directly sending the complete stable prefix text to the model service and recording an audit event. When the cache storage service is unavailable, it also switches to a non-caching mode, recording an audit event and triggering an alarm to notify the operations and maintenance personnel. When the model service cannot reuse the cached context representation due to internal reasons, such as the corresponding cache entry being cleared by the model service, the server switches to a recalculation mode, resending the stable prefix of the complete prompt word to the model service for calculation.

[0040] Step 104: Obtain the reference identifier associated with the stable prefix hash value and send it to the large language model to instruct the large language model to reuse the corresponding cached context representation; In one or more possible embodiments, if a match is successful, it indicates that the cache has been hit. The reference identifier corresponding to the cached context representation will be extracted and sent to the underlying model service along with the request, instructing the large language model to reuse the corresponding cached context representation. In addition, although the hash collision probability of the second generation of the secure hash algorithm is extremely low, a consistency check will still be performed on the actual content of the stable prefix text after a successful match to prevent the incorrect reuse of other people's cached entries due to the extremely low probability of hash collisions, thereby ensuring the correctness and reliability of the cache reuse logic.

[0041] Step 105: Send the stable prefix to the large language model so that the large language model can calculate the context representation of the index information of each skill pack and cache it; In one or more possible embodiments, if a match fails, it indicates a cache miss. The stable prefix is ​​then sent to the large language model, which calculates and caches the context representation of the index information of each skill pack. Subsequently, the large language model generates a corresponding reference identifier based on the context representation and sends it to the server. The server then associates the calculated stable prefix hash value with the reference identifier sent by the large language model and stores it in a hash table, waiting for the next match.

[0042] Step 106: Generate response text based on the context representation and the session request, and return it to the user terminal.

[0043] According to the prompt word caching optimization method provided in this application, the skill pack is split into two layers: index information and complete skill file. The index information is used as a stable prefix for hash calculation and cache matching. The corresponding complete skill file is loaded on demand. At the same time, version references are used instead of content copying to achieve version immutability governance of the skill pack. This constructs a prompt word caching optimization method with progressive disclosure of skill index, stable prefix cache hash reuse, performance indicator monitoring closed loop, and version immutability governance. It achieves a significant improvement in cache hit rate, a significant reduction in token consumption and response latency, an effective reduction in the leakage surface of internal skill details, and complete decoupling of skill pack version iteration and cache stability in the large language model intelligent agent platform, thus taking into account the dual goals of performance optimization and security control.

[0044] In one or more possible embodiments, the method further includes: performing intent recognition on the session request to determine the target skill corresponding to the session request; obtaining the skill file corresponding to the target skill, and sending the session request and the skill file to the large language model; wherein the skill file includes metadata of the target skill, definition information of at least one tool included in the target skill, MCP configuration information associated with the target skill, local tool configuration information associated with the target skill, and security policy information of the target skill.

[0045] Specifically, the metadata of the target skill includes skill identifier, version number, creator information, release time, and applicable scenario tags, used to describe the basic attributes of the skill package; the tool definition information includes tool identifier, tool name, tool description, input parameter mode definition, output parameter mode definition, error code definition, and a list of call examples. Both input parameter mode definitions and output parameter mode definitions adopt structured data format to ensure the standardization and parsing of parameter descriptions; the MCP configuration information associated with the target skill includes the associated Model Context Protocol service identifier, method identifier, and method descriptor reference, used to define the calling relationship between the skill package and external services; the local tool configuration information associated with the target skill includes the download address of the tool component, the operating system and architecture compatibility list, and the local area network access requirements, used to support the deployment and execution of the skill in the local environment; the security policy information of the target skill includes risk level, pre-authorization requirements, input security review rule references, and output security review rule references, used to standardize the security control of the skill package during loading and operation.

[0046] In one or more possible embodiments, when a user initiates a session request, the request is first assessed for risk across multiple dimensions to evaluate its security. Once a low-risk or risk-free request is determined, semantic recognition is performed to determine the user's intent and to initially identify the range of skills that may need to be invoked. Further checks are conducted to see if the stable prefix contains matching skill index information. If the user's intent clearly points to a skill, but the skill's index information does not appear in the stable prefix, an unauthorized access interception is triggered, preventing the skill from being loaded. If the user's intent requires invoking a skill and that skill is within the authorized scope (appearing in the stable prefix), a request to load the complete skill file is sent to the skill factory at runtime. The skill factory then reads the corresponding complete skill from storage based on the skill identifier and version reference. The server loads and returns the loaded skill file; performs security checks on the complete skill file, including format verification, integrity verification, and version matching verification, and injects it into the context of the current session after passing the checks; the large language model generates a corresponding tool invocation instruction based on the session request, stable prefix or reference identifier, and the complete skill file of the skill tool, the tool invocation instruction containing the target tool identifier and invocation parameters; the server calls the corresponding business interface to perform data query or device control operation based on the target tool identifier and the invocation parameters, and obtains the operation result; the operation result is sent to the large language model, and the server receives the response text generated by the large language model based on the operation result and returns it to the user terminal; in addition, after the invocation is completed, the skill file can be selectively removed from the context session as needed to release context resources.

[0047] Specifically, the skill corresponding to the user's request intent is matched with the skill index in the stable prefix to confirm whether the skill falls within the authorization scope of the current workspace. If it is not within the authorization scope, loading is refused. The skill factory maintains a local cache of skill files. If the cache is hit, the skill file is returned directly from the cache to avoid repeatedly reading from object storage. If the cache is not hit, it is retrieved from object storage. At the same time, at runtime, based on the preset data reach level judgment result of the current workspace, tool configurations in the complete skill file that exceed the current level are removed, and only method configurations that match the current data reach level are retained. For example, only cloud-reachable method configurations are retained while local device-related configurations are removed. In addition, if the risk level of the skill is high (e.g., R3 or R4), a pre-authorization verification process is triggered before loading the skill file at runtime. Loading is only allowed after the verification passes. For low-level skills (e.g., R1 or R2), no pre-authorization verification is required.

[0048] In one or more possible embodiments, the state machine of the skill file loading phase includes the following states: intent recognition state, skill matching state, permission verification state, skill file loading state, security verification state, skill injection state, tool execution state, and skill unloading state. The specific process is as follows: After intent recognition is completed, the system transitions to the skill matching state, where the runtime matches the corresponding skill index according to the user's intent. If the matching fails, the system transitions to the execution rejection state, terminating the current loading process. If the matching is successful, the system transitions to the permission verification state, where the runtime verifies the calling permission of the skill. After the permission verification passes, the system transitions to the skill file loading state, where the skill factory obtains the complete skill file. After loading is completed, the system transitions to the security verification state, where the skill file undergoes format and integrity verification. After the verification passes, the system transitions to the skill injection state, injecting the skill file into the context of the current session. After the tool execution is completed, the system can optionally transition to the skill unloading state, removing the skill file from the context to release context resources.

[0049] In one or more possible embodiments, when the skill factory loading times out, the runtime performs a preset number of retries (e.g., twice). If the retries still fail, a friendly message indicating that the service is temporarily unavailable is returned to the user, and an audit event is recorded for subsequent investigation. When the skill file integrity check fails, the runtime refuses to load the skill, returns a security warning to the user, generates an audit event, and notifies the skill creator to fix it. When the skill file format is incompatible, the runtime refuses to load the skill, records an audit event, and prompts the workspace administrator to upgrade the skill package version. When injecting skill files causes the total context length to exceed the model limit, the runtime adopts a truncation strategy, prioritizing the truncation of non-core content to ensure that the context length meets the model limit, and records a performance alarm. This achieves the goal of triggering the loading of the corresponding complete skill file only when it is determined that a specific skill needs to be called, avoiding context redundancy caused by full preloading, while taking into account both security verification and resource consumption control.

[0050] In one or more possible embodiments, the method further includes: obtaining the hash value matching status of the current session, the total number of tokens consumed in the current session, and the delay duration of the first byte of the response text returned by the large language model; generating a cache optimization prompt when the proportion of the number of misses in the hash value matching status within a preset time window exceeds a preset ratio; generating an alarm when the total number of tokens exceeds a preset token threshold; and generating an alarm when the delay duration of the first byte exceeds a preset delay threshold.

[0051] Specifically, in terms of data collection strategy, monitoring adopts an asynchronous and non-intrusive approach. At the start of a session, a monitoring session object is created and the start time is recorded. As the process progresses, the hit status and corresponding hash value are recorded during the cache matching phase, the loading time and token consumption of each skill are recorded during the skill loading phase, and the arrival time of the first byte and the end time of the complete response are recorded during the large model response phase. After the session ends, all collected data is written to the time series database asynchronously. The write operation does not block the main process, ensuring that the monitoring behavior itself does not slow down the request response.

[0052] The collected data covers key operational metrics for each session, including: session identifier (to uniquely identify a session request); workspace identifier and user identifier (to locate specific tenants and operators); session start and end timestamps (to calculate total session duration); cache hit status (to mark whether the current request reused cached context representations); cache hit level (to locate the specific hit level in a multi-level caching architecture); stable prefix token count, on-demand loading token count, and total prompt token count (to break down the length composition of each component of the prompt); total response token count (to calculate the length of the large model output); first response latency and total response latency (to measure the time from sending a request to receiving the first response byte and the time to receiving the complete response, respectively); skill call count and skill call list (to track the specific skills involved in the current session); cache hash value (to associate the cache entry corresponding to the current request); and skill package version reference list hash (to assist in determining cache validity when versions change).

[0053] Meanwhile, operations and maintenance personnel can view the overall cache performance of a parking lot by workspace, analyze the loading frequency and token cost of specific skills by skill pack, and observe the trend of cache hit rate by time period. Based on this data, it is possible to identify which skills are loaded on demand most frequently, which time periods have higher initial response latency, and which tenants experience abnormal fluctuations in token consumption, etc.

[0054] In one or more possible embodiments, a performance alarm is triggered when the initial response delay of a request exceeds a preset threshold, and a cost alarm is triggered when token consumption exceeds a set limit. For cache hit rate, the monitoring program statistically analyzes the trend of change by hour, day, or week. If the hit rate drops continuously by more than a preset proportion, an optimization suggestion notification is generated. If a large number of cache misses occur in the same workspace in a short period of time, it may indicate that the tenant is frequently adding or deleting skill packs or that there is an anomaly in cache storage. In this case, an operation and maintenance alarm is triggered to remind relevant personnel to intervene and investigate.

[0055] In one or more possible embodiments, the monitoring and data collection process includes the following states: monitoring initialization state, responsible for creating a monitoring session and recording the start time; metric collection state, responsible for collecting corresponding performance data at each execution stage; session end state, responsible for determining that the collection is complete; data writing state, responsible for asynchronously writing the complete data to storage; and alarm judgment state, responsible for determining whether an alarm needs to be triggered based on the collection results. The specific state transition process is as follows: The monitoring initialization state begins at the start of the session; the metric collection state is updated after each execution stage is completed; the session end state is entered at the end of the session, followed by asynchronously entering the data writing state; and the alarm judgment state is entered after data writing is complete.

[0056] The data monitoring in this application adopts a non-intrusive design, and all monitoring-related failures do not affect the core business process. When monitoring data writing fails, the data can be discarded directly and a log can be recorded without blocking the main process. When the time series database is unavailable, monitoring is downgraded to local file caching mode, and historical data is added in batches after the database is restored. For the problem of false alarms, alarm suppression rules are configured so that alarms of the same type are triggered only once within a set time to avoid alarm storms interfering with operation and maintenance judgment.

[0057] Through the aforementioned data collection, multi-dimensional aggregation, threshold alarm, and anomaly detection mechanisms, a closed-loop feedback capability is provided for the prompt word caching method of this application, enabling operation and maintenance personnel to promptly detect performance degradation and cost anomalies, and adjust the skill package subscription strategy or optimize the skill file loading method accordingly.

[0058] In one or more possible embodiments, the method further includes: obtaining a version reference list associated with the current workspace, wherein each version reference in the version reference list includes a skill pack identifier and a corresponding version identifier; responding to a release instruction for a new version of a skill pack, writing the content of the new version of the skill pack into an immutable storage area of ​​object storage, and generating a new version identifier corresponding to the new version; receiving a version upgrade instruction corresponding to the current workspace, switching the version reference of the current workspace from the old version identifier to the new version identifier, and updating the skill subscription data corresponding to the current workspace.

[0059] Specifically, this application employs an immutable snapshot design for skill pack versions. Each published version of the skill pack is stored as an independent object in object storage. The object key contains the skill identifier and version number, and the object content cannot be modified once written. The skill pack version metadata includes the following information: version identifier, parent version identifier, publisher identifier, publication timestamp, skill index content hash, complete skill file hash, and version status. Version status includes five categories: draft, under review, pending publication, published, and removed from storage. Workspace subscription to skill packs is implemented through version references, storing a combination of the skill identifier and version identifier, rather than a copy of the skill pack content.

[0060] In one or more possible embodiments, the new version release process of a skill pack is as follows: The skill pack can be freely edited by the creator in draft form. After editing, it is submitted to the evolution review draft pool. After evaluation testing, permission difference comparison, and risk difference comparison, it enters the release-ready state. During release, the skill factory calculates hash values ​​for the current version's skill index content and complete skill files, and writes them, along with version metadata, to the immutable area of ​​object storage. The released skill pack version enters a locked state, and any modification operations are rejected. When installing or upgrading a skill pack, the workspace is associated with a specific version through a version reference. At runtime, the corresponding immutable version content is read from object storage based on the version reference. After the new version is released, workspaces associated with the old version are unaffected, and users can choose whether to upgrade to the new version.

[0061] It's important to note that pre-release verification requires three checks to be completed before release: evaluation testing, permission difference comparison, and risk difference comparison. Release is prohibited if any one of these checks fails. During workspace initialization, a version reference validity check is performed to verify that all version references point to valid released versions. If a reference points to a deprecated version, a workspace administrator notification is triggered. When selecting an upgrade skill pack version in the workspace, an automatic version upgrade compatibility check is performed, comparing permission and risk differences between the old and new versions and displaying the impact of the changes to the administrator.

[0062] In one or more possible embodiments, the specific states at the time of release include: draft state, in which the skill pack content can be edited and modified; under review state, indicating that the skill pack has been submitted and is in the evolution review process; pending release state, indicating that the validation has passed and is waiting for official release; released state, indicating that the skill pack has been released and entered an immutable locked state, and any modification operations are rejected; and delisted state, indicating that this version no longer accepts new subscriptions, but the workspace associated with this version can still be used. The state transition process is as follows: after the draft is submitted, it enters the under review state; after the review is passed, it enters the pending release state; after the release operation is performed, it enters the released state; the released version can be transferred to the delisted state by the administrator, and the delisted version cannot be rolled back to the released state, and this transition is irreversible.

[0063] During version release, if object storage write fails, a pre-set number of retries (e.g., three) are performed. If the retries still fail, the version state is rolled back to pending release, and operations personnel are notified. If version hash verification fails, the hash value is recalculated when reading the released version at runtime and compared with the hash value in the metadata. If they do not match, the version is refused to be loaded using the default rejection policy, and skill creators and operations personnel are notified. If a version reference is corrupted, i.e., the version reference data in the workspace is corrupted or points to a non-existent version, the default rejection policy is used at runtime, and a repair notification is pushed to the workspace administrator.

[0064] In one or more possible embodiments, the following operations are further included: obtaining a list of version references associated with the current workspace, where each version reference in the list contains a skill pack identifier and a corresponding version identifier; responding to a release instruction for a new version of the skill pack, writing the content of the new version of the skill pack into the immutable storage area of ​​the object storage, and generating a new version identifier corresponding to the new version; and upon receiving a version upgrade instruction corresponding to the current workspace, switching the version reference of the current workspace from the old version identifier to the new version identifier, and updating the skill subscription data corresponding to the current workspace. This achieves version isolation of skill packs, ensuring that the cache stability of workspaces referencing older versions is not affected by the release of new versions.

[0065] The technical solution of this application will be explained in detail below using a specific parking lot operation scenario.

[0066] Suppose a commercial complex's parking lot has 1,500 parking spaces. Operations manager Mr. Zhang needs to query the parking lot's operational data for the week, including traffic flow distribution at different times, average parking duration, and revenue summary, and requests operational optimization suggestions for the following week. This workspace (the commercial complex's parking lot) has subscribed to four skill packages: Operational Data Analysis (Risk Level R2), used for operational indicator statistics and trend analysis; Fee Management (Risk Level R3), used for fee inquiries and deduction operations; Parking Space Status Monitoring (Risk Level R1), used for real-time parking space status inquiries; and Equipment Maintenance (Risk Level R3), used for status monitoring and control of equipment such as barriers and cameras. The current data access level of this workspace is cloud-accessible.

[0067] Mr. Zhang initiated a natural language conversation request to inquire about the parking lot's operation this week, hoping to understand the busiest time, revenue, and operational suggestions for next week. During the process, the request first undergoes an input security review and risk scoring. The risk level of the request is determined to be R2, which is an analysis and suggestion type operation. The abuse intent score is low, and there is no risk of prompt word injection or privacy exposure. After the review is passed, the process proceeds to the next step.

[0068] At runtime, the system queries the skill factory for the workspace's skill subscription list, retrieving index information for four subscribed skill packages. The Business Data Analysis skill is identified as ops / analytics / v2.3.1, applicable to business analysis, and allows cloud-based data statistics queries, with a risk cap of R2. The Expense Management skill is identified as ops / billing / v1.8.0, applicable to financial operations, and allows cloud-based expense queries and revenue aggregation, with a risk cap of R3. The Parking Status Monitoring skill is identified as ops / parking / v3.1. .2, applicable scenario is daily operation, permitted method is cloud parking space status query, risk limit is R1; the equipment operation and maintenance skill identifier is ops / device / v2.0.5, applicable scenario is equipment maintenance, permitted methods are cloud device status query and local device control, risk limit is R3; since the current data access level is cloud data accessibility level, the local device control method in the equipment operation and maintenance skill is removed, only the cloud device status query method is retained; subsequently, at runtime, all index information is sorted according to the lexicographical order of skill identifier and serialized into a standardized stable prefix text.

[0069] The second generation of the secure hash algorithm is performed on the stable prefix text to generate a 256-bit hash value. After querying the cache, it is found that a similar query was performed in the workspace 30 minutes ago. A matching hash entry already exists in the cache shard, so the cache is hit. At runtime, the corresponding context representation reference identifier is extracted and the cached stable prefix context is directly reused.

[0070] Analyzing the semantics of Mr. Zhang's session request, we identified the intent to perform a statistical query and the intent to generate operational suggestions. The statistical query includes traffic flow distribution, average parking time, and revenue summary, which fall within the scope of the business data analysis skill. The generation of operational suggestions also falls within the analytical suggestion capability of this skill. This indicates that this request does not require calling the specific deduction operation of the expense management system, nor does it require calling the control operation of the equipment maintenance system.

[0071] Since the cache has been hit and the user's intent is clearly directed to the business data analysis skill, the runtime loads the complete file of the skill from the skill factory on demand. The complete file contains a data statistics query tool, whose parameter definitions include time range, statistical dimension and aggregation method; a trend analysis tool, whose parameter definitions include comparison period and trend type; and a suggestion generation tool, whose parameter definitions include suggestion type and output format. The loading of the complete skill file is only triggered when it is determined that the skill needs to be called. The complete files of the other three skill packages are not loaded in this session.

[0072] The system sends the reference identifier, the skill file corresponding to the business data analysis skill, and the session request to the large language model, and receives the tool call instructions returned by the large language model. Based on the tool call instructions, the system executes the tool calls sequentially. First, it calls the data statistics query tool to obtain this week's traffic flow, average parking time, and revenue data. Then, it calls the trend analysis tool to analyze the distribution patterns of different time periods. Finally, it calls the suggestion generation tool to generate operational optimization suggestions for next week based on the analysis results. All operations are performed within the R2 risk level and do not require triggering the R3 or R4 level pre-authorization confirmation process.

[0073] The tool's execution results undergo twelve types of information interception and verification during output security review to ensure they do not contain sensitive information such as prompts, hidden context, or voucher boundaries. The results are output in a pre-defined five-field format, including structured results (this week's operating data table); follow-up actions (suggested operational adjustments); memory update suggestions (recording the analysis conclusions for subsequent sessions); a summary visible to operators (natural language summary); and an attachment summary (data export file description).

[0074] The above implementation demonstrates the complete execution flow of the technical solution of this application. By constructing stable prefixes for skill indexes and matching caches, the response latency in the prompt word assembly process is effectively shortened. By loading only the skill files that the user's intent explicitly points to on demand, context redundancy is avoided. Data security is ensured through output security review and risk level control. Compared with traditional solutions, this solution has significant advantages in cache hit rate, token consumption, response latency, and information leakage surface control.

[0075] This application also provides a prompt word caching optimization device, such as... Figure 2 As shown, it includes: The request receiving module 201 is used to respond to a session request from the user terminal and obtain the skill subscription data associated with the current workspace; the skill subscription data includes index information of skill packages, and the index information is a summary description of the skill package, used to characterize the identifier and available functional boundaries of the skill package; The hash calculation module 202 is used to construct a stable prefix for the prompt words based on the index information of each skill pack, and to perform hash calculation on the stable prefix to obtain the stable prefix hash value; The cache matching module 203 is used to match the stable prefix hash value with the pre-stored local hash value; If a match is successful, the hit processing module 204 obtains the reference identifier associated with the stable prefix hash value and sends it to the large language model to instruct the large language model to reuse the corresponding cached context representation. If a match fails, the miss processing module 205 sends the stable prefix to the large language model so that the large language model can calculate and cache the context representation of the index information of each skill pack. The response generation module 206 generates response text based on the context representation and the session request and returns it to the user terminal.

[0076] Based on the same inventive concept, this application also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described prompt word caching optimization method.

[0077] like Figure 3 As shown, the device includes a processor 301, a memory 302, a communication interface 303, and a bus 304. The processor 301, memory 302, and communication interface 303 are interconnected via the bus 304.

[0078] Processor 301 is configured to read instructions from memory 302 and execute them, so that at least one processor can execute the prompt word caching optimization method provided in the above embodiments.

[0079] The memory 302 is used to store various instructions and programs for the prompt word caching optimization method provided in the above embodiments.

[0080] Bus 304 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0081] Processor 301 can be a central processing unit (CPU), a network processor (NP), a graphics processing unit (GPU), or any combination of CPU, NP, and GPU. It can also be a hardware chip. The aforementioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0082] In addition, this application also provides a storage medium, such as Figure 4 As shown, the storage medium stores a computer program that enables the computer to perform any of the methods described in the above embodiments.

[0083] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) 401 and / or cache memory 402, and may further include read-only memory (ROM) 403.

[0084] The memory may also include a program / utility 405 having a set (at least one) of program modules 404, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0085] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0087] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0088] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0089] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for optimizing prompt word caching, characterized in that, Applied to servers, including: In response to a session request from the user, the skill subscription data associated with the current workspace is obtained; the skill subscription data includes index information of skill packages, which is a summary description of the skill packages and is used to characterize the identifier and available functional boundaries of the skill packages. Based on the index information of each skill pack, a stable prefix for the prompt words is constructed, and the stable prefix is ​​hashed to obtain the stable prefix hash value; The stable prefix hash value is matched with the pre-stored local hash value; If a match is successful, the reference identifier associated with the stable prefix hash value is obtained and sent to the large language model to instruct the large language model to reuse the corresponding cached context representation; If the match fails, the stable prefix is ​​sent to the large language model so that the large language model can calculate the context representation of the index information of each skill pack and cache it. The response text is generated based on the context representation and the session request and returned to the user client.

2. The method according to claim 1, characterized in that, After sending the stable prefix to the large language model so that the large language model can calculate and cache the context representation based on the index information of each skill pack, the method further includes: Receive the reference identifier generated by the large language model based on the context representation; The reference identifier is associated with the stable prefix hash value and then stored locally.

3. The method according to claim 1, characterized in that, The index information includes at least one of the following: skill identifier, skill name, skill description, applicable scenarios, list of allowed Model Context Protocol (MCP) method identifiers, list of allowed local tool identifiers, data access level requirements, and risk level cap.

4. The method according to claim 1, characterized in that, Also includes: The session request is subjected to intent recognition to determine the target skill corresponding to the session request; Obtain the skill file corresponding to the target skill, and send the session request and the skill file to the large language model; wherein, the skill file includes the metadata of the target skill, the definition information of at least one tool included in the target skill, the MCP configuration information associated with the target skill, the local tool configuration information associated with the target skill, and the security policy information of the target skill.

5. The method according to claim 4, characterized in that, The step of generating response text based on the context representation and the session request and returning it to the user client includes: Receive the tool invocation instruction returned by the large language model; wherein the tool invocation instruction includes the target tool identifier and invocation parameters; Based on the target tool identifier and the calling parameters, the corresponding business interface is invoked to perform data query or device control operations to obtain the operation results; The operation result is sent to the large language model, and the response text generated by the large language model based on the operation result is received and returned to the user terminal.

6. The method according to claim 1, characterized in that, Also includes: Obtain the hash value matching status of this session, the total number of tokens consumed in this session, and the delay time of the first byte of the response text returned by the large language model; When the percentage of misses in the hash value matching state within a preset time window exceeds a preset proportion, a cache optimization prompt is generated; An alarm is generated when the total number of tokens exceeds a preset token threshold. An alarm is generated when the delay time of the first byte exceeds a preset delay threshold.

7. The method according to claim 1, characterized in that, Also includes: Obtain the version reference list associated with the current workspace, wherein each version reference in the version reference list contains a skill pack identifier and a corresponding version identifier; In response to the release command for a new version of the skill pack, the contents of the new version of the skill pack are written into the immutable storage area of ​​the object storage, and a new version identifier corresponding to the new version is generated; Receive the version upgrade instruction corresponding to the current workspace, switch the version reference of the current workspace from the old version identifier to the new version identifier, and update the skill subscription data corresponding to the current workspace.

8. A prompt word caching optimization device, characterized in that, include: The request receiving module is used to respond to the user's session request and obtain the skill subscription data associated with the current workspace; The skill subscription data includes index information of skill packages, which is a summary description of the skill packages and is used to characterize the identifier and available functional boundaries of the skill packages. The hash calculation module is used to construct a stable prefix for the prompt words based on the index information of each skill pack, and to perform hash calculation on the stable prefix to obtain the stable prefix hash value; A cache matching module is used to match the stable prefix hash value with a pre-stored local hash value; If a match is successful, the hit processing module obtains the reference identifier associated with the stable prefix hash value and sends it to the large language model to instruct the large language model to reuse the corresponding cached context representation. If a match fails, the missing prefix is ​​sent to the large language model so that the large language model can calculate and cache the context representation of the index information of each skill pack. The response generation module generates response text based on the context representation and the session request and returns it to the user client.

9. An electronic device, characterized in that, The device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores a computer program that enables a computer to perform the method as described in any one of claims 1-7.

Images