Model processing method and apparatus, storage medium, electronic device, and program product

Abstract

The present disclosure relates to the technical field of artificial intelligence, and provides a model processing method and device, a storage medium, an electronic device and a program product. The method comprises: obtaining prompt information input by a user; obtaining a semantic vector of the prompt information; determining an external tool to be called based on the semantic vector of the prompt information and a first KV prefix in a Transformer inference module of a model through an attention mechanism, the Transformer inference module having a plurality of first KV prefixes of external tools pre-injected, the first KV prefix of an external tool being generated based on one or more of identification information, function information and calling condition information of the external tool; calling the external tool to be called, and outputting feedback information for the prompt information according to result information fed back by the external tool to be called. The user's intention is accurately understood to accurately match the external tool to be called, and accurate feedback information is obtained.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to a model processing method, apparatus, storage medium, electronic device, and program product. Background Technology

[0002] With the rise of Large Language Models (LLMs), many companies have developed various artificial intelligence projects based on the Transformer architecture of LLMs, actively exploring the industrial applications of LLMs. The powerful text generation capabilities of LLMs have sparked a surge in artificial intelligence applications. However, as practice has deepened, LLMs have revealed limitations in areas such as real-time perception in complex environments. To address this issue, intelligent agents built around LLMs have been constructed.

[0003] In related technologies, the traditional approach to prompt concatenation in intelligent agent applications relies on inputting nodes (such as search, summarization, translation, database, and MCP services) as text prompts into the LLM. However, this significantly increases the length of the input after concatenation, leading to a substantial increase in attention FLOPs and memory load, decreased system response speed, and low resource utilization. Prompt concatenation is extremely wasteful of computational resources and lacks flexibility and scalability. Tool invocation provides technologies such as OpenAI function calls and Huggingface Agents, allowing the LLM to call external tools. However, it typically relies on explicit prompt guidance and lacks modularity and dynamic composition mechanisms. While tool invocation provides a calling mechanism, it still requires prompt guidance and cannot be modularly and dynamically composed. Summary of the Invention

[0004] The purpose of this disclosure is to provide a model processing method, apparatus, storage medium, electronic device, and program product to solve problems in the related art.

[0005] To achieve the above objectives, this disclosure provides a model processing method, including: Get the prompt information entered by the user; Obtain the semantic vector of the prompt information; Based on the semantic vector of the prompt information and the first key-value prefix in the Transformer inference module of the model, the external tool to be invoked is determined through an attention mechanism. The Transformer inference module is pre-injected with the first key-value prefixes of multiple external tools. The first key-value prefix of the external tool is generated based on one or more of the external tool's identification information, functional information, and invocation condition information. The external tool to be invoked is invoked, and feedback information is output based on the result information returned by the external tool to be invoked, in response to the prompt information.

[0006] Optionally, the first key-value (KV) prefix is ​​generated based on the identification and functional information of the external tool. The semantic vector based on the prompt information and the first KV prefix in the Transformer inference module of the model, through an attention mechanism, determines the external tool to be invoked, including: The semantic vector of the prompt information and the first key-value prefix of the multiple external tools are used to perform attention calculation to obtain the first attention feature; The external tool to be invoked is determined from the plurality of external tools based on the first attention feature.

[0007] Optionally, the first KV prefix includes a first tool K value and a first tool V value, and the step of performing attention calculation on the semantic vector of the prompt information and the first KV prefixes of the multiple external tools to obtain the first attention feature includes: Linear projection is performed on the semantic vector of the prompt information to obtain the first prompt K value, the first prompt V value, and the first prompt Q value; Attention is calculated based on the first cue K value, the first cue V value, the first cue Q value, the first tool K value and the first tool V value of the multiple external tools to obtain the first attention feature.

[0008] Optionally, the step of performing attention calculation based on the first cue K value, the first cue V value, the first cue Q value, the first tool K value and the first tool V value of the plurality of external tools to obtain the first attention feature includes: For each external tool, the first tool K value of the external tool is concatenated with the first prompt K value to obtain a first concatenated K value, and the first tool V value of the external tool is concatenated with the first prompt V value to obtain a first concatenated V value; Attention is calculated based on the first prompt Q value, the first splicing K value of each external tool, and the first splicing V value to obtain the first attention feature.

[0009] Optionally, determining the external tool to be invoked from the plurality of external tools based on the first attention feature includes: The similarity between the tool semantic vector of each external tool and the first attention feature is calculated to obtain the tool similarity corresponding to each external tool, wherein the tool semantic vector represents the semantic vector of the functional information; Based on the tool similarity of all the external tools, the external tool to be invoked is determined from the plurality of external tools.

[0010] Optionally, determining the external tool to be invoked from the plurality of external tools based on the tool similarity corresponding to all the external tools includes: The top Y external tools with high tool similarity are identified as the external tools to be invoked; or, External tools with a similarity greater than the similarity threshold are identified as the external tools that need to be invoked.

[0011] Optionally, the first KV prefix is ​​injected into the first N layers of the Transformer inference module, where N is greater than or equal to 1 and N is less than M, and M is the total number of layers of the Transformer inference module; the Transformer inference module is pre-injected with the second KV prefix of multiple external tools, and the second KV prefix of the external tools is generated based on the calling condition information of the external tools; The invocation of the external tool to be invoked includes: Inject the second KV prefix of the external tool to be invoked into any layer of the Transformer feedforward network of the model, except for the first N layers; The first attention feature is obtained by performing attention calculation on the first attention feature and the second key-value prefix of the external tool to be invoked. The first attention feature is obtained by performing attention calculation on the semantic vector of the prompt information and the first key-value prefix of the multiple external tools. The invocation parameters for invoking the external tool to be invoked are determined based on the second attention feature; The external tool to be invoked is invoked according to the invocation parameters.

[0012] This disclosure also provides a model processing apparatus, including: The first processing module is configured to obtain prompt information input by the user; The second processing module is configured to obtain the semantic vector of the prompt information; The third processing module is configured to determine the external tool to be invoked based on the semantic vector of the prompt information and the first key-value prefix in the Transformer inference module of the model through an attention mechanism. The Transformer inference module pre-injects the first key-value prefixes of multiple external tools. The first key-value prefix of the external tool is generated based on one or more of the external tool's identification information, functional information, and invocation condition information. The fourth processing module is configured to invoke the external tool to be invoked, and output feedback information for the prompt information based on the result information returned by the external tool to be invoked.

[0013] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described model processing method.

[0014] This disclosure also provides an electronic device, including: A memory on which computer programs are stored; A processor is configured to execute the computer program in the memory to implement the steps of the above-described model processing method.

[0015] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described model processing method.

[0016] The above technical solution acquires user-input prompt information; obtains the semantic vector of the prompt information; and, based on the semantic vector of the prompt information and the first key-value prefix in the Transformer inference module of the model, determines the external tool to be invoked through an attention mechanism. The Transformer inference module pre-injects multiple first key-value prefixes of external tools, which are generated based on one or more of the external tool's identification information, functional information, and invocation condition information. The required external tool is invoked, and feedback information is output based on the results from the invoked external tool. By using the semantic vector of the user-input prompt information and the pre-injected first key-value prefixes of multiple external tools, combined with the attention mechanism, the user's intent is accurately understood to precisely match the required external tool. This allows for flexible invocation of the required external tool, obtaining accurate feedback information, and solving the problems of wasted computational resources, lack of flexibility and scalability, and the inability to modularly and dynamically combine tool invocations in prompt splicing techniques.

[0017] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a model processing method according to an exemplary embodiment.

[0019] Figure 2This is a schematic diagram illustrating a model processing method according to an exemplary embodiment.

[0020] Figure 3 This is a flowchart illustrating a sub-step of step S3 according to an exemplary embodiment.

[0021] Figure 4 This is a flowchart illustrating a sub-step of step S31 according to an exemplary embodiment.

[0022] Figure 5 This is a flowchart illustrating a sub-step of step S312 according to an exemplary embodiment.

[0023] Figure 6 This is a flowchart illustrating a sub-step of step S32 according to an exemplary embodiment.

[0024] Figure 7 This is a flowchart illustrating a sub-step for invoking an external tool according to an exemplary embodiment.

[0025] Figure 8 This is a block diagram illustrating a model processing apparatus according to an exemplary embodiment.

[0026] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0027] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0028] In the following description, the words "first" and "second" are used only to distinguish the purpose of the description and should not be interpreted as indicating or implying relative importance or order.

[0029] With the rise of LLM, many companies have developed various artificial intelligence projects based on the Transformer architecture of LLM, actively exploring the industrial applications of LLM. LLM's powerful text generation capabilities have sparked a surge in artificial intelligence applications. However, as practice has deepened, LLM has revealed limitations in areas such as real-time perception in complex environments. To address this issue, intelligent agents built around LLM have been constructed.

[0030] In related technologies, the traditional approach to prompt concatenation in intelligent agent applications relies on inputting nodes (such as search, summarization, translation, database, and MCP services) as text prompts into the LLM. However, this significantly increases the length of the input after concatenation, leading to a substantial increase in attention FLOPs and memory load, decreased system response speed, and low resource utilization. Prompt concatenation is extremely wasteful of computational resources and lacks flexibility and scalability. Tool invocation provides technologies such as OpenAI function calls and Huggingface Agents, allowing the LLM to call external tools. However, it typically relies on explicit prompt guidance and lacks modularity and dynamic composition mechanisms. While tool invocation provides a calling mechanism, it still requires prompt guidance and cannot be modularly and dynamically composed.

[0031] To address the aforementioned issues, this approach involves acquiring user-input prompts; obtaining the semantic vector of these prompts; and using an attention mechanism based on the semantic vector and the first key-value (KV) prefixes in the Transformer inference module of the model to determine the external tools to be invoked. The Transformer inference module pre-injects multiple first KV prefixes for external tools, generated based on one or more of the external tool's identifier, functional information, and invocation conditions. The appropriate external tool is then invoked, and feedback information is output based on the results from the invoked external tool. By combining the semantic vector of the user-input prompts and the pre-injected first KV prefixes of multiple external tools with the attention mechanism, the user's intent is accurately understood to precisely match the required external tool. This allows for flexible invocation of the appropriate external tool, obtaining accurate feedback information and resolving the issues of wasted computational resources, lack of flexibility and scalability, and the inability to modularly and dynamically combine tool invocations in prompt concatenation techniques.

[0032] Figure 1 This is a flowchart illustrating a model processing method according to an exemplary embodiment. Figure 2 This is a schematic diagram illustrating a model processing method according to an exemplary embodiment. Please refer to [link / reference]. Figure 1 and Figure 2 The model processing method may include steps S1 to S4.

[0033] Step S1: Obtain the prompt information entered by the user.

[0034] The prompts entered by the user can be text, voice, or other forms of information.

[0035] For example, the prompts could be "What's the weather like in Beijing today?" or "The development history of artificial intelligence", etc.

[0036] Step S2: Obtain the semantic vector of the prompt information.

[0037] A semantic vector can be a numerical vector representing the text content of a prompt message. If the prompt message entered by the user is text, that text message is the text content of the prompt message. If the prompt message entered by the user is speech, the speech message is automatically recognized to obtain text information, and that text information is then used as the text content of the prompt message.

[0038] Obtaining the semantic vector of the prompt information can be understood as first encoding the text content of the prompt information to obtain the prompt encoding information, and then performing vector transformation on the prompt encoding information to obtain the semantic vector of the prompt information.

[0039] Step S3: Based on the semantic vector of the prompt information and the first KV prefix in the Transformer inference module of the model, determine the external tool to be invoked through the attention mechanism.

[0040] The Transformer inference module pre-injects the first key-value (KV) prefixes of multiple external tools. The first KV prefixes of the external tools are generated based on one or more of the external tool's identification information, functional information, and invocation condition information.

[0041] Identification information is that each external tool has a unique identifier used to distinguish different external tools.

[0042] Functional information describes the functionality of an external tool; for example, the function of a weather query tool is to provide weather information.

[0043] The call condition information is used to characterize the parameter specification or parameter architecture.

[0044] The text content of external information consists of one or more of the identification information, function information, and invocation condition information of an external tool. The first key-value (KV) prefix of the external tool can be generated in the following way: first, the text content of the external tool is encoded to obtain tool encoding information; then, the tool encoding information is vectorized to obtain the semantic vector of the external tool; and finally, the semantic vector of the external tool is linearly projected to obtain the first KV prefix.

[0045] By pre-injecting the first KV prefix of multiple external tools, redundant calculations can be avoided, thereby significantly improving inference efficiency.

[0046] Step S4: Invoke the external tool that needs to be called, and output feedback information for the prompt information based on the result information returned by the external tool.

[0047] By combining the semantic vector of the prompt information input by the user, and the first key-value prefix of multiple pre-injected external tools, with an attention mechanism, the user's intent is accurately understood to accurately match the external tool to be invoked. Then, the external tool to be invoked can be invoked to obtain accurate feedback information.

[0048] In one possible implementation, the first KV prefix is ​​generated based on the identification information and functional information of the external tool. See also... Figure 3 Step S3 may include steps S31 and S32.

[0049] Step S31: Perform attention calculation on the semantic vector of the prompt information and the first KV prefixes of multiple external tools to obtain the first attention feature.

[0050] The first attention feature is a hidden vector.

[0051] In one embodiment, the first KV prefix includes a first tool K value and a first tool V value. The semantic vector of the prompt information is used as the query vector. For the first KV prefix of each external tool, the dot product of the query vector and the first tool K value is calculated to obtain an attention score. Based on the attention score, the first tool V value is weighted and summed to obtain the first attention feature.

[0052] Step S32: Determine the external tool to be invoked from multiple external tools based on the first attention feature.

[0053] This can be understood as determining which external tool to invoke based on the similarity between the semantic vector of each external tool's functional information and the first attention feature.

[0054] In one possible implementation, the first KV prefix includes a first instrument K value and a first instrument V value. See also... Figure 4 Step S31 may include steps S311 and S312.

[0055] Step S311: Perform linear projection on the semantic vector of the prompt information to obtain the first prompt K value, the first prompt V value, and the first prompt Q value.

[0056] By using three different linear transformations, the semantic vector of the prompt information is mapped to three different vector spaces, generating the first prompt K value, the first prompt V value, and the first prompt Q value.

[0057] The formula for calculating the K value can be expressed as follows: First hint: K value = WK • The semantic vector of the prompt message; The formula for calculating the first hint, V, can be expressed as: First hint: V value = W V • The semantic vector of the prompt message; The formula for calculating the Q value can be expressed as follows: First hint: Q value = W Q • The semantic vector of the prompt message; Among them, W K W V W Q The weight matrices represent different vector spaces respectively.

[0058] The first prompt's K value can represent the key features of the prompt message, the first prompt's V value can represent the content of the prompt message, and the first prompt's Q value can represent the query or focus of the prompt message.

[0059] Step S312: Based on the first cue K value, the first cue V value, the first cue Q value, the first tool K value and the first tool V value of multiple external tools, attention is calculated to obtain the first attention feature.

[0060] In one possible implementation, please refer to Figure 5 Step S312 includes steps S3121 and S3122.

[0061] Step S3121: For each external tool, concatenate the first tool K value and the first prompt K value of the external tool to obtain the first concatenated K value, and concatenate the first tool V value and the first prompt V value of the external tool to obtain the first concatenated V value.

[0062] Step S3122: Based on the first cue Q value, the first splicing K value of each external tool, and the first splicing V value, attention is calculated to obtain the first attention feature.

[0063] This can be understood as follows: First, attention is calculated based on the first cue Q value and the first concatenation K value of each external tool to obtain an attention score. Then, a weighted sum is performed based on the attention score and the first concatenation V value of each external tool to obtain the first attention feature.

[0064] The formula for calculating the attention score is as follows:

[0065] in, Characterizing the first cue Q value, Characterizing the first splicing K value, d k It is the dimension of the K value, used to scale the dot product to avoid the gradient vanishing problem.

[0066] The formula for calculating the first attention feature is as follows: First attention characteristic = Attention score · v 拼接 in, Characterizes the V value of the first splice.

[0067] The first attention feature is a vector that integrates cue information and external tool information, and can represent the correlation between cue information and various external tools.

[0068] In one embodiment, when user input passes through the first N Transformer layers, it interacts with the injected first key-value prefix for computation, resulting in the Transformer layer output x, which is the first attention feature. The relevant procedure can be as follows: def transformer_attention(query, key, value, mask=None): """ query: [B, T_q, d] from the current input key:[B, T_k, d] Original context key value: [B, T_k, d] Original context value mask:[B, T_q, T_k] Attention mask """ d_k = query.size(-1) # Attention Weight Calculation scores = (query @ key.transpose(-2, -1)) / sqrt(d_k)# [B, T_q, T_k_total] if mask is not None: scores = scores.masked_fill(mask == 0, -1e9) attn_weights = softmax(scores, dim=-1)# [B, T_q, T_k_total] output = attn_weights @ value# [B, T_q, d] return output, attn_weights def transformer_layer_forward(x, prefix_k=None, prefix_v=None): """ x: Current layer input [B, T, d] prefix_k, prefix_v: KV prefixes, in the form [B, T_prefix, d] """ # 1. Projection yields q / k / v q = x @ Wq# [B, T, d] k = x @ Wk# [B, T, d] v = x @ Wv# [B, T, d] # 2. Concatenate the KV prefix (if it exists) if prefix_k is not None and prefix_v is not None: k = concat([prefix_k, k], dim=1)# [B, T_prefix + T, d] v = concat([prefix_v, v], dim=1) #3. Calculate attention out, attn_weights = transformer_attention(q, k, v) # 4. Residual Join + Norm + FFN (details omitted) return out In one possible implementation, please refer to Figure 6 Step S32 may include steps S321 and S322.

[0069] Step S321: Calculate the similarity between the tool semantic vector of each external tool and the first attention feature to obtain the tool similarity corresponding to each external tool.

[0070] Among them, the tool semantic vector represents the semantic vector of functional information.

[0071] The tool semantic vector first encodes the functional information of the external tool to obtain functional encoding information; then it performs vector transformation on the functional encoding information to obtain the tool semantic vector.

[0072] Similarity can be calculated using methods such as, but is not limited to, cosine similarity, dot product similarity, etc.

[0073] Step S322: Based on the tool similarity of all external tools, determine the external tool that needs to be called from among multiple external tools.

[0074] In one possible implementation, step S322 may include: The top Y external tools with high tool similarity are identified as the external tools to be invoked; or, External tools with a similarity greater than the similarity threshold are identified as the external tools that need to be invoked.

[0075] Y is an integer. Y and the similarity threshold can be set according to the actual situation, and this embodiment does not impose any limitations on them.

[0076] In one embodiment, the similarity between x and the tool semantic vector of each external tool is calculated, and then the external parameters to be invoked are calculated based on the threshold hyperparameter. The relevant procedure is as follows: def select_tools_from_hidden_state(x, tool_vectors, top_k=2,threshold=0.6): """ x: [B, T, d] Output of the first N layers of the model tool_vectors: Dict[str, Tensor] tool semantic vectors top_k: The maximum number of tools returned threshold: Similarity threshold; external tools with similarity values ​​below this threshold will be ignored. """ x_pooled = x.mean(dim=1)# [B, d] results = {} for name, vec in tool_vectors.items(): score = cosine_similarity(x_pooled, vec) if score >= threshold: results[name] = score # Sort by similarity in descending order, returning at most the top_k external tools. sorted_tools = sorted(results.items(), key=lambda x: x[1], reverse=True) return [name for name, score in sorted_tools[:top_k]] Enabling more intelligent external tool routing through semantic-level matching offers greater generalization and scalability than traditional keyword- or rule-based methods.

[0077] In one possible implementation, the first KV prefix is ​​injected into the first N layers of the Transformer inference module, where N is greater than or equal to 1 and less than M, and M is the total number of layers in the Transformer inference module; the Transformer inference module is pre-injected with the second KV prefix of multiple external tools, and the second KV prefix of the external tools is generated based on the calling condition information of the external tools.

[0078] The first N layers in the Transformer inference module injected with the first KV prefix can be regarded as the tool selection layer, which is used to quickly identify whether a tool call is needed in the early layers.

[0079] The second KV prefix can be injected into any layer after the Nth layer in the Transformer inference module.

[0080] Please see Figure 7 Step S4 involves calling the external tool that needs to be called, which may include steps S41 to S44.

[0081] Step S41: Inject the second KV prefix of the external tool to be called into any layer of the Transformer feedforward network of the model, except for the first N layers.

[0082] This can be understood as injecting the second KV prefix of the external tool to be called from any layer after the Nth layer in the Transformer inference module into any layer of the Transformer feedforward network of the model, excluding the first N layers.

[0083] Step S42: Perform attention calculation on the first attention feature and the second KV prefix of the external tool to be invoked to obtain the second attention feature.

[0084] The first attention feature is obtained by performing attention calculation on the semantic vector of the prompt information and the first KV prefixes of multiple external tools.

[0085] The second KV prefix includes the second instrument K value and the second instrument V value.

[0086] The second KV prefix of an external tool can be generated as follows: first, the calling condition information of the external tool is encoded to obtain calling encoding information; then, the calling encoding information is vectorized to obtain the calling semantic vector of the external tool; and finally, the calling semantic vector of the external tool is linearly projected to obtain the second KV prefix.

[0087] This can be understood as follows: first, calculate the first attention feature and the second tool K value of the external tool to be invoked to perform attention calculation and obtain the attention score; then, perform a weighted sum based on the attention score and the second tool V value of the external tool to be invoked to obtain the second attention feature.

[0088] The process of obtaining the second attention feature can be referred to the process of obtaining the first attention feature, and will not be repeated here in this embodiment.

[0089] Step S43: Determine the invocation parameters for calling the external tool to be invoked based on the second attention feature.

[0090] This can be understood as extracting information related to external tool calls from the second attention features, and generating specific call parameters based on the extracted information.

[0091] Step S44: Invoke the external tool to be invoked according to the invocation parameters.

[0092] Based on the calling parameters, a request is sent through the API interface of the tool to be called, the result is obtained, and the result is processed into a format that the model can understand.

[0093] In one embodiment, the second key-value prefix of the external tool to be invoked is dynamically injected into the subsequent N layers, the model outputs the subsequent content, and the final result is generated.

[0094] def transformer_layer_forward(x, prefix_k=None, prefix_v=None): """ x: Current layer input [B, T, d] prefix_k, prefix_v: KV prefixes, in the form [B, T_prefix, d] """ # 1. Projection yields q / k / v q = x @ Wq# [B, T, d] k = x @ Wk# [B, T, d] v = x @ Wv# [B, T, d] # 2. Concatenate the KV prefix (if it exists) if prefix_k is not None and prefix_v is not None: k = concat([prefix_k, k], dim=1)# [B, T_prefix + T, d] v = concat([prefix_v, v], dim=1) #3. Calculate attention out, attn_weights = transformer_attention(q, k, v) # 4. Residual Join + Norm + FFN (details omitted) return out In one embodiment, please continue reading Figure 2 In the external tool preparation phase, available external tools are registered and their information is maintained. The tool list is structured by providing identification information (such as tool name), functional information (such as functional description), and calling condition information (such as calling template).

[0095] For example, the relevant procedure is as follows: Tool = NamedTuple("Tool", [("name", str), ("description", str), ("template", str)]) TOOL_REGISTRY = [ Tool("get_location", "Get geolocation information based on IP address", "get_location({ip})"), Tool("get_weather", "Query weather by city name", "get_weather({city},{date})"), # ... Other tools ] During the external tool injection phase, tool information is injected into the model's Transformers in the form of KV prefixes (first KV prefix and second KV prefix) in the two phases before and after.

[0096] The first key-value prefix injection involves encoding each external tool and its functional description in the format "tool.name:tool.description" using a tokenizer and embedding, transforming it into a vector, generating a key-value pair via Wk / Wv projection, and then injecting it into the attention portion of the first N layers of Transformers.

[0097] For example, the relevant procedure is as follows: def inject_selector_kv(model, tools: List[Tool], N): For tool in tools: text = f"{tool.name}: {tool.description}" tokens = tokenizer(text) emb = embedding_layer(tokens) for layer in model.layers[:N]: k = emb @ layer.Wk v = emb @ layer.Wv layer.attn.kv_cache.insert_prefix(k, v) The second key-value prefix injection involves encoding and transforming the call template of each external tool in the format "tool.name: tool.template" using tokenizer and embedding, then generating a key-value pair via Wk / Wv projection, and injecting it into the attention part of the last N layers of Transformers.

[0098] For example, the relevant procedure is as follows: def inject_executor_kv(model, tools: List[Tool], N): For tool in tools: text = f"{tool.name}: {tool.description}" tokens = tokenizer(text) emb = embedding_layer(tokens) for layer in model.layers[layer_num-N:]: k = emb @ layer.Wk v = emb @ layer.Wv layer.attn.kv_cache.insert_prefix(k, v) During the model processing stage, not only is a kv prefix injected, but attention is also incorporated into the computation.

[0099] For example, the relevant procedure is as follows: def forward_with_tool_injection(input_tokens): x = input_tokens selected_tools = [] # Intermediate variable: list of activated tool names for i, layer in enumerate(transformer_layers): # The first N layers are the Tool Selector stage if i <N: x = layer(x) if i == N-1: # In the final layer of the Tool Selector, determine which tools are activated in the attention output. selected_tools = select_tools_from_hidden_state(x) # Construct the Executor KV prefix based on the selection results kv_executor_prefix = build_executor_kv_prefix(selected_tools) # Inject it into the KV cache of the subsequent layer (Executor phase) for j in range(N, L): transformer_layers[j].attn.kv_cache.insert_prefix( kv_executor_prefix[jN].k, kv_executor_prefix[jN].v ) # Continue Argument Extractor reasoning in the next N layers else: x = layer(x) # Model outputs structured call statements return decode(x) In another embodiment, the user inputs "User: Beijing weather tomorrow?"; the first N1 layers of the Transformer inject a tool text prefix (such as the function description of weather_tool); through attention, the Planner stage is guided to identify weather_tool; the Planner outputs the tool_name of weather_tool, and in the subsequent N2-final layers, the calling method of weather_tool is obtained and output through attention calculation with the injected calling method prefix; based on the inference output, the tool call is naturally initiated, and the retrieval / call action is performed; the inference output returns the call result, or continues to execute the second stage task.

[0100] This disclosure generates key-value prefix representations for each callable external tool (search, MCP, function call, etc.) by encoding it through the original LLM tokenizer + embedding layer, without additional training. When the LLM receives user input during the policy generation stage, it uses attention computation to assist in generating tool matching policies by pre-injecting the first key-value prefix into the first few layers of the model. The second key-value prefix of the external tool to be called is determined according to the tool matching policy. The model supports adding or deleting external tools during processing without retraining.

[0101] This disclosure has at least the following advantages: First, Prompt-free calls eliminate the need for splicing tool descriptions, reducing prompt length and attention complexity; Second, it is low-cost and dynamically expandable. External tools exist in the form of embeddings and can be dynamically loaded with the plan without training. Third, modular reusability, supporting multi-agent / function orchestration, without the need for large-scale prompt rewriting; Fourth, it is compatible with existing models. Without changing the model structure, only KV prefix support needs to be added to the inference structure, making it highly practical for engineering implementation.

[0102] Based on the same inventive concept, this disclosure also provides a model processing apparatus; please refer to [link to relevant documentation]. Figure 8 , Figure 8 This is a block diagram illustrating a model processing apparatus according to an exemplary embodiment. A model processing apparatus 600 may include: The first processing module 601 is configured to obtain prompt information input by the user; The second processing module 602 is configured to acquire the semantic vector of the prompt information; The third processing module 603 is configured to determine the external tool to be invoked based on the semantic vector of the prompt information and the first KV prefix in the Transformer inference module of the model through an attention mechanism. The Transformer inference module pre-injects the first KV prefixes of multiple external tools. The first KV prefix of the external tool is generated based on one or more of the external tool's identification information, functional information, and invocation condition information. The fourth processing module 604 is configured to invoke the external tool that needs to be invoked, and output feedback information for the prompt information based on the result information returned by the external tool that needs to be invoked.

[0103] Optionally, the first KV prefix is ​​generated based on the identification information and functional information of external tools, and the third processing module 603 may include: The first sub-processing module is configured to perform attention calculation on the semantic vector of the prompt information and the first KV prefix of multiple external tools to obtain the first attention feature; The second sub-processing module is configured to determine the external tool to be invoked from among multiple external tools based on the first attention feature.

[0104] Optionally, the first KV prefix includes the first tool K value and the first tool V value, and the first sub-processing module includes: The first sub-processing unit is configured to perform a linear projection on the semantic vector of the prompt information to obtain the first prompt K value, the first prompt V value and the first prompt Q value; The second sub-processing unit is configured to perform attention calculation based on the first cue K value, the first cue V value, the first cue Q value, the first tool K value and the first tool V value of multiple external tools, to obtain the first attention feature.

[0105] Optionally, the second sub-processing unit is specifically configured as follows: For each external tool, the first tool K value and the first prompt K value of the external tool are concatenated to obtain the first concatenated K value, and the first tool V value and the first prompt V value of the external tool are concatenated to obtain the first concatenated V value. Attention is calculated based on the first cue Q value, the first concatenation K value of each external tool, and the first concatenation V value to obtain the first attention feature.

[0106] Optionally, the second sub-processing module includes: The third sub-processing unit is configured to calculate the similarity between the tool semantic vector of each external tool and the first attention feature, and obtain the tool similarity corresponding to each external tool, wherein the tool semantic vector represents the semantic vector of functional information; The fourth sub-processing unit is configured to determine the external tool to be invoked from among multiple external tools based on the tool similarity of all external tools.

[0107] Optionally, the fourth sub-processing unit is specifically configured as follows: The top Y external tools with high tool similarity are identified as the external tools to be invoked; or, External tools with a similarity greater than the similarity threshold are identified as the external tools that need to be invoked.

[0108] Optionally, the first KV prefix is ​​injected into the first N layers of the Transformer inference module, where N is greater than or equal to 1 and N is less than M, and M is the total number of layers in the Transformer inference module; the second KV prefix of multiple external tools is pre-injected into the Transformer inference module, and the second KV prefix of the external tools is generated based on the calling condition information of the external tools; The fourth processing module is specifically configured as follows: Inject the second KV prefix of the external tool to be called into any layer of the Transformer feedforward network of the model, except for the first N layers; The first attention feature is obtained by performing attention calculation on the first attention feature and the second key-value prefix of the external tool to be invoked. The first attention feature is obtained by performing attention calculation on the semantic vector of the prompt information and the first key-value prefix of multiple external tools. The invocation parameters for calling the external tool to be invoked are determined based on the second attention characteristics; The external tool to be invoked is invoked based on the invocation parameters.

[0109] Regarding the model processing apparatus in the above embodiments, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the model processing method, and will not be elaborated upon here.

[0110] Figure 9 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. Figure 9 As shown, the electronic device 700 may include a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0111] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the aforementioned model processing method. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 703 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication may include Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof; therefore, the corresponding communication component 705 may include a Wi-Fi module, a Bluetooth module, or an NFC module.

[0112] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the model processing method described above.

[0113] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the model processing method described above. For example, the computer-readable storage medium may be the memory 702 including program instructions described above, which may be executed by the processor 701 of the electronic device 700 to complete the model processing method described above.

[0114] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a processor, which, when executed by the processor, implements the steps of the model processing method described above.

[0115] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0116] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0117] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A model processing method, characterized in that, include: Get the prompt information entered by the user; The prompt information includes text information and voice information; Obtain the semantic vector of the prompt information; the semantic vector represents the numerical vector corresponding to the text content of the prompt information. Based on the semantic vector of the prompt information and the first KV prefix in the Transformer inference module of the model, the external tool to be invoked is determined through an attention mechanism. The Transformer inference module is pre-injected with the first KV prefix of multiple external tools. The first KV prefix of the external tool is generated based on one or more of the external tool's identification information, functional information, and invocation condition information. The first KV prefix includes the first tool K value and the first tool V value. The external tool to be invoked is invoked, and feedback information is output based on the result information returned by the external tool to be invoked, in response to the prompt information. The first key-value (KV) prefix is ​​generated based on the identification and functional information of the external tool. The semantic vector of the prompt information and the first KV prefix in the Transformer inference module of the model are used to determine the external tool to be invoked through an attention mechanism, including: Linear projection is performed on the semantic vector of the prompt information to obtain a first prompt K value, a first prompt V value, and a first prompt Q value; wherein, the first prompt K value represents the key feature of the prompt information, the first prompt V value represents the content of the prompt information, and the first prompt Q value represents the query or focus of the prompt information; Attention is calculated based on the first cue K value, the first cue V value, the first cue Q value, the first tool K value and the first tool V value of the multiple external tools to obtain the first attention feature; The external tool to be invoked is determined from the plurality of external tools based on the first attention feature; The first attention feature is obtained by performing attention calculation based on the first cue K value, the first cue V value, the first cue Q value, the first tool K value, and the first tool V value of the multiple external tools, including: For each external tool, the first tool K value of the external tool is concatenated with the first prompt K value to obtain a first concatenated K value, and the first tool V value of the external tool is concatenated with the first prompt V value to obtain a first concatenated V value; Attention is calculated based on the first prompt Q value, the first splicing K value of each external tool, and the first splicing V value to obtain the first attention feature.

2. The model processing method according to claim 1, characterized in that, The step of determining the external tool to be invoked from the plurality of external tools based on the first attention feature includes: The similarity between the tool semantic vector of each external tool and the first attention feature is calculated to obtain the tool similarity corresponding to each external tool, wherein the tool semantic vector represents the semantic vector of the functional information; Based on the tool similarity of all the external tools, the external tool to be invoked is determined from the plurality of external tools.

3. The model processing method according to claim 2, characterized in that, The step of determining the external tool to be invoked from the plurality of external tools based on the tool similarity corresponding to all the external tools includes: The top Y external tools with high tool similarity are identified as the external tools to be invoked; or, External tools with a similarity greater than the similarity threshold are identified as the external tools that need to be invoked.

4. The model processing method according to claim 1, characterized in that, The first KV prefix is ​​injected into the first N layers of the Transformer inference module, where N is greater than or equal to 1 and less than M, and M is the total number of layers of the Transformer inference module; the Transformer inference module is pre-injected with the second KV prefix of multiple external tools, and the second KV prefix of the external tools is generated based on the calling condition information of the external tools; The invocation of the external tool to be invoked includes: Inject the second KV prefix of the external tool to be invoked into any layer of the Transformer feedforward network of the model, except for the first N layers; The first attention feature is obtained by performing attention calculation on the first attention feature and the second key-value prefix of the external tool to be invoked. The first attention feature is obtained by performing attention calculation on the semantic vector of the prompt information and the first key-value prefix of the multiple external tools. The invocation parameters for invoking the external tool to be invoked are determined based on the second attention feature; The external tool to be invoked is invoked according to the invocation parameters.

5. A model processing device, characterized in that, include: The first processing module is configured to obtain prompt information input by the user; The prompt information includes text information and voice information; The second processing module is configured to obtain the semantic vector of the prompt information; the semantic vector represents the numerical vector corresponding to the text content of the prompt information. The third processing module is configured to determine the external tool to be invoked based on the semantic vector of the prompt information and the first KV prefix in the Transformer inference module of the model through an attention mechanism. The Transformer inference module pre-injects multiple first KV prefixes of external tools. The first KV prefix of the external tool is generated based on one or more of the external tool's identification information, functional information, and invocation condition information. The first KV prefix includes a first tool K value and a first tool V value. The fourth processing module is configured to invoke the external tool to be invoked, and output feedback information for the prompt information based on the result information returned by the external tool to be invoked; The first KV prefix is ​​generated based on the identification information and functional information of the external tool. The third processing module is specifically configured to perform linear projection on the semantic vector of the prompt information to obtain a first prompt K value, a first prompt V value, and a first prompt Q value. The first prompt K value represents the key feature of the prompt information, the first prompt V value represents the content of the prompt information, and the first prompt Q value represents the query or focus of the prompt information. Attention is calculated based on the first cue K value, the first cue V value, the first cue Q value, the first tool K value and the first tool V value of the multiple external tools to obtain the first attention feature; The external tool to be invoked is determined from the plurality of external tools based on the first attention feature; The first attention feature is obtained by performing attention calculation based on the first cue K value, the first cue V value, the first cue Q value, the first tool K value, and the first tool V value of the multiple external tools, including: For each external tool, the first tool K value of the external tool is concatenated with the first prompt K value to obtain a first concatenated K value, and the first tool V value of the external tool is concatenated with the first prompt V value to obtain a first concatenated V value; Attention is calculated based on the first prompt Q value, the first splicing K value of each external tool, and the first splicing V value to obtain the first attention feature.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the model processing method according to any one of claims 1-4.

7. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the model processing method according to any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the model processing method according to any one of claims 1-4.

Images