Heterogeneous ai inference agent function call enhancement method, system, device, and medium

Abstract

The application relates to the technical field of function call enhancement of AI reasoning agents, in particular to a heterogeneous AI reasoning agent function call enhancement method, system, device and medium. The method comprises the following steps: receiving an AI service request sent by a client, and analyzing the AI service request; constructing a function call prompt word according to the AI service request, injecting the AI service request into a prompt word template, and generating a prompt word text; calling an engine adapter to convert the prompt word text into a request conforming to the interface specification of a back-end reasoning engine, and sending a reasoning request to the back-end reasoning engine; using an intelligent buffer algorithm to perform sequence matching on original text data, and performing state analysis on the sequence-matched data through an analysis state machine to extract structured function call information. The method is used to solve the problems that the strong coupling between a reasoning engine and a hardware environment leads to poor universality, cannot support low-delay streaming function call interaction, and the function call analysis logic is not robust, resulting in insufficient output reliability.

Description

Technical Field

[0001] This invention relates to the field of function call enhancement technology for AI inference agents, and more particularly to methods, systems, devices and media for function call enhancement of heterogeneous AI inference agents. Background Technology

[0002] With the rapid development of artificial intelligence technology and the increasing prevalence of intelligent agent applications, the function call capability of large language models has become a core function of AI service platforms. Currently, mainstream AI service providers support function calls at the API level, and their API interface specifications have become the de facto industry standard. However, when adapting to domestic AI chips and heterogeneous computing environments, the level of support for function calls varies significantly among inference engines from different manufacturers. While some inference engines provide standard dialogue interfaces, their implementations do not include or fully support function call semantics, resulting in users being unable to obtain compliant function call responses. Upgrading the engine, on the other hand, faces high costs and risks.

[0003] Therefore, a solution is urgently needed to address the problems of poor versatility caused by the strong coupling between the inference engine and the hardware environment, the inability to support low-latency streaming function call interaction, and the insufficient output reliability caused by the unromantic function call parsing logic. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method, system, device and medium for enhancing function calls of heterogeneous AI inference agents, so as to solve the problems of poor versatility caused by strong coupling between the inference engine and the hardware environment, inability to support low-latency streaming function call interaction, and insufficient output reliability caused by the unromantic function call parsing logic.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: In a first aspect, the present invention provides a method for enhancing heterogeneous AI inference proxy function calls, including: receiving an AI service request sent by a client and parsing the AI ​​service request; The AI ​​service request is invoked according to the constructor function, and the AI ​​service request is injected into the prompt word template to generate the prompt word text. The engine adapter is invoked to convert the prompt text into a request that conforms to the backend inference engine interface specification, and then an inference request is sent to the backend inference engine. It receives raw text data returned by the backend inference engine, performs sequence matching on the raw text data using the intelligent buffer algorithm, and performs state parsing on the sequence-matched data through the parsing state machine to extract structured function call information; The structured function call information is encapsulated into a response message that conforms to the standard API specification and returned to the client.

[0006] The beneficial effects of this invention are as follows: It decouples function call capabilities from the underlying hardware and inference engine through an engine adapter, solving the problem of poor versatility caused by strong coupling; it utilizes an intelligent buffer algorithm and a unified parsing state machine to support unified streaming and non-streaming processing and real-time incremental returns, solving the problem that batch processing mode cannot support low-latency streaming interaction; it separates inference generation from structured parsing, avoiding errors caused by model output format deviations through an independent and robust parsing state mechanism, thus improving the reliability and robustness of parsing.

[0007] Based on the above technical solution, the present invention can be further improved as follows.

[0008] Furthermore, the calling engine adapter converts the prompt text into a request that conforms to the backend inference engine interface specification, including: Read configuration information to determine the type of the backend inference engine; Select the corresponding engine adapter based on the determined type; The engine adapter maps the uniform request parameters to the parameter format supported by the corresponding type of inference engine, and generates request messages that meet the API interface requirements of the corresponding type of inference engine.

[0009] The advantages of adopting the above-mentioned further solutions are that they achieve compatibility with different inference engines, decouple the system from the underlying hardware, and improve the system's versatility.

[0010] Furthermore, the step of using the intelligent buffer algorithm to perform sequence matching on the original text data includes: Construct a text buffer to receive and retain incremental text fragments; The current incremental text fragment is merged with the reserved incremental text fragment in the buffer to obtain the text to be detected; The text to be detected is matched with a predefined sequence of tool call markers, which includes a tool call start marker and a tool call end marker. If the text to be detected only contains a prefix of a predefined sequence, the text to be detected is kept in the buffer, marked as a reserved incremental text fragment, and waits for subsequent incremental text fragments; If the text to be detected contains a complete match or suffix of a predefined sequence, the valid content is extracted based on the matching result and the text buffer is updated. This enables accurate recognition of tool call tags across multiple text segments.

[0011] The beneficial effects of adopting the above-mentioned further scheme are as follows: through the dynamic merging of the buffer and the prefix / suffix matching logic, the boundary markers spanning multiple text segments can be accurately restored and identified, avoiding parsing errors or data loss caused by marker truncation, significantly improving the accuracy and robustness of streaming function call interaction, and ensuring reliable parsing in the low-latency interaction process.

[0012] Furthermore, the extraction of structured function call information includes: Initialize the state of the parsing state machine; When a tool call start marker is detected, the state is switched to name resolution state to resolve the function name; Once the function name is resolved, the state is switched to parameter resolution state, and an incremental JSON parsing strategy is used to perform piecewise parsing and difference extraction of the function parameters; When the end marker of a tool call is detected, the state is switched to the completed state, and the structured function call information of the current tool call is output.

[0013] The beneficial effect of adopting the above-mentioned further solution is that it provides a parsing logic framework, ensuring the accurate extraction of function names and parameters.

[0014] Furthermore, the states include: initial state, name resolution state, parameter resolution state, and completion state.

[0015] Furthermore, the AI ​​service request includes: message content and tool definition.

[0016] Secondly, this invention provides a heterogeneous AI inference agent function call enhancement system, including: The API gateway module receives AI service requests sent by clients and parses the AI ​​service requests. The model interface module calls the prompt word according to the AI ​​service request constructor, injects the AI ​​service request into the prompt word template, and generates the prompt word text; The engine adaptation module calls the engine adapter to convert the prompt text into an engine-specific format request and sends an inference request to the backend inference engine. The tool calls the parsing module to receive the raw text data returned by the backend inference engine, uses the intelligent buffer algorithm to perform sequence matching on the raw text data, and uses the parsing state machine to perform state parsing on the data after sequence matching to extract structured function call information; The response building module encapsulates structured function call information into response messages that conform to standard API specifications and returns them to the client.

[0017] Thirdly, the present invention provides a computer device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the heterogeneous AI inference proxy function call enhancement method are implemented.

[0018] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the heterogeneous AI inference agent function call enhancement method. Attached Figure Description

[0019] Figure 1 A flowchart illustrating the process of calling the enhancement method for heterogeneous AI inference proxy functions; Figure 2 A schematic diagram of the intelligent buffer algorithm for enhancing the function call method of heterogeneous AI inference proxy; Figure 3 A schematic diagram of the unified parsing state machine process for enhancing the function call of heterogeneous AI inference proxy functions; Figure 4 A flowchart illustrating the process of enhancing the system for heterogeneous AI inference proxy function calls; Figure 5 A flowchart illustrating the process of the model interface module of the system for enhancing heterogeneous AI inference proxy function calls; Figure 6 A flowchart illustrating the engine adaptation module process for enhancing the system's heterogeneous AI inference proxy function calls. Figure 7 This is a schematic diagram of the electronic device of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. In the description of the present invention, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. The "or" in the present invention is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A or B can represent: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Furthermore, in the description of the present invention, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items.

[0021] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0022] In this embodiment of the invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this embodiment of the invention should not be construed as superior or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0023] Example 1, as Figure 1 As shown in the figure, an embodiment of the present invention provides a method for enhancing heterogeneous AI inference proxy function calls, including: S1: Receive AI service requests sent by the client and parse the AI ​​service requests; S2: Based on the AI ​​service request constructor, call the prompt word, inject the AI ​​service request into the prompt word template, and generate the prompt word text; S3: Call the engine adapter to convert the prompt text into a request that conforms to the backend inference engine interface specification, and send an inference request to the backend inference engine; S4: Receives the raw text data returned by the backend inference engine, uses the intelligent buffer algorithm to perform sequence matching on the raw text data, and uses the parsing state machine to parse the state of the sequence-matched data and extract structured function call information; S5: Encapsulates the structured function call information into a response message that conforms to the standard API specification and returns it to the client.

[0024] It should be noted that by using an engine adapter to convert the unified request into an engine-specific format, the function call capability is decoupled from the underlying hardware and inference engine, which solves the problem of poor universality caused by strong coupling in the existing technology. It can transparently inject standardized function call capability into heterogeneous engines that do not support function calls. Furthermore, by using an intelligent buffer algorithm and a unified parsing state machine to perform real-time sequence matching and state parsing of the original text data, it can support unified processing of streaming and non-streaming requests, realize incremental extraction and real-time return of function call information, and solve the problem that existing technologies using batch processing mode cannot support low-latency streaming interaction. Furthermore, by separating the two concerns of model inference generation and output structured parsing, and using an independent and robust parsing state machine to process the raw text output by the model, information loss or parsing errors caused by deviations in the model output format are effectively avoided. This solves the problem of insufficient output reliability caused by the lack of robust parsing logic in existing technologies, and significantly improves the robustness and maintainability of the system.

[0025] Example 2, a second embodiment of the present invention, provides an enhanced method for calling heterogeneous AI inference proxy functions, including: It should be noted that the specific implementation method for receiving the AI ​​service request sent by the client in step S1 is as follows: Within the proxy layer service, an HTTP server compatible with the OpenAI Chat Completions API is started. This server listens on a specific port (e.g., 1025) and is configured with routing to receive user requests.

[0026] The specific implementation method for parsing the AI ​​service request in step S1 is as follows: At the request processing entry point, the request body is parsed to check for the presence of the `tools` parameter to determine if it is a function call request. Simultaneously, middleware is integrated into the request processing chain to handle authentication, logging, token counting, and request tracing. When a function call request is detected and the system is configured to enable it, the request flow is redirected to the internal tool call processing logic, rather than being directly forwarded to the backend inference engine.

[0027] It should be noted that step S1 completes the interception and identification of standard API requests.

[0028] Furthermore, in this embodiment of the application, the AI ​​service request is a message content and a tool definition.

[0029] It should be noted that the specific implementation of step S2, which involves calling the prompt word based on the AI ​​service request constructor, injecting the AI ​​service request into the prompt word template, and generating the prompt word text, is as follows: In the internal tool call processing logic, the message content and tool definition are first extracted from the intercepted requests. Then, using a preset prompt word template system, the messages2prompt method is used to convert the standard OpenAI format messages and tool lists into prompt word strings that the model can understand, containing specific function call instructions and tool descriptions.

[0030] In this embodiment of the application, step S3 involves the calling engine adapter converting the prompt text into a request that conforms to the backend inference engine interface specification, and sending an inference request to the backend inference engine, including steps A1-A3: A1: Read the configuration information to determine the type of the backend inference engine.

[0031] A2: Select the corresponding engine adapter based on the determined type.

[0032] A3: Use the engine adapter to map the unified format request parameters to the parameter format supported by the corresponding type of inference engine, and generate request messages that meet the API interface requirements of the corresponding type of inference engine.

[0033] It should be noted that the specific implementation methods for steps A1-A3 are as follows: Template injection similar <tools> ...< / tools> and<tool_call> ...< / tool_call> The proxy layer then determines the backend inference engine type based on the configured environment variable ENGINE (e.g., vllm or mindie). Depending on the engine type, the adaptation layer encapsulates the prompt string, request parameters (such as temperature and max_tokens), and stop characters into the format required by the engine's specific API (e.g., vllm's / v1 / completions or mindie's / generate interface), and asynchronously sends the request via an HTTP client. The backend inference engine receives this enhanced prompt, generates text data containing the original function call markers, and returns it to the proxy layer.

[0034] In this embodiment of the application, step S4, which involves receiving the raw text data returned by the backend inference engine, performing sequence matching on the raw text data, and parsing the state of the sequence-matched data using a parsing state machine, includes steps B1-B5: B1: Construct a text buffer to receive and retain incremental text fragments.

[0035] B2: Merge the current incremental text fragment with the reserved incremental text fragment in the buffer to obtain the text to be detected.

[0036] B3: Match the text to be detected with a predefined tool call marker sequence, which includes a tool call start marker and a tool call end marker.

[0037] B4: If the text to be detected only contains a prefix of a predefined sequence, then the text to be detected is kept in the buffer, marked as a reserved incremental text fragment, and waits for subsequent incremental text fragments.

[0038] B5: If the text to be detected contains a complete match or suffix of a predefined sequence, then extract the valid content based on the matching result and update the text buffer to achieve accurate recognition of tool call tags across multiple text segments.

[0039] In this embodiment of the application, the extraction of structured function call information in step S4 includes steps C1-C4: C1: Initializes the state machine's state, which includes the initial state, name resolution state, parameter resolution state, and completion state.

[0040] C2: When a tool call start marker is detected, the state is switched to name resolution state to resolve the function name.

[0041] C3: After the function name is resolved, the state is switched to the parameter resolution state. An incremental JSON parsing strategy is used to perform piecewise parsing and difference extraction of function parameters.

[0042] C4: When the end marker of a tool call is detected, the state is switched to the completed state, and the structured function call information of the current tool call is output.

[0043] Furthermore, the specific implementation methods for steps B1-B5 and C1-C4 are as follows: After receiving the raw text data returned by the backend inference engine, the proxy layer initiates the data parsing process. For non-streaming function responses, it directly calls the tool call parsing state machine, uses regular expressions to match all tool call blocks, parses the JSON content, and encapsulates the parsed function names and parameter strings into a list of ToolCall objects, while separating and cleaning up any accompanying plain text content. For streaming function responses, it starts a streaming parsing state machine, which operates incrementally: it processes arriving tokens one by one using a smart buffer algorithm, accurately identifying the start and end markers of tool calls and the start and end marks of the model itself; it tracks the current parsing state (such as parsing function names or parameters) through an internally maintained unified parsing state machine, and uses a "partial JSON parsing library" to gradually build up the complete function call parameters. Regardless of whether it's streaming or non-streaming, the parsing state machine ultimately transforms the raw data into structured function call information. Finally, the response building layer (such as the ResponseChatMessage class) encapsulates this structured information into a ChatCompletionResponse (non-streaming) or a series of ChatCompletionStreamResponse (streaming) message bodies containing fields such as choices and finish_reason, in accordance with the OpenAI API specification.

[0044] It should be noted that raw text data refers to the unstructured text string directly generated and returned by the backend inference engine (such as VLLM, Mindie, and other heterogeneous computing power engines) after receiving the inference request sent by the proxy layer. Raw text data is a text stream directly output based on prompt words. For the proxy layer, it is the lowest-level input data and has not yet been parsed into the JSON object format required by the client.

[0045] It should be noted that the parsing state machine maintains independent state variables and parameter buffers for each tool call index to support independent parsing of multiple tool calls that occur in parallel or consecutively.

[0046] It should be noted that, in the embodiments of this application, the preferred implementation for sequence matching of the original text data is to use a smart buffer algorithm to perform sequence matching on the original text data, such as... Figure 2 As shown, the specific implementation method is as follows: Supports multiple predefined sequences (such as<tool_call> ,< / tool_call> The algorithm internally maintains a predefined list of sequence strings (sequence_strings), which includes not only tool call markers but also model-specific start and end characters. For each newly arrived text segment (delta_text), the algorithm does not perform a simple string inclusion check. Instead, it merges the segment with the existing content in the buffer and then sequentially performs three precise matching logics: "complete match check," "prefix match check," and "suffix match check," to determine the segment's position in the complete sequence.

[0047] Automatically identifies sequence boundaries and correctly handles sequences spanning multiple tokens. This is especially important when the new text, after being merged with the buffer content, is merely a prefix of a predefined sequence (e.g., receiving '<', while the complete sequence is...).<tool_call> The algorithm caches the token in `buffered_delta_text` and returns an empty string, waiting for a subsequent token. When the merged text contains a suffix of a sequence (e.g., receiving `tool_call>` while the buffer already contains `<`), the algorithm will precisely keep the suffix (`tool_call>`) in the buffer and only return the prefix (`<`) as valid content, thus perfectly handling sequences broken up by the tokenizer.

[0048] Even in exceptional circumstances, it can still return valid content. If the merged text does not match any predefined sequence or part thereof, the algorithm will consider the currently accumulated text to be ordinary content, return the entire buffer content to the upper-level parsing logic, and clear the buffer, thereby avoiding parsing blockage or data loss due to unrecognized character sequences and ensuring the robustness of streaming transmission.

[0049] It should be noted that, as Figure 3 The specific implementation of the unified parsing state machine shown is as follows: a unified state machine is designed to manage the parsing process, which has the following characteristics; Each tool call maintains its own independent state. The parsing state machine (such as Qwen2d5ToolParser) internally uses `current_tool_id` to identify the index of the currently processed tool call, and associates each index with an independent state flag (such as `current_tool_name_sent` indicating whether the function name has been sent) and a parameter buffer (the `streamed_args_for_tool` list). When a tool call ends, `current_tool_id` is incremented, and the state machine initializes a completely new set of state variables for the new tool call, ensuring that the parsing process of multiple tool calls does not interfere with each other.

[0050] It supports fragmented parsing and transmission of parameters. In the "parameter parsing state," the parsing state machine uses the `partial_json_parser` library to parse incomplete JSON parameter strings. By comparing the parameter dictionaries (`prev_arguments` and `cur_arguments`) obtained from the two parsing processes and using the `extract_intermediate_diff` function to calculate the difference, it extracts the parameter text fragments added since the last response. Only this incremental fragment (not the complete parameters) is encapsulated in a `DeltaMessage` and streamed back to the client, achieving efficient and accurate incremental parameter transmission.

[0051] It supports parallel parsing of multiple tool calls. The state machine records the previous partial parsing result for each current_tool_id using the prev_tool_call_arr list. This allows for the continuous parsing of multiple tool calls within the same model response.<tool_call> During block execution, the state machine can correctly perform state switching and context management, and sequentially complete the parsing and streaming output of each tool call, simulating support for parallel tool calls.

[0052] In this embodiment of the application, step S5, which involves encapsulating the structured function call information into a response message conforming to the standard API specification and returning it to the client, is specifically implemented as follows: In the final stage of response construction, the proxy layer integrates usage statistics. For non-streaming function responses, it retrieves `prompt_tokens` and `completion_tokens` from the backend engine's response details (such as `mindie`'s `details` or `vllm`'s `usage` field). If the engine doesn't provide these, they are calculated using the built-in `TokenizerUtil`. For streaming function responses, the accumulated token usage is populated into the `usage` field in the final chunk. Subsequently, the proxy layer sends the fully encapsulated final response, conforming to the OpenAI function call format (a series of SSE-formatted `data:` blocks for streaming, and a complete JSON object for non-streaming), to the user client that initially initiated the request via an HTTP response. This process ensures that users receive the same experience as directly calling OpenAI services that support function calls, a capability that the actual backend inference engine might not originally possess.

[0053] Example 3, the third embodiment of the present invention, provides specific implementation methods for non-stream function call processing and stream function call processing, including: Step 1: Request reception and parsing.

[0054] # Client sends request { "messages": [{"role":"user","content":"Check Beijing weather"}], "tools": [{ "type":"function", "function": { "name":"get_weather", "description":"Get weather information", "parameters": {...} } }] }

[0055] Step 2: Constructing prompt words.

[0056] # Model Interface Module Construction Prompt # Original message converted to: #<|im_start|>system # You are a helpful assistant. # You may call functions... # <tools> [Tool Definition]< / tools> # ... #<|im_start|>user # Check Beijing Weather #<|im_end|> #<|im_start|>assistant Step 3: Inference Invocation.

[0057] Call the underlying inference engine through the engine adaptation layer. Transmit pre-constructed prompts Receive complete response Step 4: Response parsing and return.

[0058] # Utility calls in the parsed model output # The model output may include: #<tool_call> {"name":"get_weather","arguments": {"location":"Beijing"}}< / tool_call> # The parsed response returns an OpenAI format response: { "choices": [{ "message": { "role":"assistant", "content": null, "tool_calls": [{ "id":"call_123", "type":"function", "function": { "name":"get_weather", "arguments":"{"location": "Beijing"} } }] }, "finish_reason":"tool_calls" }] }

[0059] It should be noted that the specific implementation method for stream function call processing is as follows: Step 1: Establish streaming request.

[0060] # Client sends streaming request { "messages": [...], "tools": [...], "stream": true }

[0061] Step 2: Incremental processing flow.

[0062] # Smart buffer for handling token sequences # Assume the model returns by token: # 1. "<"-> Cache # 2. "tool"->Cache # 3."_call"->Cache # 4.">"->Complete sequence detected"<tool_call> Clear the buffer and process. # State Machine Management Analysis Process # - Detected<tool_call> Entering name resolution state # - Parse the function name: Send a Delta message containing the name # - Incremental parameter parsing: Send a Delta message containing parameter fragments # - Detected< / tool_call> Tool call complete Step 3: Streaming response return.

[0063] # Incremental return of Delta message # First chunk (send function name): { "choices": [{ "delta": { "tool_calls": [{ "index": 0, "id":"call_123", "function": {"name":"get_weather"} }] } }] } # Subsequent chunk (send parameter fragment): { "choices": [{ "delta": { "tool_calls": [{ "index": 0, "function": {"arguments":"{\"location\": \""} }] } }] } # Last chunk (complete): { "choices": [{ "delta": {}, "finish_reason":"tool_calls" }] }

[0064] Example 4 provides a heterogeneous AI inference proxy function call enhancement system, including: API gateway module, model interface module, engine adaptation module, tool call parsing module, and response construction module.

[0065] The API gateway module receives AI service requests sent by clients and parses the AI ​​service requests. It should be noted that the specific functions implemented by the API gateway module are: protocol compatibility, request distribution, middleware chain, and error handling; It should be noted that the specific implementation of protocol compatibility is as follows: Full compatibility with the OpenAI Chat Completions API. All endpoints provided by the API gateway module (such as / v1 / chat / completions) strictly adhere to the OpenAI official API specification, defining request parameters (such as messages, model, tools, stream) and response structures (such as choices, usage, finish_reason). Clients can directly use the SDK designed for OpenAI to call this proxy service without modifying their code.

[0066] It should be noted that the specific implementation of request distribution is as follows: Whether to enable function call processing is determined based on the request content. The gateway parses the request body (request_data) in its routing function (e.g., check_openapi_chat). If the request body contains a non-empty `tools` field, and the system-wide configuration (environment variable AICP_TOOL_CALL_EXT) has enabled tool call extensions, the request is routed to an internal, independent function call processing flow (the `tool_call` function), rather than simply being proxied to the backend inference service, thereby activating the core processing logic.

[0067] It should be noted that the specific implementation of the middleware chain is as follows: it supports multiple middleware such as authentication, logging, and token counting. The gateway application (app) registers a series of middleware in the request processing pipeline: CountingTokenMiddleware: Automatically injects the stream_options parameter into streaming requests to ensure that token usage statistics can be obtained from the backend; XRequestIdMiddleware: Generates or transmits a unique X-Request-Id for each request, used for end-to-end tracing; Authentication middleware (implemented via the dependency `check_api_key`): verifies the `Authorization:Bearer` token in the request header, ensuring secure API access. This middleware performs preprocessing before the request reaches the core business logic, enhancing system observability and security.

[0068] It should be noted that the specific implementation method for error handling is as follows: Unified exception handling and response formatting. The API gateway module captures and handles various exceptions (such as RequestValidationError for parameter validation, InvalidParamError for invalid parameters, and HTTPException for authentication failure) through global exception handlers (such as request_validation_exception_handler). All errors are converted into structured JSON error responses, with a format consistent with OpenAI API error responses (including fields such as message, type, and code), ensuring that clients receive consistent and user-friendly error messages.

[0069] The model interface module calls the prompt word based on the AI ​​service request constructor, injects the AI ​​service request into the prompt word template, and generates the prompt word text.

[0070] It should be noted that, as Figure 5As shown, the specific implementation method for generating prompt text in the model interface module is as follows: The OpenAI-formatted tool definitions are converted into a model-understandable format. This layer (such as the `messages2prompt` method of the `FunctionCall` class) reads the list of tools from the request, iterates through each tool, and serializes its function fields (containing name, description, and parameters) into a JSON string. Then, all the JSON strings of the tool definitions are concatenated and wrapped as a whole. <tools> and< / tools> Within the XML tag, it is inserted into the system instruction section, thereby converting the abstract interface description into text instructions that the model can directly process.

[0071] Add a description of the function call output format. (In the tool definition...) <tools> ...< / tools> Following this, the model interface module will then inject explicit output format instructions (such as "For each function call return a jsonobject..." within).<tool_call>< / tool_call> This instruction explicitly tells the large model to be wrapped in a specific JSON object format (XML tags).<tool_call> Replying within tags is key to guiding models that don't normally support function calls to generate structured call data.

[0072] Properly handle conversations containing tool call history. For assistant messages in the conversation history (messages), if they contain a tool_calls field, the model interface layer will reformat its content (function name and parameter string) to...<tool_call> {...}< / tool_call> The text block is then inserted into the corresponding position of the prompt word. For messages from the tool role (function execution result), its content is wrapped in...<tool_response> ...< / tool_response> Inserting tags. This process allows the model to understand the context of previous function calls in multi-turn dialogues.

[0073] The engine adaptation module calls the engine adapter to convert the prompt text into an engine-specific format request and sends an inference request to the backend inference engine. It should be noted that, as Figure 6 As shown, the specific implementation method of the engine adaptation module is as follows: The unified parameters are converted into engine-specific parameters. The engine adaptation module selects different parameter construction logic in the `tool_call` function based on the value of the environment variable `ENGINE` (such as `vllm` or `mindie`). For example, for the `max_tokens` parameter, the `vllm` adapter needs to combine `MAX_SEQ_LEN` and calculate the number of available tokens using `TokenizerUtil`, while the `mindie` adapter directly uses `MAX_NEW_TOKENS`. For the `stop` parameter, adaptation is required based on the format supported by the engine. This mapping process translates the unified request representation within this application embodiment into the specific parameter format expected by the backend engine API.

[0074] The response from different engines is converted into a unified format. After receiving the raw response from the backend engine, the adaptation layer performs standardization operations. For the vllm engine, choices[0].text is extracted from the response of the / v1 / completions interface as the generated text; for the mindie engine, generated_text is extracted from the response of the / generate interface. At the same time, regardless of whether the backend returns usage, details, or other fields, the engine adaptation module uniformly converts them into the internally defined UsageInfo structure, which contains standard fields such as prompt_tokens and completion_tokens, providing a consistent interface for the upper layer.

[0075] Handling engine-specific exceptions. The engine adaptation module needs to identify and handle unique exceptions or error status codes that different engines may throw, such as connection timeouts and model not loaded. It encapsulates engine-specific error information and converts it into a unified error response format for the proxy layer, ensuring that users are not directly exposed to the complex or inconsistent error details of the underlying engine, thus improving service stability and user experience consistency.

[0076] The tool calls the parsing module to receive the raw text data returned by the backend inference engine, uses the intelligent buffer algorithm to perform sequence matching on the raw text data, and uses the parsing state machine to perform state parsing on the data after sequence matching to extract structured function call information; The response building module encapsulates structured function call information into response messages that conform to standard API specifications and returns them to the client.

[0077] Example 5: In some embodiments, the heterogeneous AI inference agent function call enhancement system of the present invention can be implemented in a combination of hardware and software. As an example, the heterogeneous AI inference agent function call enhancement system of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the heterogeneous AI inference agent function call enhancement method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0078] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0079] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned heterogeneous AI inference proxy function call enhancement methods. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the heterogeneous AI inference proxy function call enhancement method shown in any embodiment of the present invention by calling the computer program.

[0080] In one alternative embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0081] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0082] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0083] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0084] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0085] Among them, electronic devices can also be terminal devices, which can be any device that can install applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0086] It should be noted that, Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0087] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned methods.

[0088] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0089] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the above-described...

[0090] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0091] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0092] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0093] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0094] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0095] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0096] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0097] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for enhancing heterogeneous AI inference proxy function calls, characterized in that, include: Receive AI service requests sent by clients and parse the AI ​​service requests; The AI ​​service request is invoked according to the constructor function, and the AI ​​service request is injected into the prompt word template to generate the prompt word text. The engine adapter is invoked to convert the prompt text into a request that conforms to the backend inference engine interface specification, and then an inference request is sent to the backend inference engine. It receives raw text data returned by the backend inference engine, performs sequence matching on the raw text data using the intelligent buffer algorithm, and performs state parsing on the sequence-matched data through the parsing state machine to extract structured function call information; The structured function call information is encapsulated into a response message that conforms to the standard API specification and returned to the client.

2. The heterogeneous AI inference proxy function call enhancement method according to claim 1, characterized in that, The calling engine adapter converts the prompt text into a request that conforms to the backend inference engine interface specification, including: Read configuration information to determine the type of the backend inference engine; Select the corresponding engine adapter based on the determined type; The engine adapter maps the unified format of request parameters to the parameter format supported by the corresponding type of inference engine, and generates request messages that meet the API interface requirements of the corresponding type of inference engine.

3. The heterogeneous AI inference proxy function call enhancement method according to claim 1, characterized in that, The step of using a smart buffer algorithm to perform sequence matching on the original text data includes: Construct a text buffer to receive and retain incremental text fragments; The current incremental text fragment is merged with the reserved incremental text fragment in the buffer to obtain the text to be detected; The text to be detected is matched with a predefined sequence of tool call markers, which includes a tool call start marker and a tool call end marker. If the text to be detected only contains a prefix of a predefined sequence, the text to be detected is kept in the buffer, marked as a reserved incremental text fragment, and waits for subsequent incremental text fragments; If the text to be detected contains a complete match or suffix of a predefined sequence, then the valid content is extracted based on the matching result and the text buffer is updated.

4. The heterogeneous AI inference proxy function call enhancement method according to claim 1, characterized in that, The extraction of structured function call information includes: Initialize the state of the parsing state machine; When a tool call start marker is detected, the state is switched to name resolution state to resolve the function name; Once the function name is resolved, the state is switched to parameter resolution state, and an incremental JSON parsing strategy is used to perform piecewise parsing and difference extraction of the function parameters; When the end marker of a tool call is detected, the state is switched to the completed state, and the structured function call information of the current tool call is output.

5. The heterogeneous AI inference proxy function call enhancement method according to claim 4, characterized in that, The states include: initial state, name resolution state, parameter resolution state, and completion state.

6. The heterogeneous AI inference proxy function call enhancement method according to any one of claims 1 or 4, characterized in that, The parsing state machine maintains independent state variables and parameter buffers for each tool's call index.

7. The heterogeneous AI inference proxy function call enhancement method according to claim 1, characterized in that, The AI ​​service request includes: message content and tool definition.

8. A heterogeneous AI inference proxy function call enhancement system, using the method described in any one of claims 1-7, characterized in that, include: The API gateway module receives AI service requests sent by clients and parses the AI ​​service requests. The model interface module calls the prompt word according to the AI ​​service request constructor, injects the AI ​​service request into the prompt word template, and generates the prompt word text; The engine adaptation module calls the engine adapter to convert the prompt text into an engine-specific format request and sends an inference request to the backend inference engine. The tool calls the parsing module to receive the raw text data returned by the backend inference engine, uses the intelligent buffer algorithm to perform sequence matching on the raw text data, and uses the parsing state machine to perform state parsing on the data after sequence matching to extract structured function call information; The response building module encapsulates structured function call information into response messages that conform to standard API specifications and returns them to the client.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images