A protocol-independent AI inference gateway bidirectional conversion method and system

Abstract

The application belongs to the technical field of artificial intelligence inference service, and discloses a protocol-independent AI inference gateway bidirectional conversion method and system; comprising: receiving an inference request and generating a semantic dependency graph; determining the matching relationship between each context prefix identifier and each candidate backend inference node and the corresponding cache hit rate according to the semantic dependency graph; selecting a target backend inference node and determining the corresponding protocol adaptation strategy; sending a request based on the protocol adaptation strategy; receiving a sub-response and performing analysis, identifying semantic boundary events, and dynamically adjusting the streaming semantic conversion strategy; incrementally reorganizing each sub-response after dynamic adjustment to obtain an inference response conforming to a second protocol format. The application obtains the protocol capability profile of the target backend inference node, determines differentiated protocol adaptation strategies for each semantic segment according to the multi-dimensional capability constraints of the backend node, and significantly improves the protocol adaptation capability and service stability of the gateway to the heterogeneous backend cluster.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence inference service technology, and more specifically, to a protocol-independent AI inference gateway bidirectional conversion method and system. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, different AI service providers have successively launched their own API service protocols, such as OpenAI's Chat Completions protocol and Anthropic's Messages protocol. In order to achieve interconnection between heterogeneous protocols, protocol-independent AI inference gateways have emerged. By exposing API endpoints compatible with different protocols in a single gateway process, the gateway converts the received inference requests into the protocol format supported by the target backend node before forwarding them, and converts the response returned by the backend back into the caller's protocol format, thereby achieving bidirectional conversion across protocols.

[0003] Existing protocol conversion gateways often route and convert inference requests as an indivisible whole, failing to identify reusable semantic units within the request (such as system prompts, document context, etc.). This results in only single-node cache matching based on the overall request, leading to low cache utilization and high inference latency. Secondly, existing gateways use a uniform and standardized protocol conversion strategy, which cannot be adapted to the actual protocol version, multimodal processing capabilities, and structured output constraints of the backend inference nodes. The converted requests often exceed the protocol processing capabilities of the backend nodes, causing request failures or abnormal responses. In addition, in streaming response scenarios, existing gateways rely on fixed state machines for event sequence conversion, failing to dynamically adjust the conversion strategy based on real-time semantic boundaries (such as tool call parameter assembly completion, JSON field closure, etc.), which easily leads to protocol-level errors. Furthermore, existing gateways lack the ability to perform semantic-level incremental reassembly of sub-responses returned in parallel by multiple nodes. The token uses statistical fields and only performs simple mapping, failing to reflect the actual contribution weight of each semantic fragment in the complete response.

[0004] In view of this, the present invention proposes a protocol-independent AI inference gateway bidirectional conversion method and system to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art and achieve the above objectives, the present invention provides the following technical solution: A protocol-independent bidirectional conversion method for AI inference gateways, the method comprising: Receive inference requests in the second protocol format of the second AI service protocol, and perform semantic dependency parsing on the inference requests to generate a semantic dependency graph; Obtain the context prefix identifier of each semantic segment in the semantic dependency graph, query the KV-Cache prefix cache status of each candidate backend inference node, determine the matching relationship between each context prefix identifier and each candidate backend inference node, and the cache hit rate corresponding to each matching relationship; The target backend inference node for each semantic segment is determined from the candidate backend inference nodes, and the corresponding protocol adaptation strategy is determined based on the pre-set protocol capability profile of each target backend inference node. Based on the protocol adaptation strategy, each semantic segment is converted from the second protocol format to the first protocol format and then forwarded to the corresponding target backend inference node; wherein the first protocol format is a protocol format compatible with the first AI service protocol; Receive sub-responses in the first protocol format returned by each target backend inference node, parse the incremental content of each sub-response in real time based on the semantic dependency graph to identify semantic boundary events, and dynamically adjust the streaming semantic conversion strategy of the sub-responses from the first protocol format to the second protocol format according to the semantic boundary events. Incremental recombination is performed on each dynamically adjusted sub-response to obtain an inference response that conforms to the second protocol format.

[0006] Furthermore, the method for generating a semantic dependency graph is as follows: Extract the message sequence from the inference request, as well as the message role identifier and content type identifier of each message in the message sequence; Based on message role identifiers and content type identifiers, the message sequence is split into multiple initial semantic segments, and the semantic integrity of each initial semantic segment is verified to obtain multiple semantic segments; Obtain the position and role identifier of each message in the message sequence, and establish the contextual reference relationship between each semantic fragment; Obtain the pre-set length character sequence and rolling hash digest of each semantic segment, and generate the corresponding context prefix identifier; Construct a semantic dependency graph using semantic fragments as nodes, context references as edges, and context prefixes as node attributes.

[0007] Furthermore, the method for determining the matching relationship between each context prefix identifier and each candidate backend inference node is as follows: Iterate through the KV-Cache prefix cache status of each candidate backend inference node to obtain the cached context prefix identifier of each candidate backend inference node; The context prefix identifier of each semantic segment is matched with the context prefix identifier that has been cached by each candidate backend inference node. If the match is successful, it is determined that a matching relationship exists.

[0008] Furthermore, the method for determining the cache hit rate corresponding to each matching relationship is as follows: For each candidate backend inference node with a matching relationship, query the historical request records of each candidate backend inference node for the cached context prefix identifier corresponding to the matching relationship within a preset time window, and use the ratio of the number of hits in the historical request records to the total number of requests as the cache hit rate of each candidate backend inference node.

[0009] Furthermore, the method for determining the target backend inference node for each semantic segment is as follows: Candidate backend inference nodes are divided into a set of matching nodes and a set of non-matching nodes. When the set of matching nodes is not empty, each candidate backend inference node in the set of matching nodes is sorted in descending order according to the cache hit rate, and the top k candidate backend inference nodes are selected as candidate preferred nodes. Obtain the GPU status information of each candidate node and calculate the GPU comprehensive value. Add the GPU comprehensive value of each candidate node to the cache hit rate with weighted summation to obtain the comprehensive score. Select the candidate node with the highest comprehensive score as the corresponding target backend inference node. When the set of matching nodes is empty, obtain the GPU composite value of each candidate backend inference node in the set of non-matching nodes, and determine the candidate backend inference node with the highest GPU composite value as the target backend inference node.

[0010] Furthermore, the method for determining the protocol adaptation strategy for each target backend inference node is as follows: The protocol capability profile is parsed into a multi-dimensional capability constraint vector, and the semantic fragment is parsed into a multi-dimensional semantic requirement vector. The multi-dimensional capability constraint vector and the multi-dimensional semantic requirement vector are matched to satisfy the constraints, and a set of candidate protocol adaptation strategies is obtained. Based on the contextual reference relationships of semantic segments, the context sensitivity weight and downstream reference influence factor of semantic segments are determined, and the candidate protocol adaptation strategy set is prioritized and the candidate protocol adaptation strategy with the highest ranking is determined as the protocol adaptation strategy.

[0011] Furthermore, the method for identifying semantic boundary events is as follows: Based on the contextual reference relationships of the currently processed semantic fragments, determine the expected semantic type and expected boundary features; Based on the expected semantic type and expected boundary features, the incremental content of the sub-response is synchronously parsed using dual-protocol syntax, and the first protocol abstract syntax tree and the second protocol prediction syntax tree carrying the boundary prediction confidence are incrementally constructed. The incremental nodes of the first protocol abstract syntax tree are matched with the expected semantic type. When the match is successful and the confidence of the boundary prediction exceeds the preset confidence threshold, a candidate semantic boundary is generated. The incremental node is a new syntax tree node in the first protocol abstract syntax tree that is generated due to the received incremental content. Calculate the semantic embedding vector similarity between the first protocol abstract syntax tree and the second protocol predicted syntax tree at the candidate semantic boundary, and verify the context reference relationship constraints based on the semantic dependency graph; When the semantic embedding vector similarity is greater than the preset semantic similarity threshold and the relation constraints are met, the candidate semantic boundary is confirmed as a semantic boundary event, and the event type is determined according to the expected boundary features.

[0012] Furthermore, the event types include tool call parameter assembly completion event, structured JSON field closure event, and multimodal content block end event.

[0013] Furthermore, the method for obtaining a reasoning response conforming to the second protocol format is as follows: Based on the contextual reference relationships of each semantic segment, determine the reorganization order and splicing position of each sub-response; Each sub-response is deduplicated to obtain the deduplicated sub-responses; the deduplicated sub-responses are then concatenated according to the reorganization order and splicing position to obtain the complete response content. Obtain the reference weight of each semantic fragment in the semantic dependency graph, use statistical fields to weight the token of the complete response content, and map the completion reason field according to the overall semantics of the complete response content; The complete response content, the weighted token, and the mapped completion reason field are encapsulated into an inference response conforming to the second protocol format.

[0014] A protocol-independent AI inference gateway bidirectional conversion system, the system comprising: The semantic dependency parsing module is used to receive inference requests in the second protocol format of the second AI service protocol, and to perform semantic dependency parsing on the inference requests to generate a semantic dependency graph. The cache status query module is used to obtain the context prefix identifier of each semantic segment in the semantic dependency graph, query the KV-Cache prefix cache status of each candidate backend inference node, determine the matching relationship between each context prefix identifier and each candidate backend inference node, and the cache hit rate corresponding to each matching relationship. The protocol profiling decision module is used to determine the target backend inference node for each semantic segment from the candidate backend inference nodes, and to determine the corresponding protocol adaptation strategy based on the preset protocol capability profile of each target backend inference node. The request protocol conversion module, based on the protocol adaptation strategy, converts each semantic fragment from the second protocol format to the first protocol format and then forwards it to the corresponding target backend inference node; The streaming semantic conversion module is used to receive sub-responses in the first protocol format returned by each target backend inference node, and to parse the incremental content of each sub-response in real time based on the semantic dependency graph to identify semantic boundary events. Based on the semantic boundary events, the streaming semantic conversion strategy of sub-responses from the first protocol format to the second protocol format is dynamically adjusted. The response reorganization output module is used to incrementally reorganize the dynamically adjusted sub-responses to obtain an inference response that conforms to the second protocol format.

[0015] The technical effects and advantages of the protocol-independent AI inference gateway bidirectional conversion method and system of the present invention are as follows: This invention performs semantic dependency parsing on inference requests, splits the requests into multiple semantic fragments and establishes contextual reference relationships. Based on the context prefix identifier of each semantic fragment and the KV-Cache prefix cache status of the candidate backend inference nodes, fragment-level matching is performed, thereby routing different semantic fragments to the backend nodes with optimal caches. This significantly improves the utilization rate of KV-Cache prefix cache, reduces redundant computation overhead, and lowers the overall inference latency.

[0016] This invention obtains the preset protocol capability profile of the target backend inference node, and determines differentiated protocol adaptation strategies for each semantic segment based on multi-dimensional capability constraints such as the protocol version, context window length, multimodal processing capability, and structured output constraints of the backend node. This significantly improves the gateway's protocol adaptation capability and service stability for heterogeneous backend clusters.

[0017] In streaming response mode, this invention uses a semantic dependency graph to parse incremental content of sub-responses in real time to identify semantic boundary events, and dynamically adjusts the streaming semantic transformation strategy according to the type of semantic boundary events. This effectively avoids protocol-level errors such as tool call parameter truncation, JSON structure breakage, or multimodal content block parsing errors, and significantly improves the protocol compatibility of streaming responses and the client parsing success rate.

[0018] This invention incrementally reassembles each dynamically adjusted sub-response, determines the reassembly order based on the contextual reference relationship of the semantic dependency graph, and performs deduplication verification. At the same time, it uses statistical fields to weighted allocate tokens according to the reference weight of each semantic fragment in the semantic dependency graph. This effectively solves the problems of sub-response redundancy and disordered order in multi-node parallel processing scenarios, and enables token consumption to accurately reflect the actual semantic contribution of each semantic fragment in the complete response, thereby improving the accuracy of response integrity and resource consumption measurement. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system module block diagram of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In one embodiment, a protocol-independent bidirectional conversion method for AI inference gateways is disclosed, such as... Figure 1 As shown, the method includes: Receive inference requests in the second protocol format of the second AI service protocol, and perform semantic dependency parsing on the inference requests to generate a semantic dependency graph.

[0022] The method for generating the semantic dependency graph is as follows: extracting the message sequence and the message role identifier and content type identifier of each message in the message sequence from the inference request; Based on message role identifiers and content type identifiers, the message sequence is split into multiple initial semantic segments, and the semantic integrity of each initial semantic segment is verified to obtain multiple semantic segments; Obtain the position and role identifier of each message in the message sequence, and establish the contextual reference relationship between each semantic fragment; Obtain the pre-set length character sequence and rolling hash digest of each semantic segment, and generate the corresponding context prefix identifier; Construct a semantic dependency graph using semantic fragments as nodes, context references as edges, and context prefixes as node attributes.

[0023] Specifically, after accepting and being compatible with the second protocol format of the second AI service protocol, the inference request is parsed at the protocol layer; specifically, the protocol message body of the inference request is parsed, and the message sequence and the message role identifier and content type identifier of each message in the message sequence are extracted from it; The message role identifier is used to identify the role of a message in the dialogue sequence. Its value is derived from the role field in the message structure of the second protocol format. Message role identifiers include, but are not limited to, system role, user role, assistant role, tool role, etc. The content type identifier is used to identify the specific media type of the message payload. Its value is derived from the type field in the message content structure of the second protocol format. Content type identifiers include, but are not limited to, text type, image type, document type, tool use type, etc.

[0024] Based on message role identifiers and content type identifiers, the message sequence is split into multiple initial semantic fragments. The splitting is based on the message as the smallest unit, and further subdivision is performed within a single message according to the content type identifier. The specific splitting rules are as follows: For messages with the system role identifier, they are directly split into independent system prompt fragments. The system prompt fragments contain the entire text content of the message, and no further content type subdivision is performed. For messages with the user role identifier, they are split into text fragments, image fragments, and document fragments based on content type boundaries. The method for identifying content type boundaries is as follows: In the second protocol format, the content fields of the user role message are used as an array structure. Each element in the array structure is a content block. Each content block contains a type field and corresponding type payload data. When the value of the type field of adjacent content blocks changes, a content type boundary is formed. For example, content blocks with consecutive type fields of "text" are aggregated into text fragments; content blocks with consecutive type fields of "image" are aggregated into image fragments; and content blocks with consecutive type fields of "document" are aggregated into document fragments. For messages with the role identifier "assistant", they are split into reasoning fragments and tool call fragments based on the reasoning step boundaries. The method for identifying the reasoning step boundaries is as follows: detect whether the content fields of the assistant role message contain a tool call marker. The tool call marker includes, but is not limited to: the type field of the content block of the content field has the value "tool use", the content block of the content field contains a tool call identifier field (id field) and a tool name field (name field), the content block of the content field contains a tool parameter field (input field), and the tool parameter field starts with a JSON object structure. When a tool call marker is detected, the starting position of the content block corresponding to the tool call marker is taken as the reasoning step boundary. If the continuous text content blocks before the reasoning step boundary are aggregated into reasoning fragments, and the tool call content blocks at and after the reasoning step boundary are aggregated into tool call fragments, if there is no tool call marker, the entire message is considered as a reasoning fragment. For messages with the message role identifier "tool", they are split into independent tool call result fragments.

[0025] Then, all the split segments are combined to obtain multiple initial semantic segments, and semantic integrity is verified for each initial semantic segment; specifically, the semantic integrity score of each initial semantic segment is calculated, and the calculation dimensions of the semantic integrity score include: Syntactic integrity score: Checks whether the content of the detected fragment contains unclosed JSON structures, unmatched parentheses, or incomplete XML tags. If any are found, the score will be reduced. For example, if the base score is 1.0, 0.3 points will be deducted for each unclosed JSON structure, 0.1 points for each unmatched parenthesis, and 0.2 points for each incomplete XML tag. The minimum score is 0. Semantic coherence score: The score is determined by detecting whether there are truncated sentences or incomplete conceptual expressions at the end of a passage using a natural language processing model or rule engine. If so, the score is reduced. For example, if the base score is 1.0, 0.3 points are deducted if there is a truncated sentence at the end (such as ending with a preposition, conjunction, particle, or incomplete subject-predicate structure); 0.2 points are deducted if there is an incomplete conceptual expression at the end (such as ending with words like "for example," "firstly," or "about" without any follow-up content); the minimum score is 0. Context independence score: This score checks whether the passage contains unresolved references (such as pronouns or elliptical structures) pointing to the outside of the passage. If so, the score is reduced. For example, if the base score is 1.0, if the passage contains personal pronouns (such as he, she, it), demonstrative pronouns (such as this, that), or elliptical structures (such as predicate structures that omit the subject or object), 0.1 points will be deducted for each instance. The minimum score is 0. The semantic integrity score is obtained by weighting and summing the syntactic integrity score, semantic coherence score, and context independence score. The weights of each are determined based on the relative frequency statistics of various syntactic errors, semantic breaks, and context reference errors in the historical operation logs of the gateway deployment scenario. When the semantic integrity score of the initial semantic segment is less than the preset scoring threshold (the scoring threshold is set according to the minimum tolerance requirement of semantic integrity for the downstream application scenario served by the gateway), the initial semantic segment is merged with the adjacent initial semantic segments with context reference relationships to obtain the merged semantic segment. When the semantic integrity score of the initial semantic segment is greater than or equal to the preset scoring threshold, the initial semantic segment is directly identified as a semantic segment. The above verification and merging process is repeated until the semantic integrity scores of all semantic segments are greater than or equal to the scoring threshold, and finally multiple semantic segments are obtained.

[0026] Obtain the position and role identifier of each message in the message sequence, and establish contextual reference relationships between semantic fragments; the specific rules for establishing these relationships are as follows: The semantic fragments corresponding to system roles are referenced by the semantic fragments corresponding to all user roles and assistant roles, forming a one-way reference relationship; The semantic fragments corresponding to adjacent user roles and assistant roles form a bidirectional reference relationship; The semantic fragment (tool call result fragment) corresponding to the tool role unidirectionally references the tool call fragment of the corresponding assistant role; Furthermore, the contextual reference relationship carries a reference direction identifier (one-way or two-way).

[0027] For each semantic segment, the pre-set length character sequence of the semantic segment and the rolling hash digest are obtained, and the corresponding context prefix identifier is generated. Specifically, the pre-set length character sequence is the first N characters of the semantic segment (N is a preset value, such as 128 characters). The rolling hash digest is calculated by the Rabin-Karp rolling hash algorithm on the pre-set length character sequence to obtain a hash value of fixed length. The context prefix identifier is composed of the pre-set length character sequence and the rolling hash digest.

[0028] Semantic fragments, context references, and context prefix identifiers are organized into a directed acyclic graph structure to obtain a semantic dependency graph. In the semantic dependency graph, nodes represent semantic fragments, edges represent context references, and edges carry reference direction identifiers; the context prefix identifiers corresponding to each node are stored as node attributes in the node.

[0029] By splitting inference requests from the traditional whole request processing granularity to the semantic fragment granularity and establishing contextual reference relationships between semantic fragments, fine-grained modeling of the internal semantic structure of inference requests is achieved. Semantic integrity verification ensures the syntactic and semantic consistency of the split semantic fragments, avoiding semantic breaks caused by excessive splitting. Moreover, the generation of context prefix identifiers enables subsequent steps to perform cache matching based on the semantic fragment level, rather than just matching based on the whole request, thereby significantly improving the utilization efficiency of KV-Cache prefix caching.

[0030] Obtain the context prefix identifier of each semantic segment in the semantic dependency graph, query the KV-Cache prefix cache status of each candidate backend inference node, determine the matching relationship between each context prefix identifier and each candidate backend inference node, and the cache hit rate corresponding to each matching relationship.

[0031] The method for determining the matching relationship between each context prefix identifier and each candidate backend inference node is as follows: traverse the KV-Cache prefix cache status of each candidate backend inference node and obtain the cached context prefix identifier of each candidate backend inference node. The context prefix identifier of each semantic segment is matched with the context prefix identifier cached by each candidate backend inference node. If the match is successful, it is determined that a matching relationship exists. The method for determining the cache hit rate corresponding to each matching relationship is as follows: For each candidate backend inference node with a matching relationship, query the historical request records of each candidate backend inference node for the cached context prefix identifier corresponding to the matching relationship within a preset time window, and take the ratio of the number of hits in the historical request records to the total number of requests as the cache hit rate of each candidate backend inference node.

[0032] Specifically, the KV-Cache prefix cache state of each candidate backend inference node is traversed to obtain the cached context prefix identifiers of each candidate backend inference node. The KV-Cache prefix cache state is maintained by each candidate backend inference node, recording the set of context prefix identifiers cached by that candidate backend inference node in the KV-Cache, as well as the cache metadata (such as cache time, cache token length, etc.) corresponding to each cached context prefix identifier. The context prefix identifiers of each semantic fragment are matched with the cached context prefix identifiers of each candidate backend inference node for prefix matching verification. The specific methods of prefix matching verification include exact matching, longest common prefix matching, and rolling hash similarity matching, among which: Exact match: If the context prefix identifier of the semantic fragment is exactly the same as the context prefix identifier already cached by the candidate backend inference node, then the matching condition is satisfied; Longest common prefix matching: Calculate the length of the longest common prefix between the context prefix identifier of the semantic segment and the cached context prefix identifier. When the length of the longest common prefix exceeds the preset length threshold (set based on experience, such as 100 characters), it is determined that the matching condition is met. Rolling hash similarity matching: Calculate the Hamming distance between the rolling hash digests of the two. When the Hamming distance is less than a preset distance threshold, the matching condition is determined. The preset distance threshold is determined based on the total number of bits in the rolling hash digest, which is usually 10% of the total number of bits.

[0033] When the context prefix identifier of a semantic fragment and the context prefix identifier cached by the candidate backend inference node satisfy any matching condition, it is determined that there is a matching relationship between the context prefix identifier of the corresponding semantic fragment and the corresponding candidate backend inference node.

[0034] For each candidate backend inference node identified as having a matching relationship, query the historical request records (including the number of hits and the total number of requests) of each candidate backend inference node for the cached context prefix identifier corresponding to the matching relationship within a preset time window (based on experience, such as the last 5 minutes), and calculate the cache hit rate. The cache hit rate is the ratio of the number of hits of the cached context prefix identifier within the preset time window to the total number of requests. The number of hits is the number of times the candidate backend inference node receives an inference request containing the cached context prefix identifier and successfully reuses the cache from the KV-Cache within the preset time window. The total number of requests is the total number of inference requests containing the cached context prefix identifier received by the candidate backend inference node within the preset time window.

[0035] In this approach, by introducing KV-Cache prefix cache status query and semantic fragment-level matching verification before routing decisions, the gateway can accurately identify which backend nodes have cache reuse potential for each semantic fragment. Compared to traditional whole request prefix matching, semantic fragment-level matching can separate reusable local contexts (such as system prompts and fixed document blocks) from non-reusable dynamic parts (such as the latest user input) in long requests, thereby routing different fragments to different backend inference nodes with optimal caching, which can significantly reduce redundant computation overhead and improve overall inference throughput.

[0036] The target backend inference node for each semantic segment is determined from the candidate backend inference nodes, and the corresponding protocol adaptation strategy is determined based on the pre-set protocol capability profile of each target backend inference node.

[0037] The method for determining the target backend inference node for each semantic segment is as follows: divide the candidate backend inference nodes into a set of matching nodes and a set of non-matching nodes. When the set of matching nodes is not empty, sort the candidate backend inference nodes in the set of matching nodes in descending order according to the cache hit rate, and select the top k candidate backend inference nodes as the preferred candidate nodes. Obtain the GPU status information of each candidate node, calculate the GPU comprehensive value, and weight and sum the GPU comprehensive value of each candidate node with the cache hit rate to obtain a comprehensive score. Select the candidate node with the highest comprehensive score as the corresponding target backend inference node. When the matching node set is empty, obtain the GPU composite value of each candidate backend inference node in the non-matching node set, and determine the candidate backend inference node with the highest GPU composite value as the target backend inference node. The method for determining the protocol adaptation strategy for each target backend inference node is as follows: the protocol capability profile is parsed into a multi-dimensional capability constraint vector, and the semantic fragment is parsed into a multi-dimensional semantic requirement vector; the multi-dimensional capability constraint vector and the multi-dimensional semantic requirement vector are matched to satisfy the constraints, and a set of candidate protocol adaptation strategies is obtained. Based on the contextual reference relationships of semantic segments, the context sensitivity weight and downstream reference influence factor of semantic segments are determined, and the candidate protocol adaptation strategy set is prioritized and the candidate protocol adaptation strategy with the highest ranking is determined as the protocol adaptation strategy.

[0038] Specifically, candidate backend inference nodes are divided into a matching node set and a non-matching node set. The matching node set contains candidate backend inference nodes that have a matching relationship with the context prefix identifier of the semantic segment; the non-matching node set contains candidate backend inference nodes that do not have a matching relationship.

[0039] When the matching node set is not empty, the candidate backend inference nodes in the matching node set are sorted in descending order of cache hit rate, and the top k (set according to the experience of those in the art, for example, the top 3) candidate backend inference nodes are selected as candidate preferred nodes; at the same time, the GPU status information of each candidate preferred node is obtained. The GPU status information includes, but is not limited to: GPU idle computing power ratio, remaining memory ratio, current queued request count, and inference batch size, etc. The GPU comprehensive value is calculated based on the GPU status information. For example, the GPU idle computing power ratio and remaining memory ratio are normalized to obtain the idle computing power value and remaining memory value, respectively. The current queued request count and inference batch size are first mapped to the [0,1] interval through min-max normalization. Then, subtract their respective normalized values ​​from 1 to obtain the queue idle value and batch idle value. Finally, the idle computing power value, the remaining video memory value, the queue idle value, and the batch idle value are weighted and summed to obtain the GPU comprehensive value. Here, the weight coefficients of each sub-value are preset by those skilled in the art according to the actual application scenario's requirements for balancing inference latency, throughput, and GPU resource utilization, and the sum of each weight coefficient is 1. For example, in a latency-sensitive scenario, the weight of the idle computing power value is set to 0.4, the weight of the remaining video memory value is set to 0.3, the weight of the queue idle value is set to 0.2, and the weight of the batch idle value is set to 0.1; in a throughput-priority scenario, the weight of the queue idle value is set to 0.4, the weight of the idle computing power value is set to 0.3, the weight of the remaining video memory value is set to 0.2, and the weight of the batch idle value is set to 0.1.

[0040] The overall GPU score and cache hit rate of each candidate node are weighted and summed to obtain a comprehensive score. The respective weight coefficients are pre-set by those skilled in the art based on the relative importance of cache reuse efficiency and GPU load balancing. For example, in a cache reuse efficiency priority scenario, the cache hit rate weight is set to 0.7 and the overall GPU score weight is set to 0.3; in a GPU load balancing priority scenario, the cache hit rate weight is set to 0.3 and the overall GPU score weight is set to 0.7; in a balanced scenario, both weights are set to 0.5. The higher the comprehensive score, the better the corresponding candidate node performs in terms of KV-Cache prefix cache reuse efficiency and GPU resource availability, and the more suitable it is as the target backend inference node for the semantic fragment. Therefore, the candidate node with the highest comprehensive score is determined as the target backend inference node for the corresponding semantic fragment.

[0041] When the set of matching nodes is empty, obtain the GPU composite value of each candidate backend inference node in the set of non-matching nodes, and determine the candidate backend inference node with the highest GPU composite value as the target backend inference node.

[0042] Obtain the pre-defined protocol capability profiles of each target backend inference node. The protocol capability profiles are pre-configured and stored in the gateway's local profile library, and include, but are not limited to, the following dimensions: Protocol version compatibility constraint: The first protocol version number supported by the target backend inference node (e.g., v1.0, v1.1). Context window length constraint: the maximum number of input tokens supported by the target backend inference node; Multimodal media type support constraints: A list of multimodal media types supported by the target backend inference node (such as image / jpeg, image / png, application / pdf); Structured output Schema version constraint: The structured output JSON Schema version supported by the target backend inference node; Tool call concurrency constraint: The maximum number of concurrent tool calls supported by the target backend inference node.

[0043] The protocol capability profiles of each target backend inference node are parsed into multi-dimensional capability constraint vectors. These vectors include protocol version compatibility constraints, context window length constraints, multimodal media type support constraints, structured output schema version constraints, and tool call concurrency constraints. Specifically, each constraint item in the protocol capability profile is extracted into numerical or Boolean values ​​according to a preset dimension order and organized into a fixed-length multi-dimensional capability constraint vector. For example, protocol version compatibility constraints are represented by version level values, context window length constraints are represented by the maximum number of supported tokens, multimodal media type support constraints are represented by a binary bitmap showing whether each media type is supported, structured output schema version constraints are represented by version number values, and tool call concurrency constraints are represented by the maximum concurrency, thus obtaining the multi-dimensional capability constraint vector. Semantic fragments are parsed into multidimensional semantic requirement vectors, which include protocol version requirements, context length requirements, multimodal media type requirements, structured output schema requirements, and tool call concurrency requirements. Specifically, when parsing semantic fragments into multidimensional semantic requirement vectors, requirement information is extracted according to the dimensional order corresponding to the multidimensional capability constraint vectors. For example: The protocol version requirement is expressed as a version level value, based on whether the semantic fragment uses specific version features of the first protocol format (such as whether it contains request fields or parameters specific to that version). For example, if the semantic fragment uses structured output parameters specific to version v1.1, the protocol version requirement is extracted as the value 1.1. The context length requirement is based on the actual number of tokens after the semantic fragment is segmented, and this number of tokens is used as the value of the context length requirement; for example, if the semantic fragment contains 2048 tokens, the context length requirement is extracted as the value 2048. Multimodal media type requirements identify the media content blocks (such as image blocks and document blocks) contained in the semantic fragment, count its media type set, and represent it with a binary bitmap indicating whether each media type exists; for example, if the semantic fragment contains two media types, image / png and application / pdf, then the multimodal media type requirements are extracted as the corresponding binary bitmap. The structured output schema requirement checks whether the semantic fragment contains structured output tags (such as JSON Schema constraints or specific format requirements). If so, the JSON Schema version number declared or depended on by the semantic fragment is extracted as the requirement value. For example, if the semantic fragment requires output in JSON Schema Draft-07 format, the structured output schema requirement is extracted as the version number 7. The concurrent tool call requirement is then parsed to represent the number of tool call descriptors contained in the semantic fragment (a tool call descriptor is an information structure that describes the metadata required for a tool call, including the tool name, parameter structure, parameter type, and tool description text, etc.), expressed as the maximum number of tool call descriptors required; for example, if the semantic fragment requests to call 3 different external tools at the same time, the concurrent tool call requirement is extracted as the value 3.

[0044] The multidimensional capability constraint vector and the multidimensional semantic requirement vector are matched to ensure constraint satisfaction, resulting in a candidate protocol adaptation strategy set. The specific rules for constraint satisfaction matching are as follows: Each requirement item in the multidimensional semantic requirement vector is checked to see if it is covered by the corresponding constraint item in the multidimensional capability constraint vector. If all are covered, a candidate protocol adaptation strategy set is generated. The method for generating the candidate protocol adaptation strategy set is as follows: Based on the typical capability combinations of each backend inference node (such as protocol version, context window, multimodal support type, etc.), protocol adaptation strategies corresponding to each dimension are pre-configured and stored in the gateway's local policy library, and a candidate protocol adaptation strategy library is constructed. This library stores protocol adaptation strategies corresponding to the constraint combinations of each dimension. When full coverage is determined, the corresponding candidate protocol adaptation strategy is selected from the candidate protocol adaptation strategy library based on the specific values ​​of each requirement item in the multidimensional semantic requirement vector, thus forming the candidate protocol adaptation strategy set. Based on the contextual reference relationships of corresponding semantic segments in the semantic dependency graph, the context sensitivity weight and downstream reference influence factor of the semantic segments are determined. The context sensitivity weight is determined by the number of times the semantic segment is referenced by other semantic segments. The more references, the higher the context sensitivity weight. For example, the number of times the semantic segment is referenced by other semantic segments in the semantic dependency graph is counted, and the number of references is normalized to the [0,1] interval and used as the context sensitivity weight. The downstream reference influence factor is determined by the length of the downstream reference chain of the semantic segment. The longer the downstream chain, the larger the downstream reference influence factor. For example, the length of the longest downstream reference chain that can be reached from the semantic segment along the reference direction (i.e., the number of hops of the farthest downstream semantic segment) is counted, and the length of the downstream reference chain is normalized to the [0,1] interval and used as the downstream reference influence factor. The candidate protocol adaptation strategy set is prioritized based on context sensitivity weight and downstream citation impact factor, and the candidate protocol adaptation strategy with the highest ranking is determined as the protocol adaptation strategy for the corresponding target backend inference node. Specifically, a context retention score and a downstream compatibility score are preset for each candidate protocol adaptation strategy in the candidate protocol adaptation strategy set. The preset context retention score and downstream compatibility score can be pre-set by those skilled in the art based on the completeness of the retention of semantic fragment context information and the degree of impact on the protocol compatibility of downstream semantic fragments by each candidate protocol adaptation strategy. The comprehensive priority score of each candidate protocol adaptation strategy is calculated. The comprehensive priority score is equal to the product of the context sensitivity weight and the context retention score, plus the product of the downstream citation impact factor and the downstream compatibility score. The candidate protocol adaptation strategies are sorted in descending order according to their comprehensive priority scores, and the candidate protocol adaptation strategy ranked first is determined as the protocol adaptation strategy for the corresponding target backend inference node.

[0045] This approach combines cache hit rate and GPU status to calculate a comprehensive score, thereby selecting the appropriate target backend inference node. This approach balances cache reuse efficiency and computational resource load balancing in routing decisions, avoiding extreme situations such as simply pursuing cache hits leading to GPU overload or simply pursuing GPU idleness while ignoring cache benefits. At the same time, the differentiated strategy selection driven by protocol capability profiles allows different semantic fragments in the same inference request to be tailored to the actual protocol capabilities of the target backend inference node. This avoids request failures caused by backend capability mismatch under a unified conversion strategy, significantly improving the gateway's adaptability to heterogeneous backend clusters.

[0046] Based on the protocol adaptation strategy, each semantic fragment is converted from the second protocol format to the first protocol format and then forwarded to the corresponding target backend inference node; wherein the first protocol format is a protocol format compatible with the first AI service protocol; after the conversion is completed, the converted semantic fragment is forwarded as a sub-request to the corresponding target backend inference node.

[0047] It receives sub-responses in the first protocol format returned by each target backend inference node, parses the incremental content of each sub-response in real time based on the semantic dependency graph to identify semantic boundary events, and dynamically adjusts the streaming semantic conversion strategy of the sub-responses from the first protocol format to the second protocol format according to the semantic boundary events.

[0048] The method for identifying semantic boundary events is as follows: based on the contextual reference relationship of the currently processed semantic segment, determine the expected semantic type and expected boundary features; Based on the expected semantic type and expected boundary features, the incremental content of the sub-response is synchronously parsed using dual-protocol syntax, and the first protocol abstract syntax tree and the second protocol prediction syntax tree carrying the boundary prediction confidence are incrementally constructed. The incremental nodes of the first protocol abstract syntax tree are matched with the expected semantic type. When the match is successful and the confidence of the boundary prediction exceeds the preset confidence threshold, a candidate semantic boundary is generated. The incremental node is a new syntax tree node in the first protocol abstract syntax tree that is generated due to the received incremental content. Calculate the semantic embedding vector similarity between the first protocol abstract syntax tree and the second protocol predicted syntax tree at the candidate semantic boundary, and verify the context reference relationship constraints based on the semantic dependency graph; When the semantic embedding vector similarity is greater than the preset semantic similarity threshold and the relation constraint is met, the candidate semantic boundary is confirmed as a semantic boundary event, and the event type is determined according to the expected boundary features. The event types include tool call parameter assembly completion event, structured JSON field closure event, and multimodal content block end event.

[0049] Specifically, the system receives sub-responses in the first protocol format returned by each target backend inference node. In streaming response mode, it determines the expected semantic type and expected boundary features based on the contextual reference relationship of the currently processed semantic fragment in the semantic dependency graph. The expected semantic type is determined based on the content type identifier and message role identifier of the semantic fragment, such as: text generation type, tool call parameter type, structured JSON type, multimodal content type, etc. The expected boundary features include the end marker pattern corresponding to the expected semantic type, such as: JSON closing brackets for tool call parameters, field end markers for structured JSON, media end markers for multimodal content, etc.

[0050] Based on the expected semantic type and expected boundary features, the incremental content of the sub-response is simultaneously parsed using dual-protocol syntax parsing. This incrementally constructs a first protocol abstract syntax tree and a second protocol predictive syntax tree carrying boundary prediction confidence. The first protocol abstract syntax tree parses and constructs the incremental content according to the syntax rules of the first protocol format. The second protocol predictive syntax tree predictively constructs the content according to the syntax rules of the second protocol format and the expected semantic type. Each node in the second protocol predictive syntax tree carries a boundary prediction confidence (representing the probability of whether the syntax tree node represents a semantic boundary, obtained from historical training data). Both the first and second protocol abstract syntax trees are constructed using incremental syntax parsing techniques known in the field. For example, the first protocol abstract syntax tree uses a bottom-up incremental parsing approach, parsing the incremental content token by token according to the syntax rules of the first protocol format and organizing the parsing results into a tree-like node structure. The second protocol predictive syntax tree uses a top-down predictive parsing approach, predicting the structural boundary of the incremental content under the second protocol based on the preset syntax template and expected semantic type within the second protocol format, and assigning a boundary prediction confidence to each syntax tree node. The specific process is not described in detail here.

[0051] The incremental nodes of the first protocol abstract syntax tree are matched with the expected semantic type. The incremental node is a new syntax tree node in the first protocol abstract syntax tree that is newly generated due to the received incremental content. When the incremental node is successfully matched with the expected semantic type (that is, the semantic type label carried by the incremental node after syntax parsing is consistent with the expected semantic type), and the boundary prediction confidence of the corresponding syntax tree node in the second protocol prediction syntax tree exceeds the preset confidence threshold (obtained according to the statistical distribution characteristics of the confidence of historical correct semantic boundaries, such as taking the lower quartile of the confidence corresponding to the historical correct semantic boundaries as the preset confidence threshold), a candidate semantic boundary is generated.

[0052] The similarity of semantic embedding vectors between the first protocol abstract syntax tree and the second protocol prediction syntax tree at the candidate semantic boundary is calculated. Specifically, the actual semantic payload of the first protocol abstract syntax tree and the predicted semantic payload of the second protocol prediction syntax tree at the candidate semantic boundary are extracted respectively. The actual semantic payload is the text content actually carried at the candidate semantic boundary in the sub-response of the first protocol format returned by the target backend inference node; the predicted semantic payload is the text content that should appear at the corresponding syntax tree node of the second protocol prediction syntax tree according to the grammar rules of the second protocol format and the expected semantic type. Both are encoded into semantic embedding vectors through a pre-trained language model. The pre-trained language model can be... Well-known text encoding models, such as BERT, Sentence-BERT, or RoBERTa models based on the Transformer architecture, are used to map text content into fixed-dimensional semantic embedding vectors. The cosine similarity between the two is calculated as the semantic embedding vector similarity. At the same time, the semantic dependency graph is used to verify whether the candidate semantic boundary satisfies the context reference relationship constraint between the current semantic segment and the downstream semantic segment. When the position of the candidate semantic boundary is located after the end of the context content of the current semantic segment that the downstream semantic segment depends on, and the context content that the downstream semantic segment depends on is not truncated, it is determined that the context reference relationship constraint is satisfied. When the semantic embedding vector similarity is greater than a preset semantic similarity threshold (set according to the experience of those skilled in the art, for example, 0.90) and the contextual reference relationship constraint is met, the candidate semantic boundary is confirmed as a semantic boundary event, and the event type of the semantic boundary event is determined according to the expected boundary features. Specifically, the event type is determined by matching the end marker pattern in the expected boundary features with the preset event type mapping table. When the end marker pattern is the JSON closing bracket of the tool call parameter, the event type is determined to be the tool call parameter assembly completion event; when the end marker pattern is the field end marker of structured JSON, the event type is determined to be the structured JSON field closure event; when the end marker pattern is the media end marker of multimodal content, the event type is determined to be the multimodal content block end event. Here, the preset event type mapping table is pre-configured by those skilled in the art according to the correspondence between the end marker definition of each structural type and the semantic boundary event type in the syntax specification of the second protocol format, and will not be described in detail here.

[0053] Based on the type of semantic boundary events, the streaming semantic conversion strategy of sub-responses from the first protocol format to the second protocol format is dynamically adjusted. The dynamic adjustment method is as follows: the streaming semantic conversion strategy includes text incremental generation mode, tool call parameter structured filling mode, next field parsing mode, and text generation mode; the streaming semantic conversion strategy is switched between modes according to the event type, and a protocol-level control event of the second protocol format corresponding to the event type is inserted at the corresponding boundary position; specifically, when the event type is a tool call parameter assembly completion event, the mode is switched from text incremental generation mode to tool call parameter structured filling mode and a tool call result start event is inserted; when the event type is a structured JSON field closure event, a JSON structure validation flag is inserted and the mode is switched to next field parsing mode; when the event type is a multimodal content block end event, a content block end event is inserted and the mode is switched to text generation mode; specifically: The second protocol format predefines protocol-level control events, including a tool call result start event corresponding to the tool call parameter assembly completion event, a JSON structure validation flag corresponding to the structured JSON field closure event, and a content block end event corresponding to the multimodal content block end event. When an event is detected as a tool call parameter assembly completion event, the streaming semantic transformation strategy is switched from text incremental generation mode to tool call parameter structured filling mode, and a tool call result start event in the second protocol format is inserted. When an event is detected as a structured JSON field closure event, insert a JSON structure validation flag of the second protocol format at the closure position of the structured JSON field closure event, and switch the streaming semantic transformation strategy to the next field parsing mode; When a multimodal content block end event is detected, a content block end event in the second protocol format is inserted, and the streaming semantic conversion strategy is reset to text generation mode.

[0054] By introducing a semantic boundary recognition mechanism based on dual-protocol syntax trees and semantic embedding vector similarity, key semantic segmentation points for cross-protocol conversion can be identified in real time and accurately during streaming response. Compared with traditional fixed-state machine streaming conversion, this invention dynamically adjusts the conversion strategy and inserts protocol-level control events at semantic boundaries, effectively avoiding tool call parameter truncation, JSON structure breakage, or multimodal content block parsing errors caused by fixed rules, significantly improving the protocol compatibility of streaming response and the client parsing success rate.

[0055] Incremental recombination is performed on each dynamically adjusted sub-response to obtain an inference response that conforms to the second protocol format.

[0056] The method for obtaining a reasoning response conforming to the second protocol format is as follows: determine the reorganization order and splicing position of each sub-response based on the contextual reference relationship of each semantic segment; Each sub-response is deduplicated to obtain the deduplicated sub-responses. The deduplicated sub-responses are then concatenated according to the reorganization order and splicing position to obtain the complete response content. Obtain the reference weight of each semantic fragment in the semantic dependency graph, use statistical fields to weight the token of the complete response content, and map the completion reason field according to the overall semantics of the complete response content; The complete response content, the weighted token, and the mapped completion reason field are encapsulated into an inference response conforming to the second protocol format.

[0057] Specifically, the reorganization order and concatenation position of each sub-response are determined based on the contextual reference relationships of each semantic segment in the semantic dependency graph. The reorganization order follows the semantic dependency graph, such as performing a topological sort on the semantic dependency graph, taking the semantic segment with an in-degree of zero (i.e., a semantic segment without a preceding dependency) as the sorting starting point, and placing the sub-response corresponding to the referenced semantic segment before the sub-response corresponding to the referencing semantic segment, thus obtaining the reorganization order. The concatenation position is determined based on the reference direction identifier in the contextual reference relationship, such as determining the relative position of each sub-response in the inference response based on the reference direction identifier in the contextual reference relationship. When the reference direction identifier is a one-way reference, the referenced sub-response is concatenated before the referencing sub-response; when the reference direction identifier is a two-way reference, the semantic segments are concatenated sequentially according to their position order in the original message sequence.

[0058] Each sub-response undergoes deduplication verification. Specifically, it checks whether semantically overlapping segments exist in the sub-responses returned by different target backend inference nodes. The detection of semantically overlapping segments is based on text similarity calculation. For example, it extracts text content segments from each sub-response and calculates the text similarity between any two text content segments. The text similarity is calculated using edit distance. When the edit distance between two text content segments is less than a preset edit distance threshold (pre-set based on experience), it is determined that there are semantically overlapping segments between the corresponding two sub-responses. When semantically overlapping segments are detected, deduplication is performed based on the semantic integrity score of the semantically overlapping segments. That is, the semantic segment with the highest semantic integrity score is retained and the remaining redundant semantic segments are removed to obtain the deduplicated sub-responses. The deduplicated sub-responses are then concatenated according to the recombination order and splicing position to obtain the complete response content.

[0059] The reference weights of each semantic segment in the semantic dependency graph are obtained. The reference weights are calculated based on the in-degree (number of times a segment is referenced) and out-degree (number of times it references other semantic segments) of the semantic segment in the semantic dependency graph. Specifically, the in-degree is divided by the sum of the in-degree and out-degree to obtain the reference weight of the corresponding semantic segment. Based on the reference weights of each semantic segment, the tokens of the complete response content are weighted using statistical fields. Specifically, the statistical fields for tokens include the number of input tokens and the number of output tokens. For the number of input tokens Yi allocated to the i-th semantic segment, Yi is calculated as follows: the ratio of the reference weight of the i-th semantic segment to the sum of the reference weights of all semantic segments is recorded as the allocation weight of the i-th semantic segment; the total number of input tokens is multiplied by the allocation weight of the i-th semantic segment to obtain Yi. For the number of output tokens Zi allocated to the i-th semantic segment, Zi is calculated as follows: the total number of output tokens is multiplied by the allocation weight of the i-th semantic segment to obtain Zi.

[0060] The completion reason field is mapped according to the overall semantics of the complete response content; specifically, the termination semantic features of the complete response content (such as whether it is terminated due to natural stopping, length limit, or tool call) are analyzed, and the completion reason field of the first protocol format is mapped to the corresponding completion reason field of the second protocol format.

[0061] Finally, the complete response content, the weighted token, and the mapped completion reason field are encapsulated into an inference response conforming to the second protocol format.

[0062] By incremental reorganization based on the topological order of the semantic dependency graph and deduplication verification driven by semantic integrity, the problems of redundancy and disordered order of sub-responses in multi-node parallel processing scenarios are effectively solved. At the same time, by using a reference weight-driven token weighting mechanism with statistical fields, the token consumption can reflect the actual semantic contribution of each semantic fragment in the complete response, rather than simply mapping fields, thus providing the caller with a more accurate measurement of resource consumption.

[0063] Furthermore, the first AI service protocol and the second AI service protocol of the present invention are interchangeable; that is, when the gateway receives an inference request in the first protocol format, it can also perform the above-mentioned semantic dependency parsing, cache query, protocol adaptation and incremental reorganization process to realize bidirectional conversion between the first protocol format and the second protocol format.

[0064] In one embodiment, a protocol-independent AI inference gateway bidirectional conversion system is also disclosed, such as Figure 2 As shown, the system includes: The semantic dependency parsing module is used to receive inference requests in the second protocol format of the second AI service protocol, and to perform semantic dependency parsing on the inference requests to generate a semantic dependency graph. The cache status query module is used to obtain the context prefix identifier of each semantic segment in the semantic dependency graph, query the KV-Cache prefix cache status of each candidate backend inference node, determine the matching relationship between each context prefix identifier and each candidate backend inference node, and the cache hit rate corresponding to each matching relationship. The protocol profiling decision module is used to determine the target backend inference node for each semantic segment from the candidate backend inference nodes, and to determine the corresponding protocol adaptation strategy based on the preset protocol capability profile of each target backend inference node. The request protocol conversion module, based on the protocol adaptation strategy, converts each semantic fragment from the second protocol format to the first protocol format and then forwards it to the corresponding target backend inference node; The streaming semantic conversion module is used to receive sub-responses in the first protocol format returned by each target backend inference node, and to parse the incremental content of each sub-response in real time based on the semantic dependency graph to identify semantic boundary events. Based on the semantic boundary events, the streaming semantic conversion strategy of sub-responses from the first protocol format to the second protocol format is dynamically adjusted. The response reorganization output module is used to incrementally reorganize the dynamically adjusted sub-responses to obtain an inference response that conforms to the second protocol format.

[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0066] All formulas in this manual are dimensionless and calculated numerically. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0067] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention.

Claims

1. A protocol-independent bidirectional conversion method for AI inference gateways, characterized in that the method... include: Receive inference requests in the second protocol format of the second AI service protocol, and perform semantic dependency parsing on the inference requests to generate a semantic dependency graph; Obtain the context prefix identifier of each semantic segment in the semantic dependency graph, query the KV-Cache prefix cache status of each candidate backend inference node, determine the matching relationship between each context prefix identifier and each candidate backend inference node, and the cache hit rate corresponding to each matching relationship; The target backend inference node for each semantic segment is determined from the candidate backend inference nodes. The corresponding protocol adaptation strategy is determined based on the preset protocol capability profile of each target backend inference node. The method for determining the protocol adaptation strategy is to parse the protocol capability profile into a multi-dimensional capability constraint vector and parse the semantic segment into a multi-dimensional semantic requirement vector. The multidimensional capability constraint vector and the multidimensional semantic requirement vector are matched to satisfy the constraints to obtain a set of candidate protocol adaptation strategies. Based on the context sensitivity weight and downstream reference influence factor determined by the context reference relationship of the semantic fragment, the candidate protocol adaptation strategy set is prioritized and the candidate protocol adaptation strategy with the highest ranking is determined as the protocol adaptation strategy. Based on the protocol adaptation strategy, each semantic segment is converted from the second protocol format to the first protocol format and then forwarded to the corresponding target backend inference node; wherein the first protocol format is a protocol format compatible with the first AI service protocol; Receive sub-responses in the first protocol format returned by each target backend inference node, parse the incremental content of each sub-response in real time based on the semantic dependency graph to identify event types and semantic boundary events, and dynamically adjust the streaming semantic conversion strategy of sub-responses from the first protocol format to the second protocol format according to the event types and semantic boundary events. The dynamic adjustment method is as follows: the streaming semantic transformation strategy includes incremental text generation mode, tool call parameter structured filling mode, next field parsing mode and text generation mode, and the event types include tool call parameter assembly completion event, structured JSON field closure event and multimodal content block end event; the streaming semantic transformation strategy is switched between modes according to the event type, and a protocol-level control event of the second protocol format corresponding to the event type is inserted at the corresponding boundary position; Incremental recombination is performed on each dynamically adjusted sub-response to obtain an inference response that conforms to the second protocol format.

2. The method according to claim 1, characterized in that, The method for generating a semantic dependency graph is as follows: Extract the message sequence from the inference request, as well as the message role identifier and content type identifier of each message in the message sequence; Based on message role identifiers and content type identifiers, the message sequence is split into multiple initial semantic segments, and the semantic integrity of each initial semantic segment is verified to obtain multiple semantic segments; Obtain the position and role identifier of each message in the message sequence, and establish the contextual reference relationship between each semantic fragment; Obtain the pre-set length character sequence and rolling hash digest of each semantic segment, and generate the corresponding context prefix identifier; Construct a semantic dependency graph using semantic fragments as nodes, context references as edges, and context prefixes as node attributes.

3. The method according to claim 2, characterized in that, The method for determining the matching relationship between each context prefix identifier and each candidate backend inference node is as follows: Iterate through the KV-Cache prefix cache status of each candidate backend inference node to obtain the cached context prefix identifier of each candidate backend inference node; The context prefix identifier of each semantic segment is matched with the context prefix identifier that has been cached by each candidate backend inference node. If the match is successful, it is determined that a matching relationship exists.

4. The method according to claim 3, characterized in that, The method for determining the cache hit rate corresponding to each matching relationship is as follows: For each candidate backend inference node with a matching relationship, query the historical request records of each candidate backend inference node for the cached context prefix identifier corresponding to the matching relationship within a preset time window, and use the ratio of the number of hits in the historical request records to the total number of requests as the cache hit rate of each candidate backend inference node.

5. The method according to claim 4, characterized in that, The method for determining the target backend inference node for each semantic segment is as follows: Candidate backend inference nodes are divided into a set of matching nodes and a set of non-matching nodes. When the set of matching nodes is not empty, each candidate backend inference node in the set of matching nodes is sorted in descending order according to the cache hit rate, and the top k candidate backend inference nodes are selected as candidate preferred nodes. Obtain the GPU status information of each candidate node and calculate the GPU comprehensive value. Add the GPU comprehensive value of each candidate node to the cache hit rate with weighted summation to obtain the comprehensive score. Select the candidate node with the highest comprehensive score as the corresponding target backend inference node. When the set of matching nodes is empty, obtain the GPU composite value of each candidate backend inference node in the set of non-matching nodes, and determine the candidate backend inference node with the highest GPU composite value as the target backend inference node.

6. The method according to claim 5, characterized in that, The method for identifying semantic boundary events is as follows: Based on the contextual reference relationships of the currently processed semantic fragments, determine the expected semantic type and expected boundary features; Based on the expected semantic type and expected boundary features, the incremental content of the sub-response is synchronously parsed using dual-protocol syntax, and the first protocol abstract syntax tree and the second protocol prediction syntax tree carrying the boundary prediction confidence are incrementally constructed. The incremental nodes of the first protocol abstract syntax tree are matched with the expected semantic type. When the match is successful and the confidence of the boundary prediction exceeds the preset confidence threshold, a candidate semantic boundary is generated. The incremental node is a new syntax tree node in the first protocol abstract syntax tree that is generated due to the received incremental content. Calculate the semantic embedding vector similarity between the first protocol abstract syntax tree and the second protocol predicted syntax tree at the candidate semantic boundary, and verify the context reference relationship constraints based on the semantic dependency graph; When the semantic embedding vector similarity is greater than the preset semantic similarity threshold and the relation constraints are met, the candidate semantic boundary is confirmed as a semantic boundary event, and the event type is determined according to the expected boundary features.

7. The method according to claim 6, characterized in that, The method for obtaining an inference response conforming to the second protocol format is as follows: Based on the contextual reference relationships of each semantic segment, determine the reorganization order and splicing position of each sub-response; Each sub-response is deduplicated to obtain the deduplicated sub-responses; the deduplicated sub-responses are then concatenated according to the reorganization order and splicing position to obtain the complete response content. Obtain the reference weight of each semantic fragment in the semantic dependency graph, use statistical fields to weight the token of the complete response content, and map the completion reason field according to the overall semantics of the complete response content; The complete response content, the weighted token, and the mapped completion reason field are encapsulated into an inference response conforming to the second protocol format.

8. A protocol-independent AI inference gateway bidirectional conversion system, implementing the method according to any one of claims 1-7, characterized in that, The system includes: The semantic dependency parsing module is used to receive inference requests in the second protocol format of the second AI service protocol, and to perform semantic dependency parsing on the inference requests to generate a semantic dependency graph. The cache status query module is used to obtain the context prefix identifier of each semantic segment in the semantic dependency graph, query the KV-Cache prefix cache status of each candidate backend inference node, determine the matching relationship between each context prefix identifier and each candidate backend inference node, and the cache hit rate corresponding to each matching relationship. The protocol profiling decision module is used to determine the target backend inference node for each semantic segment from the candidate backend inference nodes, and to determine the corresponding protocol adaptation strategy based on the preset protocol capability profile of each target backend inference node. The request protocol conversion module, based on the protocol adaptation strategy, converts each semantic fragment from the second protocol format to the first protocol format and then forwards it to the corresponding target backend inference node; The streaming semantic conversion module is used to receive sub-responses in the first protocol format returned by each target backend inference node, and to parse the incremental content of each sub-response in real time based on the semantic dependency graph to identify semantic boundary events. Based on the semantic boundary events, the streaming semantic conversion strategy of sub-responses from the first protocol format to the second protocol format is dynamically adjusted. The response reorganization output module is used to incrementally reorganize the dynamically adjusted sub-responses to obtain an inference response that conforms to the second protocol format.

Images