Method and system for binding reasoning service instance

Abstract

The embodiment of the invention relates to an inference service instance binding method and system, and the method comprises the steps: obtaining instance identification information of at least one candidate inference service instance through an inference client in an inference session establishment process; under the condition that the historical instance identification information is locally stored in the reasoning client, matching the locally stored historical instance identification information with the instance identification information of at least one candidate reasoning service instance; if the matching is successful, determining the successfully matched candidate inference service instance as a target inference service instance, and establishing or maintaining session connection with the target service instance to carry out an inference dialogue; wherein the historical instance identification information is instance identification information of inference service instances connected to the inference client in the last round of inference session; according to the method, when the inference session is newly built, the client can be directly connected to the original instance in which the historical cache is stored, computing resources are reused, meanwhile, a centralized component does not need to be introduced, and the performance bottleneck and the single-point fault risk are avoided.

Description

Technical Field

[0001] This invention relates to the field of computer network technology, and in particular to a method and system for binding inference service instances. Background Technology

[0002] With the development and application of artificial intelligence technology, AI inference services have become a key support for various online applications. In many real-world scenarios, user tasks often require multiple, continuous inference interactions, such as multi-turn dialogues, document continuation, or code generation. During each inference process, the server needs to consume significant computing resources to generate and save intermediate calculation results, typically referred to as a key-value cache (KVCache). This cache can be directly reused in subsequent inferences as input for the next calculation, thereby significantly reducing response latency and conserving computing resources.

[0003] In a distributed inference service architecture, multiple service instances are typically deployed, and users access them through a unified domain name. The Domain Name System (DNS) dynamically schedules user requests to different service instances for processing based on factors such as real-time network conditions, instance load, and user geographic location. While this scheduling mechanism improves service availability and efficiency, it also means that multiple requests from the same user session may be assigned to different instances. If a user initiates a new request due to actions such as closing the browser or switching networks, DNS may redirect them to a new service instance. This prevents the KVCache previously stored in the original instance from being reused, requiring all intermediate results to be recalculated and resulting in a significant waste of computing resources.

[0004] Several technical solutions exist to address the above problems, but all have significant limitations: One approach is to synchronously replicate the KVCache generated by the user to all service instances in the system, thereby ensuring that subsequent user requests can directly reuse the cache regardless of which instance they are scheduled to. However, since the KVCache capacity of a single session can reach hundreds of MB to tens of GB, this approach will result in extremely high storage costs and network transmission overhead in actual deployments, making it difficult to support large-scale concurrent scenarios and lacking scalability.

[0005] Another approach is to introduce a centralized proxy that records the mapping between each session and its corresponding service instance. When a user re-initiates a request, the proxy can query the history and route it to the original instance. While this method avoids full copying of cached data, it easily creates a system performance bottleneck because all traffic must pass through a single proxy node. Furthermore, the proxy itself becomes a critical single point of failure; its failure will render the entire service unavailable.

[0006] Therefore, there is an urgent need for a new solution that can stably bind inference sessions to specific service instances while avoiding performance bottlenecks and single points of failure. Summary of the Invention

[0007] Embodiments of the present invention provide a method and system for binding inference service instances to solve the technical problem of how to achieve stable binding of inference sessions with specific service instances while avoiding the introduction of performance bottlenecks and single points of failure.

[0008] In a first aspect, embodiments of the present invention provide a method for binding inference service instances, applied to an inference client. The method includes: during the establishment of an inference session, obtaining instance identification information of at least one candidate inference service instance; if the inference client has historical instance identification information stored locally, matching the locally stored historical instance identification information with the instance identification information of the at least one candidate inference service instance; if the matching is successful, determining the successfully matched candidate inference service instance as the target inference service instance, and establishing or maintaining a session connection with the target service instance to conduct inference dialogue; wherein, the historical instance identification information is the instance identification information of the inference service instance connected by the inference client in the previous round of inference session.

[0009] In some embodiments, the method further includes: if the historical instance identification information is not stored locally on the inference client or the matching fails, selecting one from the at least one candidate inference service instance as the target inference service instance.

[0010] In some embodiments, the instance identification information is a unique identifier separate from the network access address; then, obtaining the instance identification information of at least one candidate inference service instance during the inference session establishment process includes: sending an inference service domain name resolution request to the domain name resolution system to receive a candidate list returned by the domain name resolution system, the candidate list including the network access addresses of at least one candidate inference service instance; if the inference client has historical instance identification information stored locally, matching the locally stored historical instance identification information with the instance identification information of the at least one candidate inference service instance; if the match is successful, determining the successfully matched candidate inference service instance as the target inference service instance, and establishing or maintaining a session connection with the target service instance to conduct inference dialogue, including: confirming... The process involves: identifying the current candidate inference service instance from the candidate list; establishing a session connection with the current candidate inference service instance and receiving a unique identifier pushed by the current candidate inference service instance; if the inference client has historical unique identifiers stored locally, matching the locally stored historical unique identifiers with the unique identifier of the current candidate inference service instance; if the match is successful, determining the current candidate inference service instance as the target inference service instance and maintaining a session connection with the target service instance to conduct inference dialogue; if the match fails, disconnecting the session connection with the current candidate inference service instance, selecting the next candidate inference service instance from the candidate list as the current candidate inference service instance, and repeating the step of establishing a session connection with the current candidate inference service instance.

[0011] In some embodiments, the method further includes: if the inference client does not store historical unique identifiers locally, or if the locally stored historical unique identifiers fail to match each unique identifier in the candidate list, determining the first candidate inference service instance in the candidate list as the target inference service instance.

[0012] In some embodiments, where the instance identification information is a network access address, obtaining the instance identification information of at least one candidate inference service instance during the inference session establishment process includes: sending an inference service domain name resolution request to a domain name resolution system to receive a candidate list returned by the domain name resolution system, the candidate list including the network access addresses of at least one candidate inference service instance; if the inference client locally stores historical instance identification information, matching the locally stored historical instance identification information with the instance identification information of the at least one candidate inference service instance; if the match is successful, determining the successfully matched candidate inference service instance as the target inference service instance, and establishing or maintaining a session connection with the target service instance to conduct inference dialogue, includes: if the inference client locally stores historical network access addresses, matching the locally stored historical network access addresses with the network access addresses in the candidate list; if the match is successful, determining the successfully matched candidate inference service instance as the target inference service instance, and establishing a session connection with the target inference service instance to conduct inference dialogue.

[0013] In some embodiments, the method further includes: if the inference client does not store historical network access addresses locally, or if the locally stored historical network access addresses fail to match each network access address in the candidate list, then the first candidate inference service instance in the candidate list is selected as the target inference service instance.

[0014] Secondly, the present invention provides a method for binding inference service instances, applied to inference service instances. The method includes: when the inference service instance is a candidate inference service instance, during the inference session establishment process, sending instance identification information to the inference client; wherein, the instance identification information of the candidate inference service instance is used by the inference client to match the locally stored historical instance identification information with the instance identification information of the candidate inference service instance if the client has historical instance identification information stored locally; if the match is successful, the successfully matched candidate inference service instance is determined as the target inference service instance, and a session connection is established or maintained with the target service instance to conduct inference dialogue, wherein the historical instance identification information is the instance identification information of the inference service instance connected by the inference client in the previous round of inference session.

[0015] In some embodiments, the method further includes: pre-registering the network access address of the inference service instance with a domain name resolution system, so that the domain name resolution system returns a candidate list to the inference client after receiving a domain name resolution request sent by the inference client, the candidate list including the network access address of at least one candidate inference service instance.

[0016] Thirdly, the present invention provides a system for binding inference service instances, comprising: an inference client and at least one inference service instance; the inference client is used to execute the method for binding inference service instances as described in any of the first aspects, and the inference service instance is used to execute the method for binding inference service instances as described in the second aspect.

[0017] In some embodiments, the system further includes a domain name resolution system; the domain name resolution system is configured to return a candidate list to the inference client in response to an inference service domain name resolution request sent by the inference client, the candidate list including the network access address of at least one candidate inference service instance.

[0018] This invention provides a method and system for binding inference service instances. The inference client locally stores historical instance identifiers and actively matches these identifiers with candidate instance identifiers when a new session is established, reconnecting to the corresponding inference service instance that stores the user's historical session context cache. This method ensures that user inference requests across multiple sessions consistently point to the same service instance, thereby directly reusing the generated large-scale KVCache. This avoids redundant calculations caused by instance switching, significantly saving computing resources and reducing request latency. Furthermore, this binding mechanism does not rely on any centralized proxy or centralized state synchronization component, structurally eliminating single-point-of-failure risks and traffic bottlenecks, achieving highly available, easily scalable, and low-maintenance session persistence capabilities. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A typical inference session scheduling diagram provided for existing technologies; Figure 2 A schematic diagram of inference session scheduling based on a centralized agent is provided for existing technologies; Figure 3 A flowchart illustrating a method for binding an inference service instance to an inference client, provided in an embodiment of the present invention; Figure 4 A flowchart illustrating another method for binding a reasoning service instance to a reasoning client, provided in an embodiment of the present invention; Figure 5 A flowchart illustrating another method for binding an inference service instance to an inference client, provided in an embodiment of the present invention; Figure 6 A flowchart illustrating a method for binding a reasoning service instance to a reasoning service instance, provided in an embodiment of the present invention; Figure 7 A flowchart illustrating a method for binding inference service instances provided in an embodiment of the present invention; Figure 8 A flowchart illustrating another method for binding inference service instances provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of a binding inference service instance system provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Figure 1 A typical inference session scheduling diagram provided for existing technologies, such as Figure 1 As shown, the inference client initiates a domain name resolution request for the inference service to the DNS. Each inference service instance (instance 1 to instance N in the figure) is pre-deployed in different cloud architectures (such as cloud A, cloud B, and cloud C) and its network access address (usually an IP address) is registered with the DNS. When the DNS receives the client's resolution request, it returns at least one available inference service instance IP address from the registered instances. It is important to note that this DNS scheduling process is stateless and memoryless; each resolution request may return a different instance address based on real-time policies (such as load balancing and geographic location). Subsequently, the inference client uses the obtained IP address to establish a connection with the corresponding inference service instance (e.g., in this case, connecting to instance 2 in cloud B) and begins the inference session.

[0024] However, this approach has a significant drawback: when a user closes their browser or their network location changes and they re-initiate a request, the client will re-resolve the domain name with the DNS. Due to the memoryless nature of DNS, the user may be reassigned to a different instance than before (for example, in this case, to instance 1 in cloud A). Since this new instance does not store the user's historical session KVCache, all computations must be re-executed to generate the KVCache, resulting in a significant waste of computing resources.

[0025] To address the aforementioned technical issues, existing technologies offer two solutions. The first solution is a full KVCache replication solution. This solution attempts to synchronously replicate the KVCache for a user session to all other inference service instances in the system immediately after a service instance generates it. Its goal is to ensure that subsequent user requests, regardless of which instance they are scheduled to, can directly access the required KVCache.

[0026] However, this approach inherently incurs enormous resource overhead. Due to the massive volume of KVCache data, continuous data synchronization across all instances in scenarios with high concurrency will result in unacceptable network bandwidth consumption and cause the storage cost per instance to increase exponentially. This approach sacrifices computing resource savings for extremely high storage and network transmission costs, lacks scalability, and cannot be implemented in real-world large-scale production environments.

[0027] The second approach is a centralized proxy record scheme. For example... Figure 2 This paper presents a schematic diagram of a centralized proxy-based inference session scheduling architecture. This architecture introduces a globally centralized inference service proxy between the inference client and the distributed inference service instances (instance 1 to instance N). Each inference service instance registers its network access address with the DNS through this proxy. When the inference client initiates its first inference service domain name resolution request, the DNS returns at least one available inference service instance IP address. The inference client establishes a session with one of its selected instances (e.g., instance 1 in cloud A) through this proxy, and the proxy records the mapping relationship between the session ID and the selected instance address. When the user initiates a request again, the request first reaches the proxy; the proxy, by querying its maintained mapping table, directs the request to the same instance that previously processed the session, thus theoretically enabling the reuse of the KVCache.

[0028] While this approach avoids fully replicating KVCache data between instances and reduces network transmission pressure, its architecture has inherent technical flaws: all user request and response traffic must flow through a centralized proxy for forwarding and routing decisions, making it highly susceptible to becoming a performance bottleneck for the overall system throughput and limiting service scalability; at the same time, the proxy server constitutes a critical single point of failure, and once it fails, it will cause all user services to be interrupted, even though all backend inference service instances may be in a healthy state.

[0029] To address the aforementioned technical problems, the technical concept of this invention is as follows: Session binding is achieved through a negotiation mechanism between the inference client and the inference service instance: The inference client stores the identifiers of instances connected to historical sessions, and when creating a new session, it actively compares the identifier with the identifiers of candidate inference service instances to locate the original instance containing historical cache, thereby achieving the reuse of computing resources.

[0030] Figure 3 This is a flowchart illustrating a method for binding an inference service instance to an inference client, as provided in an embodiment of the present invention. Figure 3 As shown, it includes the following steps: Step S301: During the inference session establishment process, obtain the instance identification information of at least one candidate inference service instance.

[0031] Step S302: If the inference client has historical instance identification information stored locally, match the locally stored historical instance identification information with the instance identification information of the at least one candidate inference service instance.

[0032] The historical instance identification information refers to the instance identification information of the inference service instance that the inference client connected to in the previous inference session.

[0033] Step S303: If a match is successful, the candidate inference service instance that has been successfully matched is determined as the target inference service instance, and a session connection is established or maintained with the target service instance to conduct inference dialogue.

[0034] Specifically, when a new inference session needs to be established, the inference client first obtains the identifiers of a set of currently available candidate inference service instances. Then, the inference client checks if a historical instance identifier (i.e., the identifier of the service instance actually connected to in the previous inference session) exists locally. If the historical instance identifier exists, the inference client compares it sequentially with the identifiers of each currently obtained candidate inference service instance. Once a comparison is successful, the inference client identifies the corresponding candidate inference service instance as the target inference service instance for this round of the session and establishes or maintains a session connection with it to continue the inference task. In this way, the inference client can relocate the original inference service instance storing the KVCache, thereby directly reusing the already generated KVCache, effectively saving computing resources and improving response efficiency.

[0035] In some embodiments, the method further includes: if the historical instance identification information is not stored locally on the inference client or the matching fails, selecting one from the at least one candidate inference service instance as the target inference service instance.

[0036] Specifically, when the inference client does not have a local historical instance identifier (e.g., during a user's first visit), or when matching reveals that the previously connected inference service instance is not among the current candidate inference service instances (possibly due to instance offline status or changes in network scheduling policies), the inference client will select an instance from the currently acquired candidate inference service instances according to preset rules as the target service inference instance. Subsequently, the inference client saves the identifier of this instance as the basis for the next matching. This mechanism ensures that sessions can always be established and also updates the historical identifier base for subsequent session bindings.

[0037] The binding inference service instance method provided in this invention involves the inference client storing historical instance identifiers locally and actively matching these identifiers with the identifiers of candidate instances when a new session is established, reconnecting to the corresponding inference service instance that stores the KVCache. This method ensures that inference requests from users across multiple sessions can continuously point to the same service instance, thereby directly reusing the already generated large-scale KVCache, avoiding redundant calculations caused by instance switching, significantly saving computing resources, and reducing request latency. Furthermore, this binding mechanism does not rely on any centralized proxy or centralized state synchronization component, structurally eliminating the risk of single point of failure and traffic bottlenecks, achieving highly available, easily scalable, and low-maintenance session persistence capabilities.

[0038] Based on the foregoing embodiments, Figure 4 This is a flowchart illustrating another method for binding an inference service instance to an inference client, provided by an embodiment of the present invention. The instance identification information is a unique identifier separate from the network access address. For example... Figure 4As shown, it includes the following steps: Step S401: Send a domain name resolution request for inference service to the domain name resolution system to receive a candidate list returned by the domain name resolution system, wherein the candidate list includes the network access address of at least one candidate inference service instance.

[0039] Step S402: Determine the current candidate inference service instance in the candidate list.

[0040] Step S403: Establish a session connection with the current candidate inference service instance and receive the unique identifier pushed by the current candidate inference service instance.

[0041] Step S404: If the inference client has a locally stored historical unique identifier, match the locally stored historical unique identifier with the unique identifier of the current candidate inference service instance.

[0042] Step S405: Determine whether the match is successful.

[0043] If not, proceed to step S406 and return to step S403; if yes, proceed to step S407.

[0044] Step S406: Disconnect the session connection from the current candidate inference service instance and select the next candidate inference service instance in the candidate list as the current candidate inference service instance.

[0045] Step S407: Determine the current candidate reasoning service instance as the target reasoning service instance, and maintain a session connection with the target service instance to conduct reasoning dialogue.

[0046] Specifically, the instance identification information is a unique identifier (ID) separate from the network access address. First, the inference client sends a domain name resolution request for the inference service to the DNS. The DNS filters based on factors such as network distance between service instances and returns a list containing network access addresses (such as IP addresses and ports) of at least one candidate inference service instance. Next, the inference client attempts to connect to each candidate inference service instance in the list. For example, the inference client first establishes a session connection with the candidate inference service instance corresponding to the first network access address in the list. During the connection process, the inference client receives its own ID from that candidate inference service instance. Then, the inference client checks if it has a historical ID stored locally corresponding to the last successful session. If so, it compares the locally stored historical ID with the ID of the newly received candidate inference service instance. If they do not match, it indicates that the currently connected candidate inference service instance is not the instance the user previously used. The inference client then disconnects the current connection and attempts to connect to the next network access address in the candidate list, repeating the connection, ID reception, and comparison process. This loop continues until an instance with a matching ID is found. Once a match is found, the inference client stops searching, identifies the currently connected candidate inference service instance as the target inference service instance for this round of the session, and continues the inference conversation on this connection. Since the instance's ID matches the historical ID, it is the original instance containing the KVCache, thus achieving direct reuse of computing resources.

[0047] In addition, when attempting to connect to candidate inference service instances, the inference client can use two access methods: sequential access or parallel connection initiation. When using the parallel method, the client will simultaneously initiate session connection requests to multiple candidate instances. Subsequently, based on the actual order in which the connections are established, it will sequentially receive the identifiers pushed by each instance and compare them. This parallel method can reduce the total latency caused by sequential connection attempts without affecting the matching accuracy, thereby improving matching efficiency.

[0048] In some embodiments, the method further includes: if the inference client does not store historical unique identifiers locally, or if the locally stored historical unique identifiers fail to match each unique identifier in the candidate list, determining the first candidate inference service instance in the candidate list as the target inference service instance.

[0049] Specifically, if the inference client does not save the historical ID locally (e.g., for the user's first use), or if no match is found after trying all candidate inference service instances (e.g., the original instance has gone offline), the inference client will directly select the first candidate inference service instance in the candidate list returned by DNS as the target inference service instance for this session, and can push the ID of the target inference service instance to the inference client for saving. This mechanism ensures that the session can always be successfully established, maintains service continuity, and also establishes a new basis for instance binding in subsequent sessions.

[0050] Building upon the aforementioned embodiments, instance identification information employs a unique identifier separate from the network access address. This design decouples the instance's network location from its logical identity, ensuring that even if the instance's IP address changes due to operational adjustments, the inference client can still accurately identify and reconnect to the original instance storing the user's historical KVCache through identifier comparison, thereby maintaining the stability and accuracy of the binding relationship. Through a distributed negotiation mechanism where the inference client actively establishes connections with multiple candidate instances and performs identifier matching, the system can achieve accurate session positioning without the intervention of a centralized component. This not only completely avoids single points of failure and performance bottlenecks but also directly guarantees the efficient reuse of large-scale KVCaches, significantly saving computing resources. Furthermore, the built-in circular matching and backup connection mechanism in the solution can maintain normal service operation even when historical instances are unavailable, thus achieving highly reliable, low-cost, and easily scalable session persistence capabilities.

[0051] Based on the foregoing embodiments, Figure 5 This is a flowchart illustrating another method for binding an inference service instance to an inference client, provided by an embodiment of the present invention. The instance identification information is a network access address. Figure 5 As shown, it includes the following steps: Step S501: Send a domain name resolution request for inference service to the domain name resolution system to receive a candidate list returned by the domain name resolution system, wherein the candidate list includes the network access address of at least one candidate inference service instance.

[0052] Step S502: If the inference client has historical network access addresses stored locally, match the locally stored historical network access addresses with the network access addresses in the candidate list.

[0053] Step S503: If a match is successful, the candidate reasoning service instance that has been successfully matched is determined as the target reasoning service instance, and a session connection is established with the target reasoning service instance to carry out reasoning dialogue.

[0054] Specifically, the instance identification information is the network access address (IP). First, the inference client sends a domain name resolution request for the inference service to the DNS. The DNS filters based on factors such as network distance between each inference service instance and returns a list containing the IP addresses of at least one candidate inference service instance. Then, the inference client checks if it has a historical IP address stored locally for the inference service instance used in the last successful session. If the historical IP address exists, it is compared with the IP addresses of each candidate inference service instance in the currently obtained candidate list. If the historical IP address appears in the candidate list, the inference client directly establishes a new session connection with the candidate inference service instance corresponding to that address and performs the current inference over this connection. Since the connection is still to the same instance that previously stored the user's historical KVCache, the existing computation context can be directly reused, avoiding the resource overhead of regenerating the cache.

[0055] In some embodiments, the method further includes: if the inference client does not store historical network access addresses locally, or if the locally stored historical network access addresses fail to match each network access address in the candidate list, then the first candidate inference service instance in the candidate list is selected as the target inference service instance.

[0056] Specifically, if the client does not have a historical IP address locally (e.g., during the user's first access), or if the historical IP address is not in the candidate list returned by the current DNS (possibly due to the original instance being offline or a change in scheduling policy), the client will automatically select the first instance in the candidate list as the target inference service instance and establish a connection with it. Simultaneously, the inference client saves the IP address of this target inference service instance. This mechanism ensures that service requests are responded to normally, maintaining service continuity, and also provides a new address basis for binding subsequent sessions.

[0057] Building upon the aforementioned embodiments, this method achieves efficient and direct binding by directly using network access addresses as instance identifiers. The inference client locally records the address of the instance with which the last successful connection was established. When a new session is established, it simply compares the historical address with the candidate address list returned by the domain name resolution system. Once a match is found, a connection can be established with the target instance. This method is concise, faster in matching, and particularly suitable for environments with stable network topologies. It also effectively directs user sessions to specific instances that store KVCache, avoiding wasted computing resources and increased latency caused by instance switching. Furthermore, the entire binding process is completed autonomously by the client, completely independent of any centralized component, thus ensuring resource reuse while maintaining a simple, reliable, and scalable system architecture.

[0058] Figure 6This is a flowchart illustrating a method for binding a reasoning service instance to a reasoning service instance, provided as an embodiment of the present invention. Figure 6 As shown, it includes the following steps: Step S601: When the inference service instance is a candidate inference service instance, during the inference session establishment process, send instance identification information to the inference client.

[0059] The instance identifier information of the candidate inference service instance is used by the inference client to match the locally stored historical instance identifier information with the instance identifier information of the candidate inference service instance when the client has stored historical instance identifier information locally. If the match is successful, the successfully matched candidate inference service instance is determined as the target inference service instance, and a session connection is established or maintained with the target service instance to carry out inference dialogue. The historical instance identifier information is the instance identifier information of the inference service instance that the inference client connected to in the previous round of inference session.

[0060] Specifically, when an inference service instance participates in session establishment as a candidate inference service instance, it sends its instance identification information to the initiating inference client. Upon receiving this instance identification information, the inference client matches it with locally stored historical instance identification information. If a match is found, the inference client identifies the instance as the target instance and continues the inference dialogue based on an existing connection or a newly established connection. The historical instance identification is the identifier of the inference service instance that the inference client actually connected to in the previous inference session.

[0061] In some embodiments, the method further includes: pre-registering the network access address of the inference service instance with a domain name resolution system, so that the domain name resolution system returns a candidate list to the inference client after receiving a domain name resolution request sent by the inference client, the candidate list including the network access address of at least one candidate inference service instance.

[0062] Specifically, each inference service instance pre-registers its network access address with a DNS server. After registration, when an inference client sends a query request to the DNS for the inference service domain name, the system can return a candidate list containing the instance's address. This list provides the necessary addressing basis for the client's subsequent matching and connection operations, thus supporting the smooth initiation and execution of the entire binding process.

[0063] The binding inference service instance method provided in this invention actively sends its own identification information when establishing a session with the inference client, enabling the inference client to obtain key evidence for matching. This method, combined with the client-side matching logic, accurately redirects the user session to a specific instance containing historical KVCache. While ensuring the reuse of computing resources, it eliminates performance bottlenecks and single-point-of-failure risks from an architectural perspective, improving the reliability and scalability of the entire inference service system.

[0064] To further illustrate the embodiments of the present invention, the following two specific examples will be used to describe the solution in detail.

[0065] Example 1: The instance identification information is a unique ID separate from the IP address, such as... Figure 7 A flowchart illustrating a method for binding inference service instances provided in an embodiment of the present invention is shown below. Figure 7 As shown, it includes the following steps: Step S701: Each inference service instance registers its IP address with the DNS.

[0066] Step S702: The inference client sends a domain name resolution request for the inference service to the DNS for the first time.

[0067] Step S703: DNS returns a first candidate list, which contains the IP addresses of at least one currently available inference service instance.

[0068] Step S704: The inference client selects a candidate inference service instance (e.g., the first one) from the first candidate list and establishes a session connection with it.

[0069] Step S705: The inference service instance actively pushes its own unique ID to the inference client.

[0070] Step S706: The inference client receives and saves this ID as historical instance identification information: historical ID.

[0071] Step S707: The inference client sends a domain name resolution request to the DNS again.

[0072] Step S708: DNS returns a second candidate list containing the IP addresses of at least one currently available inference service instance (this list may differ from the first one).

[0073] Step S709: The inference client begins to establish a session connection with an instance in the second candidate list (e.g., the first instance).

[0074] Step S710: The inference service instance actively pushes its own unique ID to the inference client.

[0075] Step S711: The inference client matches the received ID with the historical IDs stored locally.

[0076] Step S712: If a match is successful, the inference client maintains the session connection with this instance and conducts an inference dialogue on this connection.

[0077] Step S713: If the match fails, the inference client disconnects the current connection and establishes a connection with the next instance in the second candidate list, and repeats the "get ID-match" process from steps S710 to S711.

[0078] Step S714: If all instances in the second candidate list fail to match, the first instance in the second candidate list selected by the inference client is taken as the target inference service instance for this time.

[0079] Step S715: Establish a session connection with the finally determined target inference service instance and conduct inference dialogue; at the same time, save the ID of the new instance to update the local historical instance identification information.

[0080] Example 2: The instance identification information is the IP address, such as... Figure 8 A flowchart illustrating another method for binding inference service instances provided in this embodiment of the invention is shown below. Figure 8 As shown, it includes the following steps: Step S801: Each inference service instance registers its IP address with the DNS.

[0081] Step S802: The inference client sends a domain name resolution request for the inference service to the DNS for the first time.

[0082] Step S803: DNS returns a first candidate list, which contains the IP addresses of at least one currently available inference service instance.

[0083] Step S804: The inference client selects an instance (e.g., the first one) from the first candidate list and establishes a session connection with it.

[0084] Step S805: The inference client saves the IP address of the instance as historical instance identification information: historical IP address.

[0085] Step S806: The inference client sends a domain name resolution request to the DNS again.

[0086] Step S807: DNS returns a second candidate list containing the IP addresses of at least one currently available inference service instance.

[0087] Step S808: The inference client reads the locally saved historical IP addresses and directly compares and matches them with all IP addresses in the second candidate list.

[0088] Step S809: If a match is successful (i.e., the historical IP address exists in the second candidate list), the inference client directly establishes a new session connection with the instance corresponding to that address and begins an inference dialogue.

[0089] Step S810: If the match fails (i.e., the historical IP address is not in the second candidate list), the inference client selects the first instance in the list as the target inference service instance for this time.

[0090] Step S811: Establish a session connection with the finally determined target inference service instance and initiate an inference dialogue. Simultaneously, the inference client saves the IP address of the new instance to update the local historical instance identification information.

[0091] In summary, regardless of whether the instance identifier is an independent logical ID or its network address, it brings significant benefits: First, by ensuring that user sessions can continuously locate the same instance storing their historical KVCache, it enables the direct reuse of massive intermediate computation results, fundamentally avoiding redundant computations caused by instance switching, greatly saving computing resources and reducing request latency. Second, the entire binding process is completely decentralized, relying on distributed negotiation between the client and multiple instances, without any centralized proxy or state synchronization components, thus completely eliminating the risk of single point of failure and performance bottlenecks in the system, enabling the service architecture to have high availability and elastic scalability. Finally, its built-in failover connection mechanism ensures service continuity under any circumstances, while establishing a new foundation for subsequent session binding, achieving stable, efficient, and low-cost intelligent session maintenance.

[0092] Figure 9 This is a schematic diagram of the structure of a bound inference service instance system provided in an embodiment of the present invention, such as... Figure 9 As shown, the system includes: an inference client and at least one inference service instance; the inference client is used to execute the binding inference service instance method as described in any of the first aspects, and the inference service instance is used to execute the binding inference service instance method as described in any of the second aspects.

[0093] In some embodiments, the system further includes a domain name resolution system; The domain name resolution system is used to respond to the domain name resolution request for the inference service sent by the inference client and return a candidate list to the inference client, the candidate list including the network access address of at least one candidate inference service instance.

[0094] The implementation principle and technical effects of the binding inference service instance system provided in this embodiment of the invention are similar to those of the above embodiments, and will not be repeated here.

[0095] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0096] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for binding inference service instances, characterized in that, Applied to an inference client, the method includes: During the establishment of the inference session, obtain the instance identification information of at least one candidate inference service instance; If the inference client has historical instance identification information stored locally, the locally stored historical instance identification information is matched with the instance identification information of the at least one candidate inference service instance; If a match is successful, the candidate inference service instance that has been matched is determined as the target inference service instance, and a session connection is established or maintained with the target service instance to conduct inference dialogue. The historical instance identification information refers to the instance identification information of the inference service instance that the inference client connected to in the previous inference session.

2. The method according to claim 1, characterized in that, The method further includes: If the historical instance identification information is not stored locally on the inference client or the matching fails, select one from the at least one candidate inference service instance to determine as the target inference service instance.

3. The method according to claim 1 or 2, characterized in that, The instance identification information is a unique identifier separate from the network access address; therefore, obtaining the instance identification information of at least one candidate inference service instance during the inference session establishment process includes: Send a domain name resolution request for inference service to the domain name resolution system to receive a candidate list returned by the domain name resolution system, the candidate list including the network access address of at least one candidate inference service instance; When the inference client locally stores historical instance identification information, the locally stored historical instance identification information is matched with the instance identification information of the at least one candidate inference service instance; if the match is successful, the successfully matched candidate inference service instance is determined as the target inference service instance, and a session connection is established or maintained with the target service instance to conduct inference dialogue, including: Determine the current candidate inference service instance in the candidate list; Establish a session connection with the current candidate inference service instance and receive a unique identifier pushed by the current candidate inference service instance; If the inference client has a locally stored historical unique identifier, the locally stored historical unique identifier is matched with the unique identifier of the current candidate inference service instance; If a match is successful, the current candidate inference service instance is determined as the target inference service instance, and a session connection is maintained with the target service instance to conduct inference dialogue; If a match fails, the session connection with the current candidate inference service instance is disconnected, and the next candidate inference service instance in the candidate list is selected as the current candidate inference service instance. The step of establishing a session connection with the current candidate inference service instance is repeated.

4. The method according to claim 3, characterized in that, The method further includes: If the inference client does not store any historical unique identifiers locally, or if the locally stored historical unique identifiers fail to match any of the unique identifiers in the candidate list, the first candidate inference service instance in the candidate list will be determined as the target inference service instance.

5. The method according to claim 1 or 2, characterized in that, If the instance identification information is a network access address, then obtaining the instance identification information of at least one candidate inference service instance during the inference session establishment process includes: Send a domain name resolution request for inference service to the domain name resolution system to receive a candidate list returned by the domain name resolution system, the candidate list including the network access address of at least one candidate inference service instance; When the inference client locally stores historical instance identification information, the locally stored historical instance identification information is matched with the instance identification information of the at least one candidate inference service instance; if the match is successful, the successfully matched candidate inference service instance is determined as the target inference service instance, and a session connection is established or maintained with the target service instance to conduct inference dialogue, including: If the inference client has historical network access addresses stored locally, the stored historical network access addresses will be matched with the network access addresses in the candidate list. If a match is successful, the candidate inference service instance that has been matched is determined as the target inference service instance, and a session connection is established with the target inference service instance to conduct inference dialogue.

6. The method according to claim 5, characterized in that, The method further includes: If the inference client does not store any historical network access addresses locally, or if the locally stored historical network access addresses fail to match any of the network access addresses in the candidate list, the first candidate inference service instance in the candidate list will be used as the target inference service instance.

7. A method for binding inference service instances, characterized in that, Applied to an inference service instance, the method includes: when the inference service instance is a candidate inference service instance, sending instance identification information to the inference client during the inference session establishment process; The instance identifier information of the candidate inference service instance is used by the inference client to match the locally stored historical instance identifier information with the instance identifier information of the candidate inference service instance when the client has stored historical instance identifier information locally. If the match is successful, the successfully matched candidate inference service instance is determined as the target inference service instance, and a session connection is established or maintained with the target service instance to carry out inference dialogue. The historical instance identifier information is the instance identifier information of the inference service instance that the inference client connected to in the previous round of inference session.

8. The method according to claim 7, characterized in that, The method further includes: The network access addresses of inference service instances are pre-registered with the domain name resolution system so that when the domain name resolution system receives a domain name resolution request sent by the inference client, it returns a candidate list to the inference client, and the candidate list includes the network access addresses of at least one candidate inference service instance.

9. A system for binding inference service instances, characterized in that, include: An inference client and at least one inference service instance; The inference client is used to execute the binding inference service instance method as described in any one of claims 1-6, and the inference service instance is used to execute the binding inference service instance method as described in claim 7 or 8.

10. The system according to claim 9, characterized in that, It also includes the domain name resolution system; The domain name resolution system is used to respond to the domain name resolution request for the inference service sent by the inference client and return a candidate list to the inference client, the candidate list including the network access address of at least one candidate inference service instance.

Images