Processing method of inference request and storage medium

Abstract

The application provides a processing method of an inference request and a storage medium, and relates to the technical field of computers. The processing method comprises the following steps: in response to the inference request, detecting a distributed computing power node and a centralized computing power node connected to a computing power scheduling platform; if there is a schedulable computing power node in the distributed computing power node, obtaining an evaluation result of the distributed computing power node based on a target parameter of the distributed computing power node, the evaluation result being used to indicate whether the distributed computing power node is available; obtaining a target computing power node in the distributed computing power node and the centralized computing power node based on the evaluation result of the distributed computing power node; and executing an inference task corresponding to the inference request through the target computing power node and outputting an inference result. The processing method can improve the stability when the inference task is executed in a controllable cost case by executing the inference task through a suitable target computing power node.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method for processing reasoning requests and a storage medium in computer technology. Background Technology

[0002] When performing inference tasks, terminals expect to obtain stable and low-latency large model inference capabilities through a unified application programming interface (API). Related technologies typically employ distributed or centralized computing nodes (e.g., enterprise-level Model-as-a-Service interfaces). However, distributed computing nodes frequently come online and go offline, making them susceptible to preemption, leading to inference service interruptions and incomplete streaming output, among other system instability issues. Centralized computing nodes, on the other hand, are constrained by quotas and pricing strategies, resulting in higher inference costs.

[0003] Therefore, under the premise of controllable costs, how to utilize computing power nodes to improve the stability of inference tasks is a technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a method for processing inference requests and a storage medium. This method can call appropriate target computing power nodes to execute inference tasks while keeping costs under control, thereby improving the stability of inference task execution.

[0005] Firstly, a method for processing inference requests is provided, the method being applied to a computing power scheduling platform, and the method includes: In response to an inference request, the distributed computing power nodes and centralized computing power nodes connected to the computing power scheduling platform are detected. The inference request is used to request the computing power scheduling platform to execute an inference task. If there are schedulable computing nodes among the distributed computing power nodes, an evaluation result of the distributed computing power node is obtained based on the target parameters of the distributed computing power node, and the evaluation result is used to indicate whether the distributed computing power node is available; Based on the evaluation results of the distributed computing power nodes, the target computing power node among the distributed computing power nodes and the centralized computing power nodes is obtained; The inference task corresponding to the inference request is executed by the target computing node, and the inference result is output.

[0006] In the embodiments of this application, the evaluation results of the distributed computing power nodes are obtained based on the target parameters of the distributed computing power nodes. Based on the evaluation results, target computing power nodes are determined between the distributed and centralized computing power nodes, and the inference task corresponding to the inference request is executed through the target computing power nodes. Compared to related technologies that use a single type of computing power node, such as only using distributed or only using centralized computing power nodes, which leads to unstable inference services or high inference costs, this solution can select a suitable target computing power node from distributed and centralized computing power nodes based on the evaluation results. By evaluating the availability of distributed computing power nodes in real time, the status, online fluctuations, and resource preemption risks of distributed computing power nodes can be dynamically perceived, ensuring that suitable target computing power nodes can be dynamically scheduled to execute inference tasks. This avoids the problem of inference service anomalies caused by using a single type of node, and ensures that the stability and fault tolerance of the computing power scheduling platform when executing inference services are enhanced while keeping costs under control, thereby effectively improving the success rate of inference task execution.

[0007] In conjunction with the first aspect, in certain implementations of the first aspect, obtaining the evaluation result of the distributed computing power node based on the target parameters of the distributed computing power node includes: Based on the target parameters of the distributed computing power node, the evaluation score of the distributed computing power node is determined; If there is a computing power node among the distributed computing power nodes whose evaluation score is greater than a preset score, it is determined that the evaluation result indicates that the distributed computing power node is available; If there is no distributed computing power node with an evaluation score greater than a preset score, the evaluation result indicates that the distributed computing power node is unavailable.

[0008] In the embodiments of this application, if there is a computing power node among the distributed computing power nodes with an evaluation score greater than a preset score, it indicates that there is a schedulable computing power node that can be used to execute inference tasks. Therefore, the evaluation result indicates that the distributed computing power node is available. If there is no computing power node among the distributed computing power nodes with an evaluation score greater than the preset score, it indicates that the distributed computing power node cannot execute inference tasks (for example, the schedulable computing power node is in a fluctuating state or the risk of resource preemption is high). Therefore, the evaluation result indicates that the distributed computing power node is unavailable. This ensures that the distributed computing power node can be evaluated based on the evaluation score corresponding to the target parameter, obtaining a more accurate evaluation result, and ensuring that the evaluation result can effectively reflect whether the distributed computing power node can execute inference tasks (i.e., whether the distributed computing power node is available).

[0009] In conjunction with the first aspect and the above-described implementations, in some implementations of the first aspect, the processing method further includes: Obtain the target parameters, which include at least one of the performance score, risk assessment level, and cache benefit of the distributed computing node, wherein the cache benefit is used to represent the performance benefit of reusing cached content for inference operations; Determining the evaluation score of the distributed computing power node based on the target parameters of the distributed computing power node includes: The evaluation score is obtained based on at least one of the performance score, the risk assessment level, and the caching benefit, wherein the evaluation score is positively correlated with the performance score, negatively correlated with the risk assessment level, and positively correlated with the caching benefit.

[0010] In the embodiments of this application, an evaluation score is determined based on at least one of the performance score, risk assessment level, and caching benefit of the distributed computing power node. This ensures that the evaluation score reflects the availability of the distributed computing power node, thereby ensuring that the evaluation results of the distributed computing power node can be obtained. Since the evaluation score is positively correlated with the performance score, computing power nodes with better performance can be prioritized, which helps reduce inference latency, increase throughput, and ensure the smoothness and response speed of streaming output. Since the evaluation score is negatively correlated with the risk assessment level, nodes with high operating risk, easy preemption, and poor stability can be avoided, thereby reducing problems such as inference task interruption, semantic incompleteness, and service fluctuations, and improving the system reliability of the computing power scheduling platform. Since the evaluation score is positively correlated with caching benefit, nodes with higher caching benefit can be prioritized for scheduling, improving cache hit rate, effectively improving resource utilization, and reducing inference time.

[0011] In combination with the first aspect and the above implementation methods, in some implementation methods of the first aspect, obtaining the target parameter includes: The performance score is determined based on the performance parameters of the distributed computing nodes, including the throughput, latency, and token generation rate of the distributed computing nodes. The risk assessment level is determined based on the heartbeat, remaining lease time, historical preemption frequency, and system prompt signals of the distributed computing power nodes. If the inference request carries a handle and the handle is valid, the cache benefit is determined based on the first performance metric when the cache is hit and the second performance metric when the cache is missed.

[0012] In the embodiments of this application, a performance score is determined based on the performance parameters of the distributed computing nodes, including the throughput, latency, and token generation rate of the distributed computing nodes, ensuring that the performance score reflects the performance of the distributed computing nodes. A risk assessment level is determined based on the heartbeat, remaining lease time, historical preemption frequency, and system alert signals of the distributed computing nodes, ensuring that the risk assessment level reflects the degree of risk of preemption of the distributed computing nodes. If the inference request carries a handle and the handle is valid, cache benefits are determined based on a first performance indicator of cache hits and a second performance indicator of cache misses, ensuring that cache benefits reflect the performance benefits generated by the distributed computing nodes reusing cached content for inference operations.

[0013] In conjunction with the first aspect and the above implementation methods, in some implementation methods of the first aspect, obtaining the target computing power node among the distributed computing power nodes and the centralized computing power nodes based on the evaluation results of the distributed computing power nodes includes: If the evaluation result indicates that the distributed computing power node is available, the computing power node with the highest evaluation score among the distributed computing power nodes shall be determined as the target computing power node; If the evaluation result indicates that the distributed computing node is unavailable, the centralized computing node will be identified as the target computing node.

[0014] In the embodiments of this application, if the evaluation result indicates that the distributed computing power node is available, the computing power node with the highest evaluation score among the distributed computing power nodes is determined as the target computing power node; if the evaluation result indicates that the distributed computing power node is unavailable, the centralized computing power node is determined as the target computing power node. This ensures that when the distributed idle computing power node is available, low-cost and flexible distributed computing power is prioritized to undertake inference tasks; when the distributed nodes are unavailable, fluctuate, or are preempted, the task is switched to a highly reliable centralized computing power node to perform inference tasks, balancing stability and economy, achieving computing power synergy and complementarity, and effectively reducing the probability of inference service interruption and inference request failure while keeping costs under control.

[0015] In conjunction with the first aspect and the above-described implementations, in some implementations of the first aspect, the processing method further includes: Determine the lease status and heartbeat information of the distributed computing power nodes; If there is a node in the distributed computing power nodes whose lease status indicates that the lease is valid and whose heartbeat information indicates that the heartbeat has not timed out, it is determined that there is a schedulable computing power node in the distributed computing power nodes. If the lease status of the distributed computing power node indicates that the lease has expired or the heartbeat information indicates that the heartbeat has timed out, it is determined that there is no schedulable computing power node among the distributed computing power nodes.

[0016] In the embodiments of this application, if the lease of a distributed computing power node is valid, it indicates that the resource usage authorization between the distributed computing power node and the computing power scheduling platform is within the validity period (i.e., the computing power scheduling platform has the resource usage rights of the distributed computing power node), and the heartbeat of the distributed computing power node not timed out indicates that the communication link between the distributed computing power node and the computing power scheduling platform is unobstructed. If there is a node among the distributed computing power nodes with a valid lease and a heartbeat that has not timed out, it indicates that the computing power scheduling platform can use the computing power resources of the node and is in a state of unobstructed communication link with the node. Therefore, it is determined that there is a schedulable computing power node among the distributed computing power nodes.

[0017] In conjunction with the first aspect and the above implementation methods, in some implementation methods of the first aspect, when the target computing power node is the distributed computing power node, the processing method further includes: If it is detected that the target computing power node is occupied, the lease has expired, or the heartbeat timeout has occurred, control the target computing power node to stop receiving inference requests; If the target computing node is executing an inference task, determine the reserved computing resources of the target computing node. The reserved computing resources represent the computing resource quota reserved for the inference task being executed before the computing node's resources are reclaimed. The current inference task is executed based on the reserved computing resources.

[0018] In the embodiments of this application, when the target computing power node is a distributed computing power node, if it is detected that the target computing power node is occupied, the lease expires, or the heartbeat times out, that is, the target computing power node changes from an available state to an unavailable state, the target computing power node is controlled to stop receiving inference requests, ensuring that it no longer responds to inference requests when the target computing power node is unable to perform inference tasks. If the target computing power node is performing an inference task, the current inference task is executed according to the reserved computing power resources, reducing the probability of incomplete content output when the computing power node is preempted.

[0019] In combination with the first aspect and the above implementation methods, in some implementation methods of the first aspect, the execution of the current inference task based on the reserved computing power resources includes: Determine the completion predicate corresponding to the streaming protocol type of the current session. The completion predicate takes the streaming output event sequence as input and outputs a completion flag. When the reserved computing power resources are not exhausted and the completion predicate does not output a completion flag, the inference task is executed using the reserved computing power resources of the target computing power node.

[0020] In the embodiments of this application, the completion predicate corresponding to the elapsed protocol type of the current session is determined. When the reserved computing power resources are not exhausted and no completion flag is output, the inference task continues to be executed through the reserved computing power of the target computing power node, ensuring that the computing power scheduling platform can do its best to complete the in-transit session and output a unified end event within the budget window, thereby reducing the probability of incomplete content output.

[0021] In conjunction with the first aspect and the above-described implementations, in some implementations of the first aspect, the processing method further includes: When the reserved computing power resources are exhausted and the completion predicate does not output a completion flag, the inference task is executed through the centralized computing power node, wherein the number of times the centralized computing power node executes the inference task is less than a preset number; If the number of times the centralized computing node executes the inference task exceeds a preset number and the completion predicate does not output a completion flag, the inference task is determined to have failed.

[0022] In the embodiments of this application, when reserved computing power resources are exhausted and the completed predicate does not output a completion flag, the inference task is executed by a centralized computing power node, and the number of times the centralized computing power node executes the inference task is limited to less than a preset number. This ensures that when the distributed computing power nodes are unable to execute the inference task, the centralized computing power node executes the inference task, reducing the risk of inference task execution failure. Furthermore, by limiting the number of times the centralized computing power node executes the inference task, the probability of a retry storm occurring on the computing power scheduling platform is reduced, ensuring the system stability of the schedulable computing power platform.

[0023] Secondly, a processing apparatus for reasoning requests is provided, the processing apparatus comprising: The detection module is used to detect distributed computing nodes and centralized computing nodes connected to the computing power scheduling platform in response to an inference request. The inference request is used to request the computing power scheduling platform to execute an inference task. The processing module is configured to, if there are schedulable computing nodes among the distributed computing nodes, obtain an evaluation result of the distributed computing nodes based on the target parameters of the distributed computing nodes, wherein the evaluation result is used to indicate whether the distributed computing nodes are available; based on the evaluation result of the distributed computing nodes, obtain a target computing node among the distributed computing nodes and the centralized computing nodes; execute the inference task corresponding to the inference request through the target computing node, and output the inference result.

[0024] Thirdly, a computing power scheduling platform is provided, which includes a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the computing power scheduling platform executes the processing method in the first aspect or any possible implementation of the first aspect.

[0025] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to perform the processing method described in the first aspect or any possible implementation thereof.

[0026] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the processing method described in the first aspect or any possible implementation thereof. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this application; Figure 2 This is a schematic flowchart illustrating a method for processing inference requests provided in an embodiment of this application; Figure 3 This is a schematic flowchart illustrating another method for processing inference requests provided in an embodiment of this application; Figure 4 This is a schematic flowchart illustrating another reasoning request processing method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of a reasoning request processing device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computing power scheduling platform provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0029] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0030] When performing inference tasks, terminals expect to obtain stable and low-latency large model inference capabilities through a unified application programming interface (API). Related technologies typically employ distributed or centralized computing nodes. Distributed computing nodes distribute computing power across multiple independent devices in different locations, scheduling idle computing resources on these nodes to execute inference tasks and output inference results. Centralized computing nodes centrally deploy computing resources within a unified central server or data center, uniformly scheduling, managing, and processing all tasks. However, distributed computing nodes frequently come online and go offline, making them susceptible to preemption, leading to inference service interruptions and incomplete streaming output, among other system instability issues. Centralized computing nodes, constrained by quotas and pricing strategies, result in higher inference costs.

[0031] The following examples further illustrate the technical problems existing in the relevant technologies.

[0032] For example, distributed computing nodes have lower resource usage costs, but due to the lack of fixed public network access addresses, nodes frequently come online and go offline. Furthermore, during computing resource scheduling, they are easily preempted by higher-priority tasks. This leads to systemic instability issues when using distributed computing nodes for inference tasks, such as inference service interruptions, incomplete streaming output, and concentrated retries of numerous requests (i.e., distributed computing nodes are low-cost but have poor stability). Enterprise-level Model-as-a-Service (MAS) interfaces, on the other hand, are standardized calling services provided by professional vendors. They rely on stable centralized computing nodes, resulting in more reliable service availability, response speed, and output results, and higher stability. However, these services require payment based on the number of calls, making inference relatively expensive. Service providers typically set call quotas, bandwidth limits, and corresponding pricing strategies. Due to the lack of a sustainable and controllable mechanism for introducing low-cost computing power, the system is prone to excessively high unit request costs or insufficient available throughput in high-concurrency or cost-sensitive scenarios, thus limiting service scale and economics (i.e., centralized computing nodes are highly stable but have higher costs).

[0033] Furthermore, online inference for large language models widely employs streaming output to reduce initial packet latency and improve the user experience. When distributed computing nodes are in a preemptible environment, if a node is preempted or interrupted during the inference service execution process, it's difficult to guarantee the complete delivery of streaming semantics, and output may easily terminate midway. Clients often trigger retries to obtain complete results. These retries can amplify requests in a short time, causing queue congestion and increased tail latency, and may further induce cascading failures, thus affecting the stability of the inference service.

[0034] Therefore, under the premise of controllable costs, how to utilize computing power nodes to improve the stability of inference tasks is a technical problem that needs to be solved.

[0035] In view of this, this application provides a method for processing inference requests and a storage medium. Through embodiments of this application, the evaluation results of distributed computing power nodes are obtained based on the target parameters of the distributed computing power nodes. Based on the evaluation results, target computing power nodes are determined between the distributed computing power nodes and the centralized computing power nodes. The inference task corresponding to the inference request is then executed through the target computing power node. By real-time evaluation of the availability of distributed computing power nodes, the status, online fluctuations, and resource preemption risks of distributed computing power nodes can be dynamically perceived. This allows for the dynamic scheduling of suitable target computing power nodes, avoiding inference service anomalies caused by a single type of computing power anomaly. This ensures that, while maintaining controllable costs, the stability and fault tolerance of the computing power scheduling platform when executing inference services are enhanced, effectively improving the success rate of inference task execution.

[0036] Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this application.

[0037] For example, Figure 1 The system architecture 100 shown is the system architecture of the computing power scheduling platform. The computing power scheduling platform is a system platform that uniformly manages, allocates, and schedules various computing power resources (including centralized computing power, distributed computing power, etc.).

[0038] like Figure 1As shown, the system architecture 100 includes terminals, a unified service entry point, a perception and scheduling module, distributed computing power nodes (providing idle computing power inference services), and centralized computing power nodes (providing enterprise-level inference services). The unified service entry point is responsible for external consistency (i.e., different terminals all call the inference services of computing power nodes through the unified service entry point), the perception and scheduling module is responsible for internal decision-making closure, the idle computing power inference service is responsible for low-cost bearing and emptying management under preemption conditions, and the enterprise-level inference service is responsible for executing inference tasks when the idle computing power inference service is unavailable. Through the above layering, the computing power scheduling platform can maintain a unified interface, controllable costs, and continuous delivery of streaming semantics under the constraints of dynamic online and offline status of idle computing power and preemption.

[0039] The composition and function of each module in the system architecture will be further explained below.

[0040] For example, the unified service entry point provides a consistent Application Programming Interface (API) to the terminal and standardizes the terminal's inference requests before handing them over to the perception and scheduling module for routing and governance. The unified service entry point can be implemented as a Hypertext Transfer Protocol (HTTP) gateway, Hypertext Transfer Protocol Secure (HTTPS) gateway, API gateway, reverse proxy, etc., preferably supporting streaming output and long connection multiplexing. Among them, the HTTP / HTTPS gateway refers to a network service that encrypts and authenticates data through the SSL / TLS protocol (SSL / TLS is a security protocol that provides encryption, integrity, and authentication in a network) to ensure the security of data transmission; the API gateway refers to receiving all external requests, performing identity verification, access control, traffic limiting, log monitoring, routing and forwarding, and distributing requests to computing power nodes; the reverse proxy refers to the proxy server receiving client requests, where the client is unaware of the existence of the real backend server, and the request is sent directly to the reverse proxy, which then distributes it to the backend server for processing and returns the result to the client.

[0041] For example, the unified service entry point's functional components include at least: a request access and authentication module, a protocol adaptation and unified parameter parsing module, a rate limiting and circuit breaking module, and a request tracing and auditing module. The request access and authentication module receives end-user requests and performs signature verification, token verification, authentication, and tenant identification. The protocol adaptation and unified parameter parsing module normalizes differences in protocols between different model providers or clients. The rate limiting and circuit breaking module performs rate limiting based on metrics such as tenant, model, and concurrency, and performs circuit breaking when the backend is abnormal (i.e., cutting off the request path to the backend and ceasing requests to the backend when the backend is abnormal). The request tracing and auditing module assigns a global request ID to each request, records link tracing information, and is used for problem localization and billing reconciliation.

[0042] After the terminal initiates a request to the unified service entry point, the unified service entry point processes it according to the following steps: receiving the request and verifying the authentication information; parsing parameters such as the model name, maximum number of output tokens, and streaming flag to generate an internal standard request object; generating a Request ID and initializing the streaming output channel; forwarding the standard request to the perception scheduling module, and the request enters the closed loop of scheduling, execution, and feedback; finally, receiving the streaming incremental event from the perception scheduling module and writing it back to the terminal until the output ends.

[0043] For example, the perception and scheduling module is the core decision-making unit of this invention, used to continuously perceive changes in the status of idle computing power nodes, determine schedulable computing power nodes, and perform request routing and rollback management. The perception and scheduling module includes at least: a node access management module, a performance calibration and scoring module, a risk assessment and preemption prediction module, a routing selection and context affinity module, a rollback management and semantic completion determination module, and a rollback and retry suppression module.

[0044] The system comprises the following modules: a node access management module for maintaining the connection status, lease status, and heartbeat status of idle computing power nodes, and constructing a set of schedulable computing power nodes; a performance calibration and scoring module for scoring the performance of schedulable computing power nodes; a risk assessment and preemption prediction module for estimating the risk level of distributed computing power nodes being preempted based on node heartbeats, remaining lease time, historical preemption frequency, and system alert signals; a routing and context affinity module for determining the target computing power node (which can be a distributed or centralized computing power node) by combining at least one of performance scores, caching benefits, and risk assessment levels; and a queuing and semantic completion determination module for performing queueing when distributed computing power nodes are preempted or experience fluctuations. Empty governance (the empty governance state indicates that when a distributed computing power node is preempted, it stops receiving new inference requests and continues to output the streaming increment of the current session by reserving computing power resources), and determines whether the output is completed by completing the predicate library; the rollback and retry suppression module is used to roll back the enterprise-level inference service of the centralized computing power node when there is insufficient idle computing power or the risk increases, so that the enterprise-level inference service performs the inference task, and suppresses the number of times the enterprise-level inference service performs the inference task to prevent retry storm. Here, retry storm means that when a large number of inference requests fail or time out, the client automatically initiates dense repeated requests, resulting in an explosive retry traffic in a short period of time, which aggravates service congestion, resource exhaustion, and ultimately leads to the overall service avalanche.

[0045] For example, an idle computing power inference service is an inference service deployed in a distributed computing power node that can be preempted. An enterprise-level inference service refers to an inference service deployed in a centralized computing power solution, used to execute inference tasks when the distributed computing power node is unavailable. The idle computing power inference service includes a drain agent and an idle inference engine.

[0046] The drain proxy is used to address the issues of distributed computing nodes lacking fixed public network addresses and streaming interruptions caused by node preemption. The drain proxy includes at least an outbound long-connection establishment module, a multiplexing transmission module, a lease reception and renewal module, and a drain state machine module. The outbound long-connection establishment module proactively establishes an outbound-only connection with the aggregation platform, allowing the platform to access nodes without relying on their inbound addresses. The multiplexing transmission module reuses control plane messages and inference data streams on the same outbound connection. The lease reception and renewal module receives lease identifiers and lease validity windows from the aggregation platform and renews leases by sending heartbeat messages according to a heartbeat cycle. The drain state machine module enters a drain governance state upon receiving a preemption notification or lease revocation instruction from a distributed computing node. In the drain governance state, distributed computing nodes stop receiving new inference requests, continue outputting the current session's streaming increment, and send a controlled termination reason back to the computing power scheduling platform when the drain budget is exhausted.

[0047] The outbound long-lived connection can take the form of WebSocket, HTTP / 2 (Hypertext Transfer Protocol 2) bidirectional streaming, or a custom Transport Layer Security (TLS) / Transmission Control Protocol (TCP) long-lived connection. WebSocket is a long-lived connection established between a browser and a server, enabling bidirectional real-time communication without frequent disconnections and reconnections. HTTP / 2 bidirectional streaming is a high-performance communication protocol based on HTTP / 2, allowing the server and client to send data streams to each other simultaneously. TCP / TLS long-lived connections refer to defining a custom communication format on top of the underlying TCP layer and establishing a stable, secure, and persistent long-lived connection through TLS encryption. The multiplexed frame of the multiplexed transmission module includes at least a frame type field, a session identifier field, and optional fields. The frame type field is used for control / heartbeat / inference request / incremental output / end / error. The session identifier field is used to distinguish multiple inference sessions on the same connection. Optional fields include the packet sequence number (Seq), payload length (Length), and checksum.

[0048] The idle-time inference engine executes inference tasks corresponding to inference requests and provides streaming output. The idle-time inference engine includes at least an inference execution module, a key-value cache management module, a context handle generation module, and a metric collection and reporting module. The inference execution module performs the prefill and decoding phases and outputs incremental tokens to the agent. Here, a token represents a key or word, which is the smallest semantic unit in a language model. The key-value (KV) cache management module maintains the session's key-value cache to improve decoding efficiency and support context reuse. The context handle generation module generates context handle metadata without sharing the KV cache. The metric collection and reporting module periodically reports throughput (tokens / s, representing the number of tokens processed per second), latency, available GPU memory, queue length, etc.

[0049] For example, the enterprise-grade inference service is a deterministic resource pool with public network accessibility, stability, and a controllable failure rate. It primarily serves as a fallback target when idle computing power is insufficient; as a deterministic backup when risks increase or preemption is frequent; and as a resource to carry requests requiring higher consistency or higher reliability. The awareness and scheduling module can interface with one or more enterprise-grade inference service interfaces and maintain their billing, quota, and rate limiting policies.

[0050] The following is combined Figures 2 to 4The method for processing inference requests provided in the embodiments of this application will be described in detail.

[0051] Figure 2 This is a schematic flowchart illustrating a method for processing inference requests provided in an embodiment of this application.

[0052] For example, Figure 2 The processing method 200 shown can be executed by the computing power scheduling platform. The computing power scheduling platform (also known as the computing power aggregation platform) is a central scheduling system that uniformly receives, schedules, and manages inference tasks. It is responsible for the management of computing power nodes, task routing, resource allocation, and streaming output control.

[0053] like Figure 2 As shown, the reasoning request processing method 200 includes S210 to S240. The reasoning request processing methods shown in S210 to S240 will be described in detail below.

[0054] S210, in response to inference requests, detects distributed and centralized computing nodes connected to the computing power scheduling platform.

[0055] Among them, the inference request is initiated by the user through the terminal and is used to request the computing power scheduling platform to call the computing power node to execute the inference task of text generation. The inference request usually includes parameters such as model name, input prompt, maximum number of output tokens, and streaming tags.

[0056] For example, a computing node refers to the smallest unit or device capable of independently providing computing, storage, and network resources. The computing power scheduling platform connects to at least one distributed computing node and a centralized computing node; wherein, a distributed computing node refers to a computing node whose computing power is dispersed across multiple independent devices in different locations, and the computing power scheduling platform schedules the distributed idle computing resources on the distributed computing nodes to execute inference tasks and output inference results; a centralized computing node refers to a computing node whose computing resources are centrally deployed in a unified central server or data center, and the computing power scheduling platform manages and processes inference tasks uniformly by scheduling the centralized computing nodes.

[0057] For example, distributed computing nodes proactively establish outbound long-lived connections with the computing power scheduling platform. A long-lived connection means that multiple data packets can be sent continuously within a single network connection, maintaining the connection state during the connection period. An outbound long-lived connection refers to the distributed computing nodes proactively connecting to the platform and maintaining that connection. By maintaining communication links with the platform through the distributed computing nodes, the platform can manage and schedule inference tasks for the distributed computing nodes without needing to know the nodes' public Internet Protocol (IP) addresses, ports, or other inbound reachable addresses.

[0058] It should be noted that idle distributed computing nodes are typically located within intranets or firewall environments, lacking fixed public addresses that can be actively accessed. This prevents the computing power scheduling platform from proactively sensing the status of distributed computing nodes. Existing computing power scheduling platforms often rely on static addresses or weakly consistent node registration information for forwarding, making it difficult for the platform to detect changes in node availability in a timely and accurate manner. This can easily lead to node unreachability and routing mismatches, further amplifying system queuing delays and resource waste, resulting in increased failure rates and retries for inference tasks. Therefore, this invention addresses this by having distributed computing nodes proactively establish outbound long connections, enabling the computing power scheduling platform to maintain reachability and perform session-level forwarding without relying on node inbound addresses. This ensures that communication links are maintained through outbound long connections, allowing the computing power scheduling platform to issue commands in real time. Furthermore, outbound long connections prevent nodes from exposing public ports, improving the security of distributed computing nodes, ensuring real-time data backhaul during streaming inference, and reducing data transmission latency.

[0059] In one implementation, in response to an inference request, the inference request is validated. If the validation passes, the parameters of the inference request are parsed and a standard request is generated. The parameters of the inference request include the model name, the maximum number of output tokens, streaming tokens, and other parameters.

[0060] For example, the verification of inference requests includes at least one of signature verification, token verification, authentication, and tenant identification. Signature verification verifies the signature information carried in the inference request to confirm whether the inference request data has been tampered with during transmission and to verify the authenticity and integrity of the inference request source. Token verification verifies the validity of the access token carried in the inference request, checking whether the token has expired, is valid, or is within its authorized validity period. Authentication verifies whether the requester triggering the inference request has the authority to access the corresponding model service, call the corresponding interface, and perform the corresponding operation, ensuring that the access behavior complies with the preset permission policy. Tenant identification parses and determines the tenant entity to which the current request belongs.

[0061] In the embodiments of this application, by verifying inference requests and converting them into standard requests, external requests from different sources and in different formats are uniformly converted into a standardized internal request structure that can be recognized and executed by the computing power nodes, thereby decoupling the upper-layer access from the lower-layer computing power execution. This ensures the access security of the computing power scheduling platform, prevents illegal inference requests from entering the computing power scheduling process, and provides a basis for subsequent task scheduling, resource isolation, and traffic control.

[0062] S220, If there are schedulable computing nodes among the distributed computing power nodes, the evaluation result of the distributed computing power nodes is obtained based on the target parameters of the distributed computing power nodes.

[0063] Among them, schedulable computing power nodes are distributed computing power nodes that the computing power scheduling platform can schedule. The target parameters include at least one of the performance score, risk assessment level and cache benefit of the distributed computing power node. Cache benefit is used to represent the performance benefit of reusing cached content for inference operations.

[0064] The method for determining schedulable computing power nodes will be further explained below.

[0065] In one implementation, the lease status and heartbeat information of the distributed computing power nodes are determined; if there is a node in the distributed computing power nodes whose lease status indicates that the lease is valid and whose heartbeat information indicates that the heartbeat has not timed out, it is determined that there is a schedulable computing power node in the distributed computing power nodes; if the lease status of the distributed computing power nodes indicates that the lease is invalid or the heartbeat information indicates that the heartbeat has timed out, it is determined that there is no schedulable computing power node in the distributed computing power nodes.

[0066] In this context, a heartbeat refers to a short message proactively sent by a distributed computing node to the computing power scheduling platform at a preset period. This message reports the node's current online, connected, and manageable status. The computing power scheduling platform determines whether a node is disconnected, offline, or abnormal by continuously receiving heartbeat messages. A lease refers to the resource usage authorization period agreed upon between the computing power scheduling platform and the distributed computing node, representing the authorized usage period for the node to occupy computing resources and execute inference tasks.

[0067] For example, a valid lease for a distributed computing node indicates that the resource usage authorization between the distributed computing node and the computing power scheduling platform is valid (i.e., the computing power scheduling platform has the resource usage rights of the distributed computing node), and the heartbeat of the distributed computing node not timed out indicates that the communication link between the distributed computing node and the computing power scheduling platform is uninterrupted. The set of schedulable computing nodes is the set of single nodes that can be connected to the distributed computing node.

[0068] For example, when determining the lease status, the current time is compared with the lease expiration time. If the current time is earlier than or equal to the lease expiration time, the lease status is determined to be valid; if the current time is later than the lease expiration time, the lease status is determined to be invalid. The time difference between the current heartbeat time and the most recent heartbeat time is checked against a time threshold. If the time difference is greater than the time threshold, the heartbeat has not timed out; if the time difference is less than or equal to the time threshold, the heartbeat has timed out. If there are nodes in the distributed computing power nodes with valid leases and no timeout heartbeats, it means that the computing power scheduling platform can use the computing power resources of that node and has a smooth communication link with that node. Therefore, it is determined that there are schedulable computing power nodes in the distributed computing power nodes. If there are no computing power nodes in the distributed computing power nodes with valid leases and no timeout heartbeats, it is determined that there are no schedulable computing power nodes in the distributed computing power nodes.

[0069] In one implementation, the evaluation result of the distributed computing power node is obtained based on the target parameters of the distributed computing power node, including: determining the evaluation score of the distributed computing power node based on the target parameters of the distributed computing power node; if there is a computing power node among the distributed computing power nodes with an evaluation score greater than a preset score, determining that the evaluation result indicates that the distributed computing power node is available; if there is no computing power node among the distributed computing power nodes with an evaluation score greater than the preset score, determining that the evaluation result indicates that the distributed computing power node is unavailable.

[0070] Understandably, if there are computing power nodes in the distributed computing power network with an evaluation score greater than the preset score, it indicates that there are schedulable computing power nodes capable of performing inference tasks. Therefore, the evaluation result indicates that the distributed computing power nodes are available. If there are no computing power nodes in the distributed computing power network with an evaluation score greater than the preset score, it indicates that the schedulable computing power nodes in the distributed computing power network cannot perform inference tasks (e.g., the schedulable computing power nodes are in a fluctuating state or have a high risk of resource preemption). Therefore, the evaluation result indicates that the distributed computing power nodes are unavailable. It is essential to ensure that the distributed computing power nodes can be evaluated based on the evaluation score corresponding to the target parameters to obtain relatively accurate evaluation results, ensuring that the evaluation results effectively reflect whether the distributed computing power nodes can perform inference tasks (i.e., whether the distributed computing power nodes are available).

[0071] In one implementation, an evaluation score is obtained based on at least one of performance score, risk assessment level, and caching benefit, wherein the evaluation score is positively correlated with performance score, negatively correlated with risk assessment level, and positively correlated with caching benefit.

[0072] Understandably, since the evaluation score is positively correlated with the performance score, prioritizing higher-performing computing nodes helps reduce inference latency, increase throughput, and ensure smooth and responsive streaming output. Because the evaluation score is negatively correlated with the risk assessment level, nodes with high operational risks, susceptibility to preemption, and poor stability can be avoided, thus reducing issues such as inference task interruptions, incomplete semantics, and service fluctuations, and improving the system reliability of the computing power scheduling platform. Furthermore, since the evaluation score is positively correlated with cache benefits, nodes with higher cache benefits can be prioritized for scheduling, increasing cache hit rate, effectively improving resource utilization, and reducing inference time.

[0073] For example, if the target parameters include performance score, risk assessment level, and caching benefit, the assessment score can be calculated using the method shown in Formula 1: ;(Formula 1) in, This represents the evaluation score of the distributed computing nodes. This represents the weighting coefficient for the performance score. The weighting coefficients representing the benefits of caching. This represents the weighting coefficient for the risk assessment level.

[0074] It is understandable that existing KV caching optimizations are mostly limited to a single inference engine or a single cluster control domain. This makes it difficult to uniformly model the cache availability and cache hit benefits of different nodes at the computing power scheduling platform level. Consequently, request allocation struggles to effectively control the cost of the pre-filling phase, leading to fluctuations in end-to-end latency and throughput, and reducing the availability and overall efficiency of low-cost resources. In the embodiments of this application, an evaluation score is determined based on the performance rating, risk assessment level, and cache benefits of the distributed computing power nodes. This ensures that the evaluation score reflects the availability of the distributed computing power nodes, thereby ensuring that the evaluation results of the distributed computing power nodes can be obtained.

[0075] In one possible implementation, before obtaining the evaluation results of the distributed computing power nodes based on their target parameters, the method for obtaining the target parameters is further explained below: Specifically, the target parameters are obtained, including: determining the performance score based on the performance parameters of the distributed computing nodes, including the throughput, latency, and token generation rate of the distributed computing nodes; determining the risk assessment level based on the heartbeat, remaining lease time, historical preemption frequency, and system prompt signals of the distributed computing nodes; and determining the cache benefit based on the first performance indicator when the cache is hit and the second performance indicator when the cache is missed, if the inference request carries a handle and the handle is valid.

[0076] Among them, the throughput of the distributed computing power node represents the number of inference requests or the total number of tokens that the distributed computing power node can process and complete per unit time, which is used to characterize the task processing capability of the distributed computing power node; the latency represents the time consumed from the inference request being sent to the distributed computing power node to the distributed computing power node returning the first token or completing the inference, which is used to characterize the response speed of the distributed computing power node; the token generation rate represents the number of tokens generated by the distributed computing power node per unit time during the inference process, which is used to characterize the execution efficiency of the node's streaming inference.

[0077] For example, when determining the performance score, the performance score is positively correlated with throughput (the higher the throughput, the better the performance of the distributed computing node, and the higher the performance score), negatively correlated with latency (the lower the latency, the better the performance of the distributed computing node, and the higher the performance score), and positively correlated with token generation rate (the higher the token generation rate, the better the performance of the distributed computing node, and the higher the performance score).

[0078] For example, the throughput, latency, and token generation rate of the distributed computing power node are normalized respectively. The normalized throughput value is positively weighted with the performance score; the normalized latency value is inverted or taken as its reciprocal and negatively weighted with the performance score; the normalized token generation rate value is positively weighted with the performance score; the above weighted results are weighted and summed or merged to obtain the performance score of the distributed computing power node.

[0079] For example, the heartbeat of a distributed computing node is a status message actively sent by the distributed computing node to the aggregation platform according to a preset period, used to characterize the communication connectivity and online status between the node and the platform. The remaining lease time represents the remaining valid duration of the current resource authorization for the distributed computing node, and the remaining duration for which the proxy node can still be safely used. The historical preemption frequency refers to the number of times the distributed computing node has been preempted, interrupted, or reclaimed by the system within a preset period in the past. The system alert signal is a warning signal triggered when a distributed computing node is about to enter an unavailable state (e.g., the distributed computing node has been preempted, system resources have been reclaimed, or the load is too high).

[0080] For example, the risk assessment level is negatively correlated with heartbeat stability. Lower heartbeat stability, such as abnormal heartbeats or heartbeat timeouts, indicates that the distributed computing node is disconnected or unreachable, posing a risk of inference task interruption, and thus a higher risk assessment level. The risk assessment level is also negatively correlated with the remaining lease time. A shorter remaining lease time increases the probability of lease expiration and task interruption during inference task execution, resulting in a higher risk assessment level. Furthermore, the risk assessment level is positively correlated with historical preemption frequency. A higher historical preemption frequency increases the probability of the computing node being preempted again during task execution, leading to a higher risk assessment level. Finally, the risk assessment level is positively correlated with system alert signals. Detecting a system alert signal indicates a higher risk assessment level.

[0081] For example, if the inference request carries a handle and the handle is valid, the cache benefit is determined based on a first performance metric when the cache hits and a second performance metric when the cache misses. Here, a handle refers to an index identifier used to identify historical inference sessions, context caches, or model state resources, used for quickly locating and reusing existing cached resources; a valid handle indicates that the cached resource pointed to by the handle has not been released, expired, or overwritten, and can be directly accessed and reused by the current inference request. The first performance metric represents the performance metrics during the inference execution process when the cache hits, including but not limited to: inference latency, first token response latency, token generation rate, resource consumption overhead, and context loading time. The second performance metric represents the performance metrics during the inference execution process when the cache misses, and the type of the second performance metric is the same as the type of the first performance metric.

[0082] For example, based on a first performance metric when the cache is hit and a second performance metric when the cache is missed, the performance metric difference between the first and second performance metrics is predicted; the cache benefit is determined based on the performance metric difference, wherein the cache benefit is positively correlated with the performance metric difference.

[0083] In the embodiments of this application, a performance score is determined based on the performance parameters of the distributed computing nodes, including the throughput, latency, and token generation rate of the distributed computing nodes, ensuring that the performance score reflects the performance of the distributed computing nodes. A risk assessment level is determined based on the heartbeat, remaining lease time, historical preemption frequency, and system alert signals of the distributed computing nodes, ensuring that the risk assessment level reflects the degree of risk of preemption of the distributed computing nodes. If the inference request carries a handle and the handle is valid, cache benefits are determined based on a first performance indicator of cache hits and a second performance indicator of cache misses, ensuring that cache benefits reflect the performance benefits generated by the distributed computing nodes reusing cached content for inference operations.

[0084] Optionally, if there are no schedulable computing nodes among the distributed computing power nodes, the centralized computing power node is determined as the target computing power node.

[0085] It should be noted that, since there may be distributed computing power nodes connected to the computing power scheduling platform that are in a connected state but unable to communicate normally (e.g., distributed computing power nodes with expired leases or abnormal heartbeat states), when there are schedulable computing power nodes among the distributed computing power nodes, the evaluation results of the distributed computing power nodes are obtained based on the target parameters of the distributed computing power nodes, and the target computing power node is determined based on the evaluation results; when there are no schedulable computing power nodes among the distributed computing power nodes, no further calculations are performed to avoid unnecessary computing power consumption caused by unnecessary calculations.

[0086] S230, based on the evaluation results of distributed computing power nodes, obtains the target computing power node among distributed computing power nodes and centralized computing power nodes.

[0087] The evaluation results are used to indicate whether the distributed computing nodes are available, that is, whether the distributed computing nodes can be used to perform inference tasks.

[0088] Specifically, if the evaluation results indicate that the distributed computing power node is available, the computing power node with the highest evaluation score among the distributed computing power nodes will be determined as the target computing power node; if the evaluation results indicate that the distributed computing power node is unavailable, the centralized computing power node will be determined as the target computing power node.

[0089] Understandably, if the evaluation results indicate that distributed computing nodes are available, the computing node with the highest evaluation score among the distributed computing nodes will be designated as the target computing node. This ensures that when idle distributed computing nodes are available, low-cost and flexible distributed computing power will be prioritized for inference tasks. If the evaluation results indicate that distributed computing nodes are unavailable, centralized computing nodes will be designated as the target computing nodes. This ensures that when distributed nodes are unavailable, fluctuating, or preempted, the system will switch to highly reliable centralized computing nodes to execute inference tasks, balancing stability and economy, achieving computing power synergy and complementarity, and effectively reducing the probability of inference service interruptions and inference request failures while keeping costs under control.

[0090] For example, with a preset score of 80 (out of 100), the distributed computing power nodes include schedulable computing power nodes: computing power node A, computing power node B, computing power node C, and computing power node D. If, based on the target parameters of the distributed computing power nodes, the calculation method shown in Formula 1 yields an evaluation score of 60 for computing power node A, 85 for computing power node B, 87 for computing power node C, and 90 for computing power node D, and since there are computing power nodes in the distributed computing power nodes with evaluation scores higher than the preset score, then computing power node D with the highest evaluation score is determined as the target computing power node, and the inference task is executed through computing power node D. If the calculated evaluation scores of computing power nodes A, B, C, and D are all less than 80, then the centralized computing power node is determined as the target computing power node, and the inference task is executed through the enterprise-level inference service of the centralized computing power node.

[0091] It should be noted that the above are examples illustrating the preset score, distributed computing nodes, and the evaluation scores of each distributed computing node, used to describe the process of determining the target computing node based on the evaluation results. This application does not limit the specific value of the preset score, each distributed computing node, or the evaluation score of each distributed computing node.

[0092] Optionally, the parameters of the centralized computing power node are determined, including budget, limit, and current load. The evaluation score of the centralized computing power node is determined based on the parameters of the centralized computing power node. The target computing power node is determined by combining the evaluation scores of the centralized computing power node and the evaluation scores of the distributed computing power node, and at least one distributed computing power node and the computing power node with the highest score among the centralized computing power nodes are determined as the target computing power node.

[0093] S240 executes the inference task corresponding to the inference request through the target computing power node and outputs the inference result.

[0094] Among them, the reasoning result is a streaming output reasoning result. Streaming output reasoning result means that when processing reasoning requests, instead of waiting for all the content to be generated and then returning it all at once, the result is calculated and output on the same time, and the result is pushed to the caller in segments in real time in units of tokens, forming a continuous response stream.

[0095] For example, the inference task corresponding to the inference request is executed by calling the computing resources of the target computing power node and the inference result is streamed. When the target computing power node is a distributed computing power node, the inference task is executed and the streaming incremental content is generated by the idle computing power inference service of the distributed computing power node; when the target computing power node is a centralized computing power node, the inference task is executed and the streaming incremental content is generated by the enterprise-level inference service of the centralized computing power node, and the streaming incremental content is written back to the terminal that triggered the inference request until the end marker is output, thus completing the inference task.

[0096] In one implementation, when the target computing node is a distributed computing node, the processing method further includes: if it is detected that the target computing node is occupied, the lease has expired, or the heartbeat has timed out, controlling the target computing node to stop receiving inference requests; if the target computing node is executing an inference task, determining the reserved computing resources of the target computing node, where the reserved computing resources represent the quota of computing resources reserved for the currently executing inference task before the computing node's resources are reclaimed; and executing the current inference task based on the reserved computing resources.

[0097] For example, when the target computing power node is a distributed computing power node, if it is detected that the target computing power node is occupied, the lease expires, or the heartbeat times out, that is, the target computing power node changes from an available state to an unavailable state, the target computing power node is controlled to stop receiving inference requests, ensuring that it no longer responds to inference requests when the target computing power node is unable to perform inference tasks. If the target computing power node is performing an inference task, the current inference task is executed according to the reserved computing power resources, reducing the probability of incomplete content output when the computing power node is preempted.

[0098] Understandably, due to the poor stability of distributed computing resources, there are risks such as preemption, lease expiration, and abnormal heartbeats when executing inference tasks. Therefore, when the target computing node is a distributed computing node and is unavailable, inference requests are no longer responded to, reducing resource waste caused by invalid routing and repeated retries, and improving overall availability and tail latency performance. Furthermore, if distributed computing nodes can be preempted, direct terminal streaming output may result in incomplete output semantics and trigger centralized retries, causing cascading failures. Therefore, this application reduces the probability of incomplete semantics and retry storms by reserving computing resources to continue executing the current inference task.

[0099] Specifically, the current inference task is executed based on the reserved computing power resources, including: determining the completion predicate corresponding to the streaming protocol type of the current session, the completion predicate taking the streaming output event sequence as input and outputting a completion flag; and executing the inference task through the reserved computing power resources of the target computing power node when the reserved computing power resources are not exhausted and the completion predicate does not output a completion flag.

[0100] For example, the streaming protocol type corresponding to the current session is determined according to the streaming protocol identification rules, and a completion predicate corresponding to the streaming protocol is selected from the completion predicate library. The completion predicate takes the streaming output event sequence as input and the completion flag as output; wherein, the completion flag is determined by the end event or the end field satisfying the preset rules.

[0101] Among them, the streaming protocol identification rule is a judgment rule preset by the computing power scheduling platform to identify the type of streaming protocol currently used. The streaming protocol type is the transmission protocol type of the streaming output used by the current inference session. The completion predicate library is a set of end judgment rules for different streaming protocol types that are pre-maintained by the computing power scheduling platform. Each protocol type corresponds to at least one judgment rule. The completion predicate is a function or rule for judging the end of a streaming output. The input is a sequence of message events continuously generated during the streaming inference process, and the output is a Boolean completion flag used to determine whether the streaming output has ended. The streaming output event sequence is an ordered data stream composed of streaming messages returned by the computing power scheduling node one token at a time during the inference process. The end event or end field is a specific event type, message identifier, status field, or termination flag agreed upon in the streaming protocol to indicate the end of inference.

[0102] Understandably, this application supports unified access and unified termination determination for multiple types of streaming protocols, thus improving the system compatibility of the computing power scheduling platform; by completing the configuration of the predicate to realize the streaming termination logic, it can accurately locate the true termination point of streaming inference, avoid misjudgment as interruption, avoid unnecessary retries and retransmissions, and provide a unified and reliable termination basis for preemption, semantic integrity determination, session reclamation, and resource release.

[0103] For example, when the reserved computing power resources are not exhausted and the completed predicate has not output a completion flag, the computing power scheduling platform allows the target computing power node to continue to output streaming incremental results. That is, it executes inference tasks through the reserved computing power resources of the target computing power node, and outputs a unified end event to the client and submits billing settlement when the completion flag is set.

[0104] In the embodiments of this application, the completion predicate corresponding to the elapsed protocol type of the current session is determined. When the reserved computing power resources are not exhausted and no completion flag is output, the inference task continues to be executed through the reserved computing power of the target computing power node, ensuring that the computing power scheduling platform can do its best to complete the in-transit session and output a unified end event within the budget window, thereby reducing the probability of incomplete content output.

[0105] Optionally, when the reserved computing power resources are exhausted and the completed predicate does not output a completion flag, the inference task is executed through a centralized computing power node, wherein the number of times the centralized computing power node executes the inference task is less than a preset number; if the number of times the centralized computing power node executes the inference task is greater than the preset number and the completed predicate does not output a completion flag, the inference task is determined to have failed.

[0106] For example, when the computing power scheduling platform exhausts the budget and the completion flag is not set, it outputs a controlled degradation termination event to the client and transfers the session to a fallback path so that the enterprise-grade inference service of the centralized computing power node can continue to execute the inference task. When executing the inference task through the centralized computing power node, the number of times the centralized computing power node re-executes the inference task is limited to less than a preset number (e.g., 3 times).

[0107] In the embodiments of this application, when reserved computing power resources are exhausted and the completed predicate does not output a completion flag, the inference task is executed by a centralized computing power node, and the number of times the centralized computing power node executes the inference task is limited to less than a preset number. This ensures that when the distributed computing power nodes are unable to execute the inference task, the centralized computing power node executes the inference task, reducing the risk of inference task execution failure. Furthermore, by limiting the number of times the centralized computing power node executes the inference task, the probability of a retry storm occurring on the computing power scheduling platform is reduced, ensuring the system stability of the schedulable computing power platform.

[0108] In the above embodiments, the evaluation results of the distributed computing power nodes are obtained based on the target parameters of the distributed computing power nodes. Based on the evaluation results, target computing power nodes are determined between the distributed and centralized computing power nodes, and the inference task corresponding to the inference request is executed through the target computing power nodes. Compared to related technologies that use a single type of computing power node, such as only using distributed or only using centralized computing power nodes, which leads to unstable inference services or high inference costs, this solution can select a suitable target computing power node from distributed and centralized computing power nodes based on the evaluation results. By evaluating the availability of distributed computing power nodes in real time, the status, online fluctuations, and resource preemption risks of distributed computing power nodes can be dynamically perceived, ensuring that suitable target computing power nodes can be dynamically scheduled to execute inference tasks. This avoids inference service anomalies caused by using a single type of node, and enhances the stability and fault tolerance of the computing power scheduling platform when executing inference services while keeping costs under control, thereby effectively improving the success rate of inference task execution.

[0109] Figure 3 This is a schematic flowchart illustrating another reasoning request processing method provided in the embodiments of this application.

[0110] like Figure 3 As shown, the reasoning request processing method 300 includes S301 to S314. The reasoning request processing methods shown in S301 to S314 will be described in detail below.

[0111] S301, in response to the inference request, validate the inference request.

[0112] For example, the verification of inference requests includes at least one of signature verification, token verification, authentication, and tenant identification to ensure access security to the computing power scheduling platform and prevent unauthorized inference requests from entering the computing power scheduling process.

[0113] S302, If the inference request verification passes, parse the parameters of the inference request and generate a standard request.

[0114] For example, if the inference request passes the verification, the inference request is parsed to extract key inference parameters such as model name, maximum number of output tokens, and streaming tags contained in the inference request; based on the parsed parameters, the heterogeneous external request is converted into a unified standard request within the system for subsequent computing power scheduling, node allocation and inference execution.

[0115] S303 sends the standard request to the perception scheduling module.

[0116] For example, a standard request is sent to the perception scheduling module so that the perception scheduling module can route and manage the standard request.

[0117] S304 uses the perception and scheduling module to determine the distributed computing power nodes and centralized computing power nodes connected to the computing power scheduling platform.

[0118] For example, the perception and scheduling module identifies at least one distributed computing power node and one centralized computing power node connected to the computing power scheduling platform. The computing power scheduling platform executes inference tasks by scheduling distributed idle computing power resources on the distributed computing power nodes, or manages and processes inference tasks uniformly by scheduling centralized computing power nodes.

[0119] Alternatively, the implementation methods of S301 to S304 can be found in [reference needed]. Figure 2 The relevant description of S210 will not be repeated here.

[0120] S305, Does the distributed computing power node have a schedulable computing power node? If yes, proceed to S306; if no, proceed to S311.

[0121] For example, determine whether there are schedulable computing nodes among the distributed computing power nodes; if there are schedulable computing power nodes among the distributed computing power nodes, determine the target parameters of the distributed computing power nodes; if there are no schedulable computing power nodes among the distributed computing power nodes, determine that the distributed computing power nodes are unavailable.

[0122] For example, the lease status and heartbeat information of distributed computing power nodes determine whether there are schedulable computing power nodes among the distributed computing power nodes. If there is a node in the distributed computing power nodes whose lease status indicates that the lease is valid and whose heartbeat information indicates that the heartbeat has not timed out, it is determined that there are schedulable computing power nodes among the distributed computing power nodes; if the lease status of the distributed computing power node indicates that the lease is invalid or the heartbeat information indicates that the heartbeat has timed out, it is determined that there are no schedulable computing power nodes among the distributed computing power nodes.

[0123] S306, Determine the target parameters of the distributed computing nodes.

[0124] For example, if there are schedulable computing nodes among the distributed computing power nodes, it means that there are computing power nodes among the distributed computing power nodes that the computing power scheduling platform can schedule. The target parameters of the distributed computing power nodes are determined, and the target parameters include at least one of the following: the performance score of the distributed computing power node, the risk assessment level, and the caching benefit.

[0125] S307, calculate the evaluation score of the distributed computing nodes based on the target parameters.

[0126] For example, the evaluation score of the distributed computing node is calculated based on the target parameters. The evaluation score is positively correlated with the performance score, negatively correlated with the risk assessment level, and positively correlated with the caching benefit.

[0127] S308, Check if the evaluation score of the distributed computing node is greater than the preset score; if yes, execute S309 and S310; if no, execute S311 and S312.

[0128] For example, it is determined whether the evaluation score of the distributed computing power node is greater than the preset score; if the evaluation score of the distributed computing power node is greater than the preset score, the distributed computing power node is determined to be available, and the distributed computing power node with the highest evaluation score is determined to be the target computing power node; if the evaluation score of the distributed computing power node is less than or equal to the preset score, the distributed computing power node is determined to be unavailable, and the centralized computing power node is determined to be the target computing power node.

[0129] S309, confirming the availability of distributed computing nodes.

[0130] S310 identifies the distributed computing node with the highest evaluation score as the target computing node.

[0131] For example, when the evaluation score of a distributed computing node is greater than a preset score, it indicates that the distributed computing node is not easily preempted, and the distributed computing node is determined to be available. The distributed computing node with the highest evaluation score is determined as the target computing node, ensuring that when the distributed computing node is available, low-cost and flexible distributed computing power is used to undertake inference tasks.

[0132] S311 indicates that the distributed computing node is unavailable.

[0133] S312, the centralized computing power node is identified as the target computing power node.

[0134] For example, when the evaluation score of a distributed computing node is less than or equal to a preset score, it indicates that the distributed computing node is easily preempted, and the distributed computing node is determined to be unavailable. A centralized computing node is then designated as the target computing node. This ensures that when a distributed node is unavailable, fluctuates, or is preempted, the task is switched to a highly reliable centralized computing node to execute the inference task, balancing stability and economy.

[0135] Alternatively, the implementation methods of S305 to S312 can be found in [reference needed]. Figure 2 The relevant descriptions of S220 and S230 will not be repeated here.

[0136] S313 executes inference tasks through the target computing power node.

[0137] For example, the inference task corresponding to the inference request is executed by calling the computing resources of the target computing power node. When the target computing power node is a distributed computing power node, the inference task is executed through the idle computing power inference service of the distributed computing power node; when the target computing power node is a centralized computing power node, the inference task is executed through the enterprise-level inference service of the centralized computing power node.

[0138] S314, output the reasoning result.

[0139] For example, if the inference result is a streaming output, the perception scheduling module in the computing power scheduling platform will write back the streaming incremental content to the terminal that triggered the inference request until the output end marker is reached, thus completing the inference task.

[0140] Alternatively, the implementation methods of S313 to S314 can be found in [reference needed]. Figure 2 The relevant descriptions of S240 will not be repeated here.

[0141] Figure 4 This is a schematic flowchart illustrating another reasoning request processing method provided in the embodiments of this application.

[0142] like Figure 4 As shown, the reasoning request processing method 400 includes S401 to S410. The reasoning request processing methods shown in S401 to S410 will be described in detail below.

[0143] S401, During the execution of inference tasks by distributed computing nodes, a failure of the distributed computing node was detected.

[0144] For example, during the execution of inference tasks by distributed computing nodes, if it is detected that the distributed computing node is occupied, the lease has expired, or the heartbeat has timed out, it is determined that the distributed computing node has failed; if no such abnormality is detected, it is determined that the distributed computing node has not failed.

[0145] S402, determine the reserved computing resources of the distributed computing nodes and the completion predicate of the current session.

[0146] For example, reserved computing resources represent the quota of computing resources reserved for the inference task being executed before the computing node resources are reclaimed; a completion predicate is a function or rule for determining the end of a streaming output. The input is a sequence of message events continuously generated during the streaming inference process, and the output is a Boolean completion flag used to determine whether the streaming output has ended.

[0147] S403: Should the completion predicate output a completion flag? If yes, execute S409; otherwise, execute S404.

[0148] For example, determine whether the completion predicate outputs a completion flag; if the completion predicate outputs a completion flag, determine that the reasoning task is completed and output the reasoning result; if the completion predicate does not output a completion flag, further determine whether the reserved computing power resources have been exhausted.

[0149] S404, Has the reserved computing power been exhausted? If yes, proceed to S406; if no, proceed to S405.

[0150] For example, it is determined whether the reserved computing power resources have been exhausted; if the reserved computing power resources have been exhausted, the inference task is executed through the centralized computing power node; if the reserved computing power resources have not been exhausted, the inference task is executed through the reserved computing power resources of the distributed computing power node.

[0151] S405 executes inference tasks using the reserved computing resources of distributed computing nodes.

[0152] For example, if the reserved computing power resources are not exhausted, it means that the current distributed computing power node can continue to execute the inference task and output the streaming incremental results. Therefore, the inference task can continue to be executed through the distributed computing power node, thereby reducing the probability of incomplete content output.

[0153] S406 performs inference tasks through centralized computing nodes.

[0154] For example, if the reserved computing power resources are exhausted, it means that the current distributed computing power nodes cannot continue to execute the inference task. Therefore, the inference task is executed through the centralized computing power nodes.

[0155] For example, when executing an inference task through a centralized computing node, if the centralized computing node can obtain the key-value cache of the executed portion of the inference task, it continues to execute the remaining portion of the inference task; if the centralized computing node cannot obtain the key-value cache of the executed portion of the inference task, it re-executes the complete inference task. This ensures that unnecessary computing power consumption from repeated inference is reduced.

[0156] S407: Should the completion predicate output a completion flag? If yes, execute S409; otherwise, execute S408.

[0157] For example, determine whether the completion predicate outputs a completion flag; if the completion predicate outputs a completion flag, determine that the reasoning task is completed and output the reasoning result; if the completion predicate does not output a completion flag, further determine whether the number of times the reasoning task has been executed is greater than the preset number.

[0158] S408, Is the number of times the reasoning task has been executed greater than the preset number? If yes, execute S410; if no, execute S406.

[0159] For example, it is determined whether the number of times the inference task has been executed is greater than a preset number; if the number of times the inference task has been executed is greater than the preset number, the inference task is determined to have failed; if the number of times the inference task has been executed is less than the preset number, the inference task is re-executed through the centralized computing power node.

[0160] S409, confirming the completion of the reasoning task.

[0161] For example, if the predicate output completion flag is displayed, it indicates that the streaming output has ended and the reasoning task has been completed.

[0162] S410, the reasoning task has failed.

[0163] For example, if the predicate does not output a completion flag and the number of times the centralized computing power node executes the inference task is greater than the preset number, it is determined that the inference task has failed. By limiting the number of times the centralized computing power node executes the inference task, the probability of a retry storm occurring on the computing power scheduling platform is reduced, thus ensuring the system stability of the schedulable computing power platform.

[0164] In the embodiments of this application, a completion predicate library is introduced to uniformly determine the termination semantics of different streaming protocols. When preemption occurs, reserved computing resources constrain the emptying process, enabling the system to complete in-transit sessions and output a unified termination event within the budget window, thereby reducing the probability of incomplete content output and retry storms. Furthermore, this invention automatically rolls back to a centralized computing node to execute inference tasks when a distributed computing node fails. This achieves systematic governance of distributed computing power fluctuations and preemption interruptions while maintaining controllable costs, and improves the continuity, stability, and throughput efficiency of the computing power scheduling platform when executing inference tasks.

[0165] Thus, this invention achieves systematic management of idle computing power fluctuations and preemption interruptions under the premise of controllable cost, and improves the continuity, stability and throughput efficiency of aggregated services.

[0166] The above text combined Figures 1 to 4 The method for processing inference requests provided in the embodiments of this application is described in detail below; the following will be combined with Figure 5 and Figure 6 The apparatus embodiments of this application are described in detail below. It should be understood that the apparatus in the embodiments of this application can perform the various methods described in the foregoing embodiments of this application, that is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0167] Figure 5 This is a schematic diagram of the structure of a reasoning request processing device provided in an embodiment of this application.

[0168] For example, such as Figure 5 As shown, the inference request processing device 500 includes: The detection module 510 is used to detect distributed computing nodes and centralized computing nodes connected to the computing power scheduling platform in response to the inference request. The inference request is used to request the computing power scheduling platform to execute the inference task. The processing module 520 is used to, if there are schedulable computing nodes among the distributed computing nodes, obtain the evaluation result of the distributed computing nodes based on the target parameters of the distributed computing nodes, and the evaluation result is used to indicate whether the distributed computing nodes are available; based on the evaluation result of the distributed computing nodes, obtain the target computing node among the distributed computing nodes and the centralized computing nodes; execute the inference task corresponding to the inference request through the target computing node, and output the inference result.

[0169] Optionally, as an embodiment, the processing module 520 is specifically used to: determine the evaluation score of the distributed computing power node based on the target parameters of the distributed computing power node; if there is a computing power node among the distributed computing power nodes with an evaluation score greater than a preset score, determine that the evaluation result indicates that the distributed computing power node is available; if there is no computing power node among the distributed computing power nodes with an evaluation score greater than a preset score, determine that the evaluation result indicates that the distributed computing power node is unavailable.

[0170] Optionally, as an embodiment, it also includes an acquisition module, which is used to: acquire target parameters, the target parameters including at least one of the performance score of the distributed computing power node, the risk assessment level and the cache benefit, the cache benefit being used to represent the performance benefit of reusing cached content for inference operations; The processing module 520 is specifically used to: obtain an evaluation score based on at least one of performance score, risk assessment level and caching benefit, wherein the evaluation score is positively correlated with performance score, negatively correlated with risk assessment level and positively correlated with caching benefit.

[0171] Optionally, as an embodiment, the acquisition module is specifically used to: determine a performance score based on the performance parameters of the distributed computing power nodes, including the throughput, latency, and token generation rate of the distributed computing power nodes; determine a risk assessment level based on the heartbeat, remaining lease time, historical preemption frequency, and system prompt signals of the distributed computing power nodes; and if the inference request carries a handle and the handle is valid, determine the cache benefit based on the first performance indicator when the cache is hit and the second performance indicator when the cache is missed.

[0172] Optionally, as an embodiment, the processing module 520 is specifically used to: if the evaluation result indicates that the distributed computing power node is available, determine the computing power node with the highest evaluation score among the distributed computing power nodes as the target computing power node; if the evaluation result indicates that the distributed computing power node is unavailable, determine the centralized computing power node as the target computing power node.

[0173] Optionally, as an embodiment, the processing module 520 is specifically used to: determine the lease status and heartbeat information of the distributed computing power nodes; if there is a node in the distributed computing power nodes whose lease status indicates that the lease is valid and whose heartbeat information indicates that the heartbeat has not timed out, determine that there is a schedulable computing power node in the distributed computing power nodes; if the lease status of the distributed computing power nodes indicates that the lease is invalid or the heartbeat information indicates that the heartbeat has timed out, determine that there is no schedulable computing power node in the distributed computing power nodes.

[0174] Optionally, as an embodiment, the processing module 520 is specifically used to: if it is detected that the target computing power node is occupied, the lease has expired, or the heartbeat has timed out, control the target computing power node to stop receiving inference requests; if the target computing power node is executing an inference task, determine the reserved computing power resources of the target computing power node, wherein the reserved computing power resources represent the quota of computing power resources reserved for the inference task being executed before the computing power node's resources are reclaimed; and execute the current inference task based on the reserved computing power resources.

[0175] Optionally, as an embodiment, the processing module 520 is specifically used to: determine the completion predicate corresponding to the streaming protocol type of the current session, the completion predicate takes the streaming output event sequence as input and outputs a completion flag; when the reserved computing power resources are not exhausted and the completion predicate does not output a completion flag, the inference task is executed through the reserved computing power resources of the target computing power node.

[0176] Optionally, as an embodiment, the processing module 520 is further configured to: when the reserved computing power resources are exhausted and the completed predicate has not output a completion flag, execute the inference task through the centralized computing power node, wherein the number of times the centralized computing power node executes the inference task is less than a preset number; if the number of times the centralized computing power node executes the inference task is greater than the preset number and the completed predicate has not output a completion flag, determine that the inference task execution has failed.

[0177] It should be noted that the processing device for the aforementioned reasoning request is embodied in the form of a functional unit. The term "module" here can be implemented in software and / or hardware, without specific limitations.

[0178] For example, a "module" can be a software program, hardware circuit, or a combination of both that implements the above functions. Hardware circuits may include application-specific integrated circuits (ASICs), electronic circuits, processors (e.g., shared processors, proprietary processors, or group processors) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functions.

[0179] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0180] Figure 6 This is a schematic diagram of the structure of a computing power scheduling platform provided in an embodiment of this application.

[0181] For example, the computing power scheduling platform 600 includes a processor 610, a memory 620, and executable program code 630.

[0182] For example, the computing power scheduling platform 600 includes one or more processors 610, which can support the processing method for inference requests in the method embodiment of the computing power scheduling platform 600. The processor 610 can be a general-purpose processor or a special-purpose processor. For example, the processor 610 can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0183] For example, the processor 610 can be used to control the computing power scheduling platform 600, execute software programs, and process data from the software programs. The computing power scheduling platform 600 may also include a communication unit for inputting (receiving) and outputting (transmitting) signals.

[0184] For example, the computing power scheduling platform 600 may include one or more memories 620, on which executable program code 630 is stored. The executable program code 630 can be run by the processor 610 to generate instructions, causing the processor 610 to execute the inference request processing method described in the above method embodiment according to the instructions.

[0185] Optionally, the memory 620 may also store data. Optionally, the processor 610 may also read data stored in the memory 620, which may be stored at the same memory address as the executable program code 630, or the data may be stored at a different memory address than the executable program code 630.

[0186] For example, the processor 610 and memory 620 can be configured separately or integrated together, for example, integrated on a system-on-a-chip of the terminal device.

[0187] For example, the memory 620 can be used to store related programs of the inference request processing method provided in the embodiments of this application. The processor 610 can be used to call the executable program code 630 stored in the memory 620 when controlling the computing power scheduling platform to execute the inference request processing method of the embodiments of this application. For example, in response to the inference request, distributed computing power nodes and centralized computing power nodes connected to the computing power scheduling platform are detected. The inference request is used to request the computing power scheduling platform to execute an inference task. If there is a schedulable computing power node among the distributed computing power nodes, the evaluation result of the distributed computing power node is obtained based on the target parameters of the distributed computing power node. The evaluation result is used to indicate whether the distributed computing power node is available. Based on the evaluation result of the distributed computing power node, the target computing power node among the distributed computing power nodes and centralized computing power nodes is obtained. The inference task corresponding to the inference request is executed through the target computing power node, and the inference result is output.

[0188] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the reasoning request processing method of any of the foregoing embodiments.

[0189] The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, Digital Video Discs (DVDs), Compact Disc Read-Only Memory (CD-ROMs), microdrives, and magneto-optical disks, read-only memory (ROMs), random access memory (RAMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), dynamic random access memory (DRAMs), video random access memory (VRAMs), flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0190] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement a reasoning request processing method in the above embodiments.

[0191] The computing power scheduling platform, computer-readable storage medium, computer program product or chip provided in this application are all used to execute the processing method for the corresponding inference request provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the processing method for the corresponding inference request provided above, and will not be repeated here.

[0192] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0193] In the embodiments provided in this application, it should be understood that the disclosed apparatus and processing methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0194] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing inference requests, characterized in that, The processing method is applied to a computing power scheduling platform, and the processing method includes: In response to an inference request, the distributed computing power nodes and centralized computing power nodes connected to the computing power scheduling platform are detected. The inference request is used to request the computing power scheduling platform to execute an inference task. If there are schedulable computing nodes among the distributed computing power nodes, an evaluation result of the distributed computing power node is obtained based on the target parameters of the distributed computing power node, and the evaluation result is used to indicate whether the distributed computing power node is available; Based on the evaluation results of the distributed computing power nodes, the target computing power node among the distributed computing power nodes and the centralized computing power nodes is obtained; The inference task corresponding to the inference request is executed by the target computing node, and the inference result is output.

2. The processing method according to claim 1, characterized in that, The evaluation result of the distributed computing power node based on the target parameters of the distributed computing power node includes: Based on the target parameters of the distributed computing power node, the evaluation score of the distributed computing power node is determined; If there is a computing power node among the distributed computing power nodes whose evaluation score is greater than a preset score, it is determined that the evaluation result indicates that the distributed computing power node is available; If there is no distributed computing power node with an evaluation score greater than a preset score, the evaluation result indicates that the distributed computing power node is unavailable.

3. The processing method according to claim 2, characterized in that, The processing method further includes: Obtain the target parameters, which include at least one of the performance score, risk assessment level, and cache benefit of the distributed computing node, wherein the cache benefit is used to represent the performance benefit of reusing cached content for inference operations; Determining the evaluation score of the distributed computing power node based on the target parameters of the distributed computing power node includes: The evaluation score is obtained based on at least one of the performance score, the risk assessment level, and the caching benefit, wherein the evaluation score is positively correlated with the performance score, negatively correlated with the risk assessment level, and positively correlated with the caching benefit.

4. The processing method according to claim 3, characterized in that, The process of obtaining the target parameter includes: The performance score is determined based on the performance parameters of the distributed computing nodes, including the throughput, latency, and token generation rate of the distributed computing nodes. The risk assessment level is determined based on the heartbeat, remaining lease time, historical preemption frequency, and system prompt signals of the distributed computing power nodes. If the inference request carries a handle and the handle is valid, the cache benefit is determined based on the first performance metric when the cache is hit and the second performance metric when the cache is missed.

5. The processing method according to claim 1, characterized in that, The process of obtaining the target computing node among the distributed computing nodes and the centralized computing nodes based on the evaluation results of the distributed computing nodes includes: If the evaluation result indicates that the distributed computing power node is available, the computing power node with the highest evaluation score among the distributed computing power nodes shall be determined as the target computing power node. If the evaluation result indicates that the distributed computing node is unavailable, the centralized computing node will be identified as the target computing node.

6. The processing method according to claim 1, characterized in that, The processing method further includes: Determine the lease status and heartbeat information of the distributed computing power nodes; If there is a node in the distributed computing power nodes whose lease status indicates that the lease is valid and whose heartbeat information indicates that the heartbeat has not timed out, it is determined that there is a schedulable computing power node in the distributed computing power nodes. If the lease status of the distributed computing power node indicates that the lease has expired or the heartbeat information indicates that the heartbeat has timed out, it is determined that there is no schedulable computing power node among the distributed computing power nodes.

7. The processing method according to any one of claims 1 to 6, characterized in that, When the target computing power node is the distributed computing power node, the processing method further includes: If the target computing power node is detected to be occupied, its lease expires, or its heartbeat times out, the target computing power node is controlled to stop receiving inference requests. If the target computing node is executing an inference task, determine the reserved computing resources of the target computing node. The reserved computing resources represent the computing resource quota reserved for the inference task being executed before the computing node's resources are reclaimed. The current inference task is executed based on the reserved computing resources.

8. The processing method according to claim 7, characterized in that, The execution of the current inference task based on the reserved computing power resources includes: Determine the completion predicate corresponding to the streaming protocol type of the current session. The completion predicate takes the streaming output event sequence as input and outputs a completion flag. When the reserved computing power resources are not exhausted and the completion predicate does not output a completion flag, the inference task is executed using the reserved computing power resources of the target computing power node.

9. The processing method according to claim 8, characterized in that, The processing method further includes: When the reserved computing power resources are exhausted and the completion predicate does not output a completion flag, the inference task is executed through the centralized computing power node, wherein the number of times the centralized computing power node executes the inference task is less than a preset number; If the number of times the centralized computing node executes the inference task exceeds a preset number and the completion predicate does not output a completion flag, the inference task is determined to have failed.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed, implements the processing method according to any one of claims 1 to 9.

Images