Decentralized large model computing power joint scheduling method and system based on local agent

Abstract

This application discloses a decentralized large-scale computing power joint scheduling method and system based on a local proxy. The method includes: decrypting encrypted access credentials stored locally to obtain runtime access credentials that reside only in the runtime memory; obtaining quota status information carried in the large-scale API response header; determining the available quota based on the quota status information and sending resource status information to the scheduler; receiving local requests through a local proxy program and writing them to a local queue; receiving remote requests issued by the scheduler based on the resource status information and writing them to a remote queue; determining the requests to be executed from the local and remote queues, using the runtime access credentials to call the large-scale API to execute the requests, and obtaining the execution result; and sending the execution result to the scheduler or the local requester based on the source of the requests. This application helps to improve the overall utilization rate of resources called by the large-scale API while ensuring the priority of local request processing.

Description

Technical Field

[0001] This application relates to the fields of computer networks, distributed task scheduling and large model application technology, and in particular to a decentralized large model computing power joint scheduling method and system based on local agents. Background Technology

[0002] With the increasing popularity of large model services, development tools, continuous integration pipelines, and batch processing programs can perform code generation, text processing, and inference tasks through the large model application programming interface (API). Large model APIs typically have call limits set based on the number of requests and token processing volume per unit of time, and the usage status of these limits varies significantly among different accounts.

[0003] Existing solutions typically employ a centralized API gateway, which centrally stores access credentials for multiple accounts and routes requests to different accounts using methods such as round-robin, least-use, or weighted allocation. This approach creates a unified entry point, but requires aggregating multiple access credentials at the gateway.

[0004] Centralized gateways simultaneously handle credential storage and task routing. If the gateway is attacked or unauthorized access occurs, multiple access credentials may be leaked at the same time. Therefore, how to achieve joint scheduling of large-scale model API access credentials among multiple nodes, while avoiding centralized collection and use of such credentials, becomes a technical problem that needs to be solved. Summary of the Invention

[0005] This application proposes a decentralized large-scale model computing power joint scheduling method and system based on local agents, aiming to enable the scheduler to perceive the available call quota of each node and realize cross-node request scheduling and distributed execution when the large-scale model API access credentials are limited to local agent nodes.

[0006] In a first aspect, embodiments of this application provide a decentralized large-scale model computing power joint scheduling method based on a local proxy, applied to a local proxy node. The method includes: decrypting locally stored encrypted access credentials to obtain runtime access credentials that reside only in the running memory; obtaining quota status information carried in the large-scale model API response header; determining the available quota based on the quota status information, and sending resource status information to the scheduler, wherein the resource status information includes the available quota but does not include the encrypted access credentials and the runtime access credentials; receiving local requests through a local proxy program and writing them to a local queue; receiving remote requests issued by the scheduler based on the resource status information and writing them to a remote queue; determining requests to be executed from the local queue and the remote queue based on the queue priority of the local queue being higher than that of the remote queue, using the runtime access credentials to call the large-scale model API to execute the requests to be executed, and obtaining an execution result; and sending the execution result to the scheduler or the local requester based on the source of the requests to be executed.

[0007] As can be seen, in this embodiment, the local proxy node first generates runtime access credentials in its running memory and obtains quota status information. Next, it determines the available quota and sends resource status information without access credentials. Then, it receives local and remote requests respectively, executes the requests according to the queue priority (local queue is higher than remote queue), and finally outputs the execution result according to the request source. Since the scheduler only receives resource status information, and access credentials and actual API calls are limited to the local proxy node, the credential usage process is isolated from the cross-node scheduling process. This reduces the risk of leakage caused by centrally storing multiple access credentials and balances local request processing and sharing of idle call quotas.

[0008] In one possible example, determining the available loanable amount based on the credit limit status information includes: extracting a credit limit dimension field and a credit limit window field from the credit limit status information to obtain a credit limit record to be calibrated; the credit limit dimension field includes a remaining requested credit limit field and a remaining token credit limit field, and the credit limit window field includes a statistical period field and a credit limit reset field; performing time window calibration on the credit limit record to be calibrated based on the credit limit window field to obtain a windowed credit limit record; and determining the available loanable amount for the request dimension and the token dimension based on the windowed credit limit record and a preset reservation ratio to obtain the available loanable amount.

[0009] As can be seen in this example, by converting the quota field in the response header into a quota record with a unified quota dimension and statistical window, and then reserving a portion of the quota to form a loanable quota, the quota judgment deviation caused by the mixing of different quota dimensions or different statistical windows can be reduced, and resource data that can be directly used for task matching can be provided to the scheduler.

[0010] In one possible example, the step of performing time window calibration on the credit limit record to be calibrated according to the credit limit window field to obtain a windowed credit limit record includes: determining the credit limit reset time and window duration from the credit limit window field to obtain window baseline information; filtering credit limit records within the current credit limit statistics window from the credit limit record to be calibrated according to the window baseline information to obtain the current window credit limit record; and generating the windowed credit limit record based on the latest credit limit dimension fields collected at the current window credit limit record.

[0011] As can be seen in this example, by first determining the window baseline information, then filtering the quota records in the current quota statistics window, and finally selecting the latest quota dimension field at the time of collection, the remaining quota in the historical statistics window can be avoided from being incorrectly used for the current scheduling, thus improving the timeliness of the quota status.

[0012] In one possible example, the step of filtering the quota records within the current quota statistics window from the quota records to be calibrated according to the window reference information to obtain the current window quota record includes: reading the collection time of each quota record in the quota records to be calibrated to obtain a collection time sequence; determining the window affiliation of each collection time in the collection time sequence according to the window reference information to obtain a window affiliation marker; and retaining the quota records belonging to the current quota statistics window from the quota records to be calibrated according to the window affiliation marker to obtain the current window quota record.

[0013] As can be seen in this example, by collecting time series and window attribution markers to establish the correspondence between quota records and quota statistics windows, the source of the current window quota record can be traced and verified, which is beneficial for excluding expired quota records.

[0014] In one possible example, the step of determining the window affiliation of each collection time in the collection time sequence based on the window reference information to obtain a window affiliation mark includes: extracting the start time and the quota reset time of the current quota statistics window from the window reference information to obtain window boundary information; comparing each collection time in the collection time sequence with the window boundary information to obtain a time comparison result; marking the collection time indicated by the time comparison result as belonging to the current quota statistics window, and marking the remaining collection times as not belonging to the current quota statistics window, to obtain the window affiliation mark.

[0015] As can be seen, in this example, by clearly defining the start time and the time of the current quota statistics window, and by determining the window affiliation for each collection time, a window affiliation mark with clear boundaries can be formed, reducing confusion between quota records of adjacent statistics windows.

[0016] In one possible example, extracting the start time and the reset time of the current quota statistics window from the window baseline information to obtain window boundary information includes: reading the reset time and the duration of the window from the window baseline information to obtain window boundary parameters; shifting the reset time forward by the duration of the window based on the window boundary parameters to obtain the start time; and associating the start time and the reset time to obtain the window boundary information.

[0017] As can be seen in this example, when the response information provides the time of the quota reset and the duration of the window, the start time of the current quota statistics window can be restored and complete window boundary information can be formed, thus providing a unified time benchmark for determining the window ownership of quota records.

[0018] In one possible example, after determining the request to be executed from the remote queue, the method further includes: obtaining the subsequent demand quota and task progress information of the remote task corresponding to the request to be executed, and monitoring the queue status of the local queue; determining the remote execution status based on the queue status, the available quota, and the subsequent demand quota; when the remote execution status indicates that there are pending local requests in the local queue, or the available quota is less than the subsequent demand quota, stopping the determination of new requests to be executed from the remote queue after the current request to be executed has been completed; generating rescheduling information based on the remote execution status and the task progress information, and sending the rescheduling information to the scheduler so that the scheduler schedules the unexecuted requests in the remote task.

[0019] As can be seen in this example, when a local request arrives or the available credit is insufficient to support the continued execution of a remote task, the local proxy node can stop fetching new remote requests after the current request is completed, and hand over the unexecuted requests to the scheduler for rescheduling. This improves the coordination between local requests and remote tasks while avoiding interruption of the current API call.

[0020] In one possible example, generating rescheduling information based on the remote execution status and the task progress information includes: extracting the remote task identifier and unexecuted request from the task progress information to obtain task information to be migrated; determining the quota constraint information corresponding to the task information to be migrated based on the remote execution status; and associating the task information to be migrated with the quota constraint information to generate the rescheduling information.

[0021] As can be seen in this example, by associating remote task identifiers, unexecuted requests, and quota constraint information with the rescheduling information, the scheduler can identify the scope of tasks that need to be migrated and the resources required for subsequent execution, thereby reducing the possibility of completed requests being executed repeatedly.

[0022] In one possible example, after sending the resource status information to the scheduler, the method further includes: receiving a contribution control instruction generated by a local control interface; the contribution control instruction includes a pause instruction, a sharing ratio adjustment instruction, or an exit instruction; updating the contribution status and the available lending quota according to the contribution control instruction to obtain node status information; and sending the node status information to the scheduler so that the scheduler stops issuing remote requests when the contribution status is unavailable, or issues remote requests according to the updated available lending quota.

[0023] As can be seen in this example, developers can change the contribution status or sharing ratio of the local proxy node through the local control interface, and enable the scheduler to adjust the distribution behavior of remote requests in a timely manner, thereby achieving dynamic control over the contribution scope of the local proxy node to the resources it calls.

[0024] Secondly, embodiments of this application provide a decentralized large-scale model computing power joint scheduling system based on local agents. The system includes a scheduler and multiple local agent nodes. Each local agent node includes: a credential and information acquisition module, used to decrypt the encrypted access credential stored in the local agent node to obtain a runtime access credential that resides only in the running memory, and to acquire the quota status information carried in the large-scale model API response header; a quota processing module, used to determine the available quota based on the quota status information, and send resource status information to the scheduler, wherein the resource status information includes the available quota but does not include the encrypted access credential and the runtime access credential; and a queue management module, used to receive local requests through the local agent program and write them to the queue. A local queue receives remote requests issued by the scheduler based on the resource status information and writes them into a remote queue. A request execution module is used to determine the requests to be executed from the local queue and the remote queue based on the queue priority of the local queue being higher than that of the remote queue, and to execute the requests by calling the large model API using the runtime access credentials to obtain the execution results. A result output module is used to send the execution results to the scheduler or the local requester based on the source of the requests to be executed. The scheduler is used to select a target local proxy node based on the resource status information of each local proxy node, issue the remote request to the target local proxy node, and forward the execution results corresponding to the remote request to the task client.

[0025] As can be seen, the system provided in this embodiment stores and uses access credentials separately through multiple local proxy nodes, and the scheduler selects the target local proxy node based on resource status information without access credentials. This allows for the formation of a globally available resource view without centralized storage of access credentials. Each functional module sequentially transmits quota status information, resource status information, queue requests, and execution results, ensuring a complete data processing chain for the joint scheduling process.

[0026] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, a communication interface, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, the programs including instructions for performing the steps in the first aspect of embodiments of this application.

[0027] Fourthly, embodiments of this application provide a computer storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of this embodiment.

[0028] As can be seen, in this embodiment of the application, compared to the joint scheduling scheme where a centralized API gateway centrally stores and invokes multiple access credentials, this application does not require the scheduler to centrally hold the large model API access credentials. The local proxy node first decrypts the encrypted access credentials to obtain runtime access credentials that reside only in the runtime memory, and obtains the quota status information carried in the large model API response header; secondly, it determines the available quota based on the quota status information and sends resource status information without access credentials to the scheduler; then, it receives local requests and writes them to the local queue through the local proxy program, and receives remote requests issued by the scheduler and writes them to the remote queue; thirdly, it determines the requests to be executed based on the queue priority of the local queue (which is higher than the remote queue), and uses the runtime access credentials to invoke the large model API to execute the requests; finally, it sends the execution result to the scheduler or the local requester based on the source of the requests to be executed. Since the scheduler only receives resource status information and completes task matching, while access credentials and actual API calls are limited to local proxy nodes, the computing power scheduling process and the credential usage process are isolated from each other, thereby reducing the risk of batch leakage caused by centralized credential storage; at the same time, the use of real-time quota status and dual queue priority to complete the dynamic sharing of idle call quota is conducive to improving the overall utilization rate of large model API call resources while ensuring the priority of local request processing. Attached Figure Description

[0029] Figure 1 A schematic diagram of the hardware architecture of the decentralized large-model computing power joint scheduling system provided in the embodiments of this application; Figure 2A schematic diagram of the software architecture of the local proxy node provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating a scenario for joint scheduling of local development and batch inference provided in an embodiment of this application. Figure 4 A flowchart illustrating the decentralized large-scale computing power joint scheduling method based on local agents provided in this application embodiment; Figure 5 A flowchart illustrating the process of determining the available lending limit as provided in this application embodiment; Figure 6 A flowchart illustrating the quota recording time window calibration process provided in this application embodiment; Figure 7 A flowchart illustrating the current window quota record filtering process provided in this application embodiment; Figure 8 A flowchart illustrating the process of determining the attribution of the data acquisition time window provided in this application embodiment; Figure 9 A flowchart illustrating the process of determining window boundary information provided in this application embodiment; Figure 10 A flowchart illustrating the remote task pause and rescheduling process provided in this application embodiment; Figure 11 A flowchart illustrating the rescheduling information generation process provided in this application embodiment; Figure 12 A flowchart illustrating the local resource contribution control process provided in this application embodiment; Figure 13 A schematic diagram of the functional modules of the decentralized large-scale model computing power joint scheduling system provided in the embodiments of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only used to explain this application and are not intended to limit the scope of protection of this application. Where there is no conflict, the various embodiments and their technical features can be combined with each other.

[0031] This embodiment provides a unified explanation of features such as large model computing power, local proxy nodes, access credentials, quota status information, available quota, and decentralized joint scheduling.

[0032] The large model computing power referred to in this application refers to the callable inference resources provided through the large model API. These inference resources can be characterized by the number of requests allowed per unit time, the number of tokens allowed to be processed per unit time, or other rate-limited dimensions. Large model computing power does not require the local proxy node to directly provide graphics processor computing resources; it can also be represented by the quota of large model API calls that the local proxy node is authorized to use.

[0033] Local proxy nodes can be developer computers, workstations, edge servers, or other electronic devices capable of running local proxy programs. The local proxy program can run as a daemon process, providing a proxy interface on the local listening address and port. Development tools only need to adjust the original large model API access address to the local proxy address to handle request forwarding through the local proxy program.

[0034] Encrypted access credentials are large-scale API keys, access tokens, or other authentication data stored in encrypted form in the local file system, operating system credential store, or hardware security unit. Runtime access credentials are the credential data obtained by decrypting the encrypted access credentials after the local agent program starts. Runtime access credentials are only stored in the runtime memory of the local agent program; they are not written to plaintext files, nor are they included in the resource status information sent to the scheduler.

[0035] Credit status information reflects the API call limits and remaining call capacity of large models. Credit status information may include account or model identifier, remaining request credit, remaining token credit, statistical period, credit reset time, information collection time, and current consumption. Different large model service providers may use different response header field names; the local proxy program can convert these to uniform credit status information through field mapping configuration.

[0036] The available lending quota is the amount of call quota that the local proxy node allows the scheduler to allocate for remote requests after reserving local call resources. The available lending quota can include both request-level and token-level available lending quotas simultaneously, rather than being compressed into a single scalar. When selecting a target local proxy node for a remote task, the scheduler can simultaneously determine whether both quota dimensions meet the remote task's requirements.

[0037] Local requests are generated by the integrated development environment, code assistance tools, or local applications on the device where the local agent node resides. Remote requests are submitted by the task client and dispatched to the local agent node by the scheduler. A remote task can include a single remote request or multiple remote requests organized in execution order.

[0038] The local queue is used to store local requests, while the remote queue is used to store remote requests. The local queue has a higher priority than the remote queue. This means that when the local agent needs to select the next request to be executed, if there is a pending local request in the local queue, it will be retrieved first; otherwise, it will be retrieved from the remote queue. This priority does not require the forced interruption of ongoing large model API calls.

[0039] The decentralization referred to in this application means that the storage, decryption, and actual API calls of access credentials are distributed across various local proxy nodes. The scheduler can act as a resource matching node that does not handle access credentials, used to maintain resource status information and issue remote requests. Therefore, the decentralization in this application does not preclude the establishment of a unified scheduler.

[0040] See Figure 1 The decentralized large-scale model computing power joint scheduling system 100 may include a local agent node group 110, a scheduler 120, a task client 130, a large-scale model API service device 140, a local request device 150, and a communication network 160. The local agent node group 110 may include a first local agent node 111, a second local agent node 112, and an Nth local agent node 113; each local agent node may include a processor, memory, a communication interface, and a local credential memory. The scheduler 120 may include a scheduling processor 121, a scheduling memory 122, a scheduling communication interface 123, and a resource view memory 124.

[0041] The processor runs the local agent program, performing tasks such as credential decryption, response header parsing, quota processing, request enqueueing, queue selection, large model API calls, and result output. The memory stores program instructions, local queues, remote queues, quota records, task progress information, and node status information. The local credential memory stores encrypted access credentials. Decrypted runtime access credentials are only entered into the processor's corresponding runtime memory and are not written back to the local credential memory.

[0042] The communication interface is used for data interaction with the scheduler 120 and the large-scale API service device 140 via the communication network 160. A two-way authenticated secure connection can be established between the local agent node group 110 and the scheduler 120, such as a connection based on a two-way transport layer security protocol. Data transmitted through this connection may include heartbeat information, resource status information, remote requests, rescheduling information, and execution results, but does not include encrypted access credentials or runtime access credentials.

[0043] The local request device 150 can be an integrated development environment located on the same device as the first local agent node 111, or it can be a development terminal that accesses the first local agent node 111 via a local network. The local request device 150 submits a local request to the local agent program. The local agent program uses runtime access credentials to access the large model API service device 140 and returns the execution result corresponding to the local request to the local request device 150.

[0044] Task client 130 can be a continuous integration or continuous deployment pipeline, batch inference program, data processing script, or other large model task initiation device. Task client 130 submits remote tasks to scheduler 120. Scheduler processor 121 reads the resource status information of each node in resource view memory 124, selects the target local agent node, and sends a remote request through scheduling communication interface 123.

[0045] The resource view storage 124 can store node identifiers, online status, available lending quota, network latency, task occupancy status, and resource status information update time. The scheduler 120 does not store the access credentials of each local agent node. Therefore, each local agent node can control its own access credentials, and the scheduler 120 completes cross-node resource matching.

[0046] See Figure 2 The software architecture 200 of the local agent node may include a credential management module 210, a quota awareness module 220, a resource reporting module 230, a request access module 240, a queue management module 250, a priority scheduling module 260, an API call module 270, a result distribution module 280, and a contribution control module 290.

[0047] The credential management module 210 decrypts encrypted access credentials and maintains runtime access credentials in runtime memory. The quota awareness module 220 parses the large model API response header and determines the available loan quota. The resource reporting module 230 sends resource status information without access credentials to the scheduler. The request access module 240 receives local and remote requests; the queue management module 250 writes the two types of requests to the local queue and the remote queue, respectively. The priority scheduling module 260 determines the requests to be executed according to the priority of the local queue (higher than the remote queue). The API call module 270 executes the requests to be executed using runtime access credentials. The result distribution module 280 sends the execution results according to the request source. The contribution control module 290 updates the contribution status and available loan quota according to local control instructions.

[0048] See Figure 3In a scenario involving joint scheduling of local development and batch inference, the developer terminal 310 runs an integrated development environment 311 and a local agent program 312. The local credential storage area 313 stores the developer's large model API access credentials in encrypted form. The integrated development environment 311 configures the API access address as the listening address of the local agent program 312, which receives code generation, code interpretation, or text processing requests and writes them to a local queue 314.

[0049] The batch task client 320 can generate batch inference tasks 321 during nighttime builds, code reviews, or data processing scenarios. Batch inference tasks 321 are submitted to the scheduler 120. Each node in the contributing node group 340 periodically sends resource status information 360. The scheduler 120 selects nodes with available lending quotas based on the resource status information 360 and splits or organizes the batch inference tasks 321 into remote requests.

[0050] When developer terminal 310 is selected as the target node, local agent 312 writes the remote request to remote queue 315. When local queue 314 is empty, local agent 312 can retrieve the request from remote queue 315 and access large model API service device 140. If integrated development environment 311 generates a new local request, local agent 312 prioritizes processing the request in local queue 314 after the current API call is completed.

[0051] The execution result 370 corresponding to the remote request is returned to the batch task client 320 via the scheduler 120, while the execution result 370 corresponding to the local request is directly returned to the integrated development environment 311. Developers can also pause contributions or reduce the sharing ratio through the local control interface, causing the scheduler 120 to reduce or stop allocating new remote requests to the developer terminal 310.

[0052] See Figure 4 The decentralized large-scale computing power joint scheduling method based on local agents can be implemented through the following steps S101 to S105.

[0053] S101. Decrypt the encrypted access credentials stored locally to obtain runtime access credentials that reside only in the running memory; obtain the quota status information carried in the large model API response header.

[0054] The encrypted access credentials can be stored in the operating system's credential repository, a hardware security unit, or an encrypted file protected by the user's password. Upon startup, the local agent reads the encrypted access credentials, performs decryption after local identity verification, and stores the runtime access credentials in the local agent's runtime memory.

[0055] The local agent can restrict the scope of runtime access credentials, allowing the scheduling thread to access the credentials while the resource reporting thread, logging thread, and scheduler communication thread cannot read the plaintext of the credentials. When the local agent exits, it can release the memory area storing the runtime access credentials.

[0056] Credit limit status information can be obtained from the large model API response corresponding to local or remote requests. The response header can carry the remaining requested credit limit, remaining token credit limit, credit limit statistics period, and credit limit reset time. For local proxy nodes that are starting up for the first time and have not yet obtained the response header, the available credit limit can be initialized to an unavailable state, and then participate in remote task scheduling after obtaining valid credit limit status information.

[0057] S102. Determine the available loan amount based on the loan amount status information, and send resource status information to the scheduler. The resource status information includes the available loan amount but does not include the encrypted access credential and the runtime access credential.

[0058] The available credit limit represents the amount of resources that can be used to execute remote requests after reserving some resources for local calls. Resource status information may also include node identifier, supported model types, network latency, current number of tasks, contribution status, heartbeat time, and health status.

[0059] The local agent can send resource status information periodically, or immediately when the available lending quota, contribution status, or network status changes. Resource status information is sent to the scheduler via a secure, two-way authenticated connection. The resource status information data structure does not include an access credential field, nor does it carry access credentials in logs, error messages, or task return data.

[0060] After receiving resource status information from multiple local agent nodes, the scheduler can build a global view of idle resources based on node identifiers. This resource view reflects the call quota available for each node and does not include authentication data used for actual access to the large model API.

[0061] S103. Receive local requests through the local agent program and write them to the local queue; receive remote requests issued by the scheduler based on the resource status information and write them to the remote queue.

[0062] The local proxy can expose a local proxy interface on the loopback network address. Requests from this interface are configured with a local origin flag and written to a local queue. Requests sent by the scheduler over a secure connection are configured with a remote origin flag and written to a remote queue.

[0063] Request records in the queue can include request identifier, source identifier, model identifier, request body, expected request amount, expected token amount, enqueue time, and task identifier. Remote requests can also be associated with the execution order and task progress information of remote tasks.

[0064] Local queues and remote queues can be two physically isolated queues, or they can be two logical queues formed according to source tags within the same queue storage structure. As long as requests can be selected according to different queue priorities, they can be used as local queues and remote queues in this application.

[0065] S104. Based on the queue priority of the local queue being higher than that of the remote queue, determine the request to be executed from the local queue and the remote queue, and use the runtime access credentials to call the large model API to execute the request to be executed, and obtain the execution result.

[0066] When the local proxy is ready to initiate the next API call, it first checks the local queue. If there are pending local requests in the local queue, the request to be executed is determined from the local queue. If the local queue is empty, the request to be executed can be determined from the remote queue.

[0067] For requests that have already been sent and are awaiting a response from the large model API, this embodiment allows the request to continue execution without forcibly interrupting the network connection. Queue priority is primarily used to determine the next request to be executed. This approach reduces the loss of results or duplicate billing caused by canceling API calls midway.

[0068] The local agent uses runtime access credentials to generate large model API calls and sends the execution request to the corresponding large model service provider. The execution result may include the model output, call status, actual token consumption, and updated rate-limiting response headers.

[0069] S105. Based on the source of the request to be executed, send the execution result to the scheduler or the local requester.

[0070] The local agent reads the source flag of the request to be executed. When the source flag indicates a local request, the execution result is sent to the corresponding local requester; when the source flag indicates a remote request, the execution result, along with the remote task identifier, is sent to the scheduler, which then forwards it to the task client.

[0071] After the execution result is output, the local agent program can also update the execution status of the corresponding request and update the credit limit status information according to the latest response header. The updated credit limit status information can continue to be used for determining the available credit limit in the next cycle, thus forming a cyclical processing link of credit limit awareness, task execution, and credit limit update.

[0072] This embodiment first generates runtime access credentials locally and detects the call limit. Then, it only sends resource status information without access credentials to the scheduler. Next, it uses dual queues to receive requests from different sources and completes the actual API call and result distribution locally. Therefore, the scheduler can achieve cross-node task matching without storing access credentials and can prioritize responding to local requests during remote task sharing.

[0073] In one possible example, see Figure 5 The process of determining the available loan amount in step S102 can be achieved through the following steps S201 to S203.

[0074] S201. Extract the quota dimension field and quota window field from the quota status information to obtain the quota record to be calibrated; the quota dimension field includes the remaining requested quota field and the remaining token quota field, and the quota window field includes the statistical period field and the quota reset field.

[0075] The "Limit Dimension" field describes the number of requests and tokens currently available. The "Limit Window" field describes the statistical period to which the aforementioned quota belongs and the time of the next quota reset. The "Limit Records to be Calibrated" is a collection of quota data that has not yet completed unified time window processing.

[0076] Different large-scale service providers can use different field names in the response header. For example, one service provider might return the remaining request count and the remaining token count separately, while another might use different field prefixes. The local proxy can select the field mapping configuration based on the service provider identifier, converting fields with the same physical meaning into uniform quota dimension fields and quota window fields.

[0077] Each record for the quota to be calibrated may include the service provider identifier, the account local identifier, the model identifier, the remaining request quota, the remaining token quota, the statistical period, the quota reset time, and the collection time. The account local identifier may be an internal identifier generated by the local proxy node and does not contain plaintext access credentials.

[0078] In one possible embodiment of this application, a field configuration library can be set in the local agent program. The field configuration library records the correspondence between the service provider, response header field names, quota dimensions, time units, and reset field types. When a change in a response header field is detected, the local agent program can update the corresponding configuration without altering the subsequent quota processing flow. The specific problem addressed by this embodiment is the inconsistency in the response header structures of different service providers, which can easily lead to misidentification of quota fields; by unifying field mapping, the stability of quota resolution during heterogeneous large-scale model API access is improved.

[0079] S202. Perform time window calibration on the quota record to be calibrated according to the quota window field to obtain a windowed quota record.

[0080] Time window calibration is used to assign credit limit records to their corresponding actual credit limit statistics windows. For response headers that use relative remaining duration to represent the credit limit reset time, the local agent program can convert them to the credit limit reset time based on the response reception time; for response headers that directly give the absolute reset time, they can be uniformly converted to the time base used by the local agent node.

[0081] The windowed credit limit record can save the latest credit limit value of the current statistical window separately according to the requested credit limit dimension and the token credit limit dimension. Records belonging to the historical statistical window are not included in the determination of the current available lending limit.

[0082] In one possible embodiment of this application, the local agent program can simultaneously record the response generation time, the response reception time, and the credit limit reset time. When network transmission latency exists, calibration is performed preferentially using the acquisition time and window boundary under the same time base. The specific problem addressed by this embodiment is that network latency or device clock differences may cause credit limit records near the window boundary to be misclassified; by unifying the time base, the probability of cross-window misclassification can be reduced.

[0083] S203. Based on the windowed quota record and the preset reservation ratio, determine the available quota for the request dimension and the token dimension respectively, and obtain the available quota.

[0084] The preset reservation ratio is used to reserve a portion of the resources for local requests. The preset reservation ratio can be configured by the developer through the local control interface. For example, when the preset reservation ratio is 20%, the remaining quota for the request dimension is 100 requests, and the remaining quota for the token dimension is 50,000 tokens, 20 requests and 10,000 tokens can be reserved, and the remaining portion can be determined as the available quota for the corresponding dimension.

[0085] The available lending limit can be composed of the available lending limit at the request dimension and the available lending limit at the token dimension. When the scheduler issues a remote task, it should ensure that the expected number of requests for the remote task does not exceed the available lending limit at the request dimension and that the expected token consumption does not exceed the available lending limit at the token dimension.

[0086] In one possible embodiment of this application, the preset reservation ratio can be adjusted in stages based on recent changes in local requests. For example, when the frequency of local requests increases, the reservation ratio is increased; when the local queue remains empty, the reservation ratio is decreased within the range allowed by the developer. The specific problem solved by this embodiment is that a fixed reservation ratio is difficult to adapt to changes in local request load; through restricted staged adjustments, resource sharing elasticity can be improved without exceeding the developer's configuration range.

[0087] This embodiment sequentially completes the extraction of the quota field, the calibration of the time window, and the processing of the reserved quota, so that the available quota can simultaneously reflect the current statistical window, different quota dimensions, and local resource retention requirements, thereby providing a more accurate resource basis for cross-node task matching.

[0088] In one possible example, see Figure 6 The time window calibration process in step S202 can be implemented through the following steps S301 to S303.

[0089] S301. Determine the quota reset time and window duration from the quota window field to obtain window baseline information.

[0090] The credit limit reset time indicates the moment when the current credit limit statistics window ends and the next statistics window begins. The window duration indicates the length of time covered by a complete credit limit statistics window. The window baseline information is used to determine the time range of the current credit limit statistics window.

[0091] The credit limit window field can directly include the statistical period and the absolute reset time, or it can include the statistical period and the remaining time until reset. In the latter case, the local agent program can determine the credit limit reset time based on the response reception time and the remaining time.

[0092] For example, when the response header indicates a statistical period of sixty seconds and indicates that the limit will be reset at 10:01 local time, sixty seconds can be determined as the window duration and 10:01 as the limit reset time.

[0093] S302. Based on the window reference information, filter the current quota records in the current quota statistics window from the quota records to be calibrated to obtain the current window quota records.

[0094] The local agent program determines the current credit limit statistics window based on the window baseline information and checks the collection time of each credit limit record to be calibrated. Records whose collection time falls within the current credit limit statistics window are retained, while records whose collection time falls within previous statistics windows are excluded.

[0095] The current window's credit limit records can be organized according to credit limit dimensions. For example, request dimension records and token dimension records can have their own collection times, but they must belong to the corresponding current credit limit statistics window.

[0096] In one possible embodiment of this application, a response transaction identifier can be configured for the credit limit record to be calibrated. The response transaction identifier is used to associate the response header generated by an API call with its collection time. When multiple requests return concurrently, the local agent program can first complete the record aggregation according to the response transaction identifier, and then perform window filtering. The specific problem solved by this embodiment is that concurrent responses may cause mismatches between the credit limit field and the time field in different responses; transaction-level aggregation helps to maintain the consistency of the internal fields of the credit limit record.

[0097] S303. Generate the windowed quota record based on the latest quota dimension fields in the current window quota record at the time of collection.

[0098] The local proxy program can search for the latest valid records at the time of collection in both the requested quota dimension and the token quota dimension, and write the corresponding quota value into the windowed quota record. The windowed quota record can also retain the source record identifier and collection time for subsequent verification.

[0099] When the same response provides both request dimension and token dimension data, the latest data from that response can be used directly. If a response is missing one of the credit limit dimensions, the most recent valid record for that dimension can be found within the same current credit limit statistics window, but records from the previous statistics window will not be used to supplement it.

[0100] This embodiment constrains the filtering range of the quota records to be calibrated by using window reference information, and selects the latest quota data within this range. This can prevent expired quota status from entering the process of determining the available quota, and improve the timeliness and consistency of resource status information.

[0101] In one possible example, see Figure 7 The current window quota record filtering process in step S302 can be implemented through the following steps S401 to S403.

[0102] S401. Read the acquisition time of each quota record in the quota record to be calibrated to obtain the acquisition time sequence.

[0103] The collection time is the time when the local agent program obtains the corresponding quota data. The collection time can be the response reception time or the response generation time after being corrected for a unified time base.

[0104] The data collection time sequence can be arranged chronologically, and the index relationship between each collection time and the quota record is preserved. This index relationship allows the subsequently generated window belonging markers to be reused to filter the original quota records.

[0105] S402. Based on the window reference information, determine the window affiliation of each acquisition time in the acquisition time sequence to obtain the window affiliation mark.

[0106] The window attribution flag can include at least two states: belonging to the current quota statistics window and not belonging to the current quota statistics window. For records for which a valid collection time cannot be obtained or whose collection time is significantly later than the current system time, an invalid state can also be configured and the records excluded in subsequent processing.

[0107] Window attribution can be determined sequentially based on the data collection time sequence, or by performing range filtering on the sorted data collection time sequence. Regardless of the implementation method, the window attribution marker maintains a traceable correspondence with the corresponding quota record.

[0108] In one possible embodiment of this application, the acquisition time, response transaction identifier, and quota reset time can be used together as the basis for determining ownership. When the acquisition times of two responses are close to the window boundary but their quota reset times are different, the local agent program can prioritize determining their window ownership based on the quota reset time carried by the response itself. The specific problem solved by this embodiment is that concurrent responses near adjacent window boundaries may have different reset states; by introducing response-level reset times, boundary record ownership conflicts can be reduced.

[0109] S403. Based on the window affiliation marker, retain the quota records belonging to the current quota statistics window from the quota records to be calibrated, and obtain the current window quota records.

[0110] The local agent program retains records marked as belonging to the current quota statistics window based on the index relationship between quota records and window affiliation markers, excluding other records. The filtered quota records for the current window can then be further grouped according to quota dimension, model identifier, and service provider identifier.

[0111] The current window credit limit record can retain both the credit limit value and source information. The source information can include the collection time, the response transaction identifier, and the response service provider, so as to locate the corresponding response data when the available credit limit is abnormal.

[0112] This embodiment establishes a continuous data processing link by collecting time series, window attribution markers, and record filtering, enabling the filtering results to be traced back to specific quota records and reducing the possibility of historical quota data interfering with the current scheduling.

[0113] In one possible example, see Figure 8 The window ownership determination process in step S402 can be implemented through the following steps S501 to S503.

[0114] S501. Extract the start time of the current quota statistics window and the quota reset time from the window reference information to obtain window boundary information.

[0115] The start time indicates when the current quota statistics window begins accumulating the number of requests and tokens. The quota reset time indicates when the current quota statistics window ends and the next statistics window begins. Window boundary information associates the two times as the boundaries of the same quota statistics window.

[0116] Window boundary information can also be associated with the credit limit dimension and the service provider identifier to prevent the request credit limit window from being mixed with the token credit limit window or credit limits windows from different service providers.

[0117] S502. Compare each acquisition time in the acquisition time sequence with the window boundary information one by one to obtain the time comparison result.

[0118] The time comparison result is used to indicate the positional relationship between the data collection time and the start time and the credit limit reset time. For example, the time comparison result can indicate that the data collection time is earlier than the start time, is within the current credit limit statistics window, or has reached the credit limit reset time.

[0119] Local proxy programs can use a uniform time unit for comparison. If the response header uses seconds while the local data collection time uses milliseconds, the time unit conversion can be performed first before performing the time comparison.

[0120] S503. Mark the collection time indicated by the time comparison result as belonging to the current quota statistics window, and mark the other collection times as not belonging to the current quota statistics window, thus obtaining the window belonging mark.

[0121] In one implementation, collection times that reach the start time and are earlier than the credit limit reset time are marked as belonging to the current credit limit statistics window. Collection times that reach the credit limit reset time can be assigned to the next credit limit statistics window to avoid the same record belonging to two adjacent statistics windows at the same time.

[0122] For example, the current credit limit statistics window starts at 10:00 AM and the credit limit reset time is 10:01 AM. Credit limit records collected at 10:00:20 AM and 10:00:55 AM can be marked as belonging to the current credit limit statistics window; credit limit records collected at 9:59:58 AM can be marked as not belonging to the current credit limit statistics window.

[0123] In one possible embodiment of this application, a pending confirmation status can be set for response records near the time of credit limit reset, and the credit limit value of the next response can be obtained to verify whether a credit limit reset has occurred. The specific problem solved by this embodiment is that the actual time when the service provider performs the credit limit reset may have a slight deviation from the time indicated in the response header; by verifying the subsequent response, the possibility of boundary records being incorrectly used for borrowable credit limit calculation can be reduced.

[0124] This embodiment compares the collection times one by one using window boundary information and generates a window attribution mark, so that each quota record has a clear time window attribution, which helps to maintain the accuracy of the current window quota data.

[0125] In one possible example, see Figure 9 The process of determining the window boundary information in step S501 can be implemented through the following steps S601 to S603.

[0126] S601. Read the quota reset time and the window duration from the window reference information to obtain the window boundary parameters.

[0127] The window boundary parameters are a set of time parameters used to reconstruct the current quota statistical window range. The quota reset time can be an absolute time or an absolute time converted from a relative reset duration. The window duration can be determined by the statistical period field in the response header.

[0128] The local agent program can verify whether the credit limit reset time is later than the current data collection time, and whether the window duration is consistent with the restriction rules of the corresponding credit limit dimension. Window boundary parameters that fail the verification can be marked as invalid, awaiting subsequent valid responses for updating.

[0129] S602. Based on the window boundary parameters, shift the quota reset time forward by the window duration to obtain the starting time.

[0130] For example, if the credit limit reset time is 10:01 and the window duration is 60 seconds, then 10:00 can be determined as the start time. This process is completed according to the physical meaning of the credit limit reset time and the window duration, without the need to introduce additional mathematical formulas.

[0131] When the service provider sets different statistical periods for the request quota and the token quota respectively, the window boundary parameters of the two quota dimensions can be processed separately to obtain the start time of the request quota window and the start time of the token quota window.

[0132] In one possible embodiment of this application, window baseline information can be maintained for different credit limit dimensions, and the corresponding window identifier can be retained when the available credit limit is formed. The subdivision problem solved by this embodiment is that the requested credit limit and the token credit limit may use different statistical periods. If a single window is forcibly shared, it is easy to cause a mismatch in credit limit status. By maintaining the window boundaries in different dimensions, the accuracy of multi-dimensional credit limit scheduling can be improved.

[0133] S603. Associate the start time and the quota reset time to obtain the window boundary information.

[0134] Window boundary information can include start time, credit limit reset time, window duration, credit limit dimension, service provider identifier, and model identifier. Once the start time and credit limit reset time are associated, they can serve as common inputs for determining window ownership in subsequent data collection.

[0135] When entering the next quota statistics window, the local agent program can update the window boundary information according to the new quota reset time and transfer the previous window boundary information to the history to support scheduling anomaly verification.

[0136] This embodiment uses the quota reset time and window duration to restore the start time and form window boundary information with correlation, so that the current quota statistics window can still be determined even when the response header does not directly provide the start time.

[0137] In one possible example, see Figure 10 After determining the request to be executed from the remote queue, the remote task pause and reschedule process can be implemented through the following steps S701 to S704.

[0138] S701. Obtain the subsequent demand amount and task progress information of the remote task corresponding to the request to be executed, and monitor the queue status of the local queue.

[0139] A remote task can consist of multiple remote requests executed in a specific order. The subsequent demand limit represents the expected request and token limits that will be consumed by unexecuted requests following the currently pending request. The subsequent demand limit can be submitted by the task client or determined by the local agent based on the number of unexecuted requests and the expected token consumption.

[0140] Task progress information may include remote task identifier, total number of requests, completed request identifier, currently executing request identifier, and unexecuted request identifier. Queue status may include the number of pending requests in the local queue, the earliest enqueue time, and whether a new local request has appeared.

[0141] S702. Determine the remote execution status based on the queue status, the available loan amount, and the subsequent demand amount.

[0142] Remote execution status can include continue execution status, local request pending status, and insufficient quota status. The continue execution status can be determined when there are no pending requests in the local queue and the available quota is sufficient to meet subsequent demand.

[0143] When there are pending local requests in the local queue, the pending status of the local requests can be determined. When the available lending limit at the request dimension or token dimension is less than the corresponding subsequent demand limit, the insufficient limit status can be determined.

[0144] The remote execution status can be updated as the queue status and available lending quota change. Therefore, the local agent can continuously determine whether the conditions are met to continue retrieving remote requests during the execution of a remote task.

[0145] S703. When the remote execution status indicates that there are pending local requests in the local queue, or when the available lending amount is less than the subsequent demand amount, stop determining new pending requests from the remote queue after the current pending request has been executed.

[0146] Stopping the fetching of new remote requests does not require canceling currently ongoing large model API calls. The local agent can wait for the currently pending requests to obtain execution results, update task progress information, and then pause remote queue scheduling.

[0147] For example, a remote task consists of ten remote requests. While the first four requests are completed and the fifth request is executing, local requests appear in the local queue. The local agent can wait for the fifth request to complete, then stop fetching the sixth request and prioritize processing the local requests in the local queue.

[0148] S704. Generate rescheduling information based on the remote execution status and the task progress information, and send the rescheduling information to the scheduler so that the scheduler can schedule the unexecuted requests in the remote task.

[0149] The rescheduling information can be understood as a task migration notification message sent by a local agent node to the scheduler when it is unable to continue executing a remote task. This message triggers the scheduler to reassign unexecuted requests in the current remote task to other available local agent nodes. The rescheduling information may include the remote task identifier, the remote execution status that triggered the rescheduling, the scope of completed requests, the scope of unexecuted requests, and the corresponding quota requirements. After receiving the rescheduling information, the scheduler can select another available local agent node from the global resource view and issue the unexecuted requests.

[0150] Specifically, in step S703, when extracting new requests to be executed stops, the scope of completed requests and the scope of unexecuted requests in the task progress information are updated; in step S704, the remote task identifier and unexecuted requests are extracted from the updated task progress information as task information to be migrated, the reason for triggering rescheduling and the corresponding quota constraint information are extracted from the remote execution status, and the task information to be migrated and the quota constraint information are associated as the rescheduling information; the rescheduling information is used to indicate to the scheduler the completed requests, unexecuted requests and the corresponding quota requirements.

[0151] In the example of the ten remote requests mentioned above, the rescheduling information can indicate that the first five requests have been completed and the sixth through tenth requests have not yet been executed. The scheduler only reschedules the sixth through tenth requests, thereby reducing the possibility of completed requests being invoked repeatedly.

[0152] In this embodiment, when a local request arrives or the available loan amount is insufficient, subsequent remote scheduling is paused with the completion of the current API call as the boundary, and unexecuted requests are returned to the scheduler for migration. This approach can balance the integrity of the current call, the priority of local request processing, and the continuity of remote tasks.

[0153] In one possible example, see Figure 11 The rescheduling information generation process in step S704 can be implemented through the following steps S801 to S803.

[0154] S801. Extract the remote task identifier and unexecuted request from the task progress information to obtain the task information to be migrated.

[0155] The remote task identifier is used to uniquely identify the remote task submitted by the task client. Unexecuted requests are those that have not yet been invoked to the large model API, excluding requests that have already obtained execution results.

[0156] The information for the task to be migrated may include the remote task identifier, the identifier of the unexecuted request, the unexecuted request body or the storage location of the request body, the request execution order, and the task client identifier. For remote tasks that need to maintain order, the position of the next request to be executed can also be recorded.

[0157] S802. Determine the quota constraint information corresponding to the task information to be migrated based on the remote execution status.

[0158] When the remote execution status is insufficient, the quota constraint information may include the minimum request quota and minimum token quota required for the unexecuted request. When the remote execution status is local request pending, the quota constraint information may include the original estimated quota for the unexecuted request and the allowed target node conditions.

[0159] Quota constraint information may also include target model type, request timeout time, and network latency requirements. The scheduler can use quota constraint information to exclude local proxy nodes that cannot meet the requirements of unexecuted requests.

[0160] S803. Associate the task information to be migrated with the quota constraint information to generate the rescheduling information.

[0161] Rescheduling information can be in the form of structured task messages. These messages carry both information about the tasks to be migrated and quota constraints, enabling the scheduler to identify requests that need to continue execution and the resource conditions that the target node must meet.

[0162] In one possible embodiment of this application, an idempotency flag can be configured for each remote request and carried in the rescheduling information. Before executing the request, the target local proxy node can query whether the idempotency flag already has a completion record. The specific problem solved by this embodiment is that network retries or state synchronization delays may cause the same request to be executed repeatedly; request-level idempotency verification helps reduce duplicate calls and duplicate billing.

[0163] This embodiment extracts the scope of tasks to be migrated from the task progress information, forms quota constraint information according to the remote execution status, and associates the two as rescheduling information, so that the scheduler can select nodes that meet the subsequent execution conditions without repeatedly scheduling completed requests.

[0164] In one possible example, see Figure 12 After sending resource status information to the scheduler, the local resource contribution control process can be implemented through the following steps S901 to S903.

[0165] S901, Receive contribution control instructions generated by the local control interface; the contribution control instructions include pause instructions, sharing ratio adjustment instructions, or exit instructions.

[0166] The local control interface can be a command-line tool, a desktop control program, or a web interface accessible only to the local machine. The command-line tool can provide commands such as status query, pausing sharing, resuming sharing, adjusting sharing ratios, and exiting the contribution network.

[0167] The pause command is used to temporarily stop receiving new remote requests. The share ratio adjustment command is used to change the preset reserved ratio or the maximum allowed sharing ratio. The exit command is used to remove the local agent node from the contributing network and stop subsequent resource status heartbeats.

[0168] S902. Update the contribution status and the available lending amount according to the contribution control instruction to obtain node status information.

[0169] Contribution status can include available, suspended, and exited status. When a suspension instruction is received, the contribution status can be updated to suspended, and the available lending limit can be updated to unschedulable limit. When a sharing ratio adjustment instruction is received, the available lending limit can be re-determined according to the adjusted sharing ratio.

[0170] When a withdrawal command is received, the contribution status can be updated to a withdrawal status. Node status information can include the node identifier, the updated contribution status, the updated available lending limit, the effective time of the status, and the reason for the status change.

[0171] S903. Send the node status information to the scheduler so that the scheduler stops issuing remote requests when the contribution status is unavailable, or issues remote requests according to the updated lendable quota.

[0172] Node status information can be sent as an immediate status message without waiting for the next heartbeat cycle. After receiving the node status information, the scheduler updates the node's availability and available loan quota in the global resource view.

[0173] For local agent nodes in a paused or exited state, the scheduler stops issuing new remote requests. For remote requests that have already started executing, the scheduler can wait for the current request to complete and return the execution result; requests that have not yet been executed can be migrated to other nodes through the rescheduling process.

[0174] When a shared resource ratio adjustment command is issued, the scheduler determines whether the node can accept new remote tasks based on the updated available loan quota. Therefore, adjustments made by developers to the scope of local resource contributions can promptly impact the distribution of subsequent tasks.

[0175] This embodiment converts developer operations into contribution control instructions, and then converts the contribution control instructions into node status information that can be processed by the scheduler, enabling local agent nodes to dynamically pause, adjust, or withdraw from resource contributions, while maintaining consistency between the scheduler's resource view and the local contribution intentions.

[0176] See Figure 13 The decentralized large-scale computing power joint scheduling system 100 based on local agents may include a scheduler 120 and a local agent node group 110. The scheduler 120 may include a global resource view unit 271, a target node selection unit 1312, and a result forwarding unit 1313. Each local agent node in the local agent node group 110 may include a credential and information acquisition module 1321, a quota processing module 1322, a queue management module 1323, a request execution module 1324, and a result output module 1325.

[0177] The credential and information acquisition module 1321 is used to decrypt the encrypted access credentials stored in the local proxy node, obtain the runtime access credentials that reside only in the running memory, and obtain the quota status information carried in the large model API response header.

[0178] The credential and information acquisition module 1321 can connect to the local credential storage and the large model API response interface. The credential decryption function and the response header parsing function can be implemented by different threads within the same process, or by different local processes with controlled data interfaces. Regardless of the implementation method, runtime access to credentials does not output to the scheduler 120.

[0179] The quota processing module 1322 is used to determine the available quota based on the quota status information and send the resource status information to the scheduler 120; the resource status information includes the available quota but does not include the encrypted access credential and the runtime access credential.

[0180] The quota processing module 1322 can perform quota field extraction, time window calibration, reserved quota processing, and resource status encapsulation. The global resource view unit 271 receives resource status information sent by multiple local agent nodes and maintains available resource records according to node identifiers.

[0181] The queue management module 1323 is used to receive local requests and write them to the local queue through the local agent program, and to receive remote requests issued by the scheduler 120 based on resource status information and write them to the remote queue.

[0182] The target node selection unit 1312 reads the resource status information from the global resource view unit 271, selects a target local proxy node that can meet the remote request quota requirements, and sends the remote request to the target local proxy node. The queue management module 1323 saves local requests and remote requests according to the request source.

[0183] The request execution module 1324 is used to determine the request to be executed from the local queue and the remote queue based on the queue priority of the local queue being higher than that of the remote queue, and to execute the request by calling the large model API using runtime access credentials, and to obtain the execution result.

[0184] The request execution module 1324 can establish a credential call interface with the credential and information acquisition module 1321 for use only within the local process. The request execution module 1324 uses runtime access credentials through this interface, but does not write the runtime access credentials to the request log, task status information, or execution results.

[0185] The result output module 1325 is used to send the execution result to the scheduler 120 or the local requester based on the source of the request to be executed.

[0186] When the request to be executed is a local request, the result output module 1325 sends the execution result to the local requester. When the request to be executed is a remote request, the result output module 1325 sends the execution result and the remote task identifier to the scheduler 120. The result forwarding unit 1313 forwards the execution result corresponding to the remote request to the task client according to the remote task identifier.

[0187] During system operation, the voucher and information acquisition module 1321 provides quota status information to the quota processing module 1322, the quota processing module 1322 provides resource status information to the scheduler 120, the scheduler 120 provides remote requests to the queue management module 1323, the queue management module 1323 provides the queue to be scheduled to the request execution module 1324, and the request execution module 1324 provides the execution result to the result output module 1325. Thus, the inputs and outputs of each module form a continuous data processing relationship.

[0188] This embodiment uses multiple local agent nodes to manage access credentials and make actual large-scale model API calls. The scheduler 120 performs resource view maintenance, target node selection and result forwarding without touching access credentials. This reduces the security risks caused by centralized storage of access credentials and enables cross-node joint scheduling by utilizing the borrowable quota of each local agent node.

[0189] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes an electronic device.

[0190] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0191] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0193] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0195] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0196] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0197] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A decentralized large-scale model computing power joint scheduling method based on local agents, characterized in that, Applied to a local proxy node, the method includes: Decrypt the encrypted access credentials stored locally to obtain runtime access credentials that reside only in the runtime memory; retrieve the quota status information carried in the large model API response header; Based on the quota status information, the available quota is determined, and resource status information is sent to the scheduler. The resource status information includes the available quota but does not include the encrypted access credential and the runtime access credential. The local agent receives local requests and writes them to the local queue; it also receives remote requests issued by the scheduler based on the resource status information and writes them to the remote queue. Based on the queue priority of the local queue being higher than that of the remote queue, a request to be executed is determined from the local queue and the remote queue. The request to be executed is then executed by calling the large model API using the runtime access credentials, and the execution result is obtained. The execution result is sent to the scheduler or the local requester based on the source of the request to be executed.

2. The method according to claim 1, characterized in that, The step of determining the available loan amount based on the credit limit status information includes: Extract the quota dimension field and quota window field from the quota status information to obtain the quota record to be calibrated; the quota dimension field includes the remaining requested quota field and the remaining token quota field, and the quota window field includes the statistical period field and the quota reset field; The time window calibration is performed on the quota record to be calibrated according to the quota window field to obtain a windowed quota record; Based on the windowed quota record and the preset reservation ratio, the available loan quota is determined for the request dimension and the token dimension, respectively, and the available loan quota is obtained.

3. The method according to claim 2, characterized in that, The step of performing time window calibration on the credit limit record to be calibrated according to the credit limit window field to obtain a windowed credit limit record includes: The credit limit reset time and window duration are determined from the credit limit window field to obtain window baseline information; Based on the window reference information, the current window credit records are obtained by filtering the credit records to be calibrated within the current credit statistics window. The windowed quota record is generated based on the latest quota dimension fields collected at the current window quota record.

4. The method according to claim 3, characterized in that, The step of filtering the current credit limit records within the current credit limit statistics window from the credit limit records to be calibrated based on the window reference information to obtain the current window credit limit records includes: Read the acquisition time of each quota record in the quota record to be calibrated to obtain the acquisition time sequence; Based on the window reference information, the window attribution of each acquisition time in the acquisition time sequence is determined to obtain the window attribution marker; Based on the window affiliation marker, retain the quota records belonging to the current quota statistics window from the quota records to be calibrated, and obtain the current window quota record.

5. The method according to claim 4, characterized in that, The step of determining the window affiliation for each acquisition time in the acquisition time sequence based on the window reference information to obtain a window affiliation marker includes: The window boundary information is obtained by extracting the start time of the current quota statistics window and the quota reset time from the window reference information; Each acquisition time in the acquisition time sequence is compared with the window boundary information to obtain the time comparison result; The collection time indicated by the time comparison result between the start time and the quota reset time is marked as belonging to the current quota statistics window, and the other collection times are marked as not belonging to the current quota statistics window, thus obtaining the window belonging mark.

6. The method according to claim 5, characterized in that, The step of extracting the start time of the current quota statistics window and the quota reset time from the window reference information to obtain window boundary information includes: Read the quota reset time and the window duration from the window baseline information to obtain the window boundary parameters; Based on the window boundary parameters, the start time is obtained by shifting the quota reset time forward by the window duration. By associating the start time and the quota reset time, the window boundary information is obtained.

7. The method according to any one of claims 1 to 6, characterized in that, After determining the request to be executed from the remote queue, the method further includes: Obtain the subsequent demand amount and task progress information of the remote task corresponding to the request to be executed, and monitor the queue status of the local queue; The remote execution status is determined based on the queue status, the available lending limit, and the subsequent demand limit. When the remote execution status indicates that there are pending local requests in the local queue, or when the available lending amount is less than the subsequent demand amount, the process of determining new pending requests from the remote queue will stop after the current pending request has been executed. Rescheduling information is generated based on the remote execution status and the task progress information, and the rescheduling information is sent to the scheduler so that the scheduler can schedule the unexecuted requests in the remote task.

8. The method according to claim 7, characterized in that, The step of generating rescheduling information based on the remote execution status and the task progress information includes: The remote task identifier and unexecuted requests are extracted from the task progress information to obtain the task information to be migrated; Determine the quota constraint information corresponding to the task information to be migrated based on the remote execution status; The rescheduling information is generated by associating the task information to be migrated with the quota constraint information.

9. The method according to any one of claims 1 to 6, characterized in that, After sending the resource status information to the scheduler, the method further includes: Receive contribution control instructions generated by the local control interface; the contribution control instructions include pause instructions, sharing ratio adjustment instructions, or exit instructions. Update the contribution status and the available lending amount according to the contribution control instruction to obtain node status information; The node status information is sent to the scheduler so that the scheduler stops issuing remote requests when the contribution status is unavailable, or issues remote requests according to the updated lendable quota.

10. A decentralized large-scale model computing power joint scheduling system based on local agents, characterized in that, It includes a scheduler and multiple local agent nodes; each of the local agent nodes includes: The credential and information acquisition module is used to decrypt the encrypted access credentials stored in the local proxy node to obtain runtime access credentials that reside only in the running memory, and to obtain the quota status information carried in the large model API response header. The quota processing module is used to determine the available quota based on the quota status information and send resource status information to the scheduler. The resource status information includes the available quota but does not include the encrypted access credential and the runtime access credential. The queue management module is used to receive local requests and write them to the local queue through a local agent program, and to receive remote requests issued by the scheduler based on the resource status information and write them to the remote queue. The request execution module is used to determine the request to be executed from the local queue and the remote queue based on the queue priority of the local queue being higher than that of the remote queue, and to execute the request to be executed by calling the large model API using the runtime access credentials, thereby obtaining the execution result; The result output module is used to send the execution result to the scheduler or the local requester according to the source of the request to be executed; The scheduler is used to select a target local proxy node based on the resource status information of each local proxy node, send the remote request to the target local proxy node, and forward the execution result corresponding to the remote request to the task client.

Images