A batch computing system with active backhaul and size load splitting

Abstract

The application discloses a kind of active backhaul and size load shunt's batch processing computing system, by calling end receives unit to be executed, parameter sequence, working directory and resource demand, organization is N logical task, generates job identification T and logical task identification i, release task data and control request;Submit end checks control request, calculates execution slot quantity P, applies P to batch processing resource scheduler Array execution slot, generates task description and submits / cancels;Each array execution slot of execution end scans from staggered starting position, and exclusive acquisition includes job identification T, logical task identification i, the task data of load, executes user function, support user function explicit call active backhaul interface during execution to release active backhaul (Callback) record, and after active backhaul (Callback) execution load shunt, write in process and terminal state record to execution result, and solve the problem that present batch processing computing environment and remote function call are combined.

Description

Technical Field

[0001] This invention belongs to the field of high-performance computing resource scheduling technology, specifically relating to a batch processing computing system with active backhaul and load splitting. Background Technology

[0002] Batch processing computing environments allocate resources such as CPU, memory, and GPU among multiple users and computing nodes through queues and job scheduling, making them suitable for offline and high-performance computing that require resource requests on a task-by-task basis. Remote function calls emphasize that the calling end submits the unit to be executed and parameters and obtains the results in a manner close to local calls. Combining the two requires establishing a stable connection between convenient call semantics and batch processing resource requests, job execution, and result retrieval.

[0003] The combination of existing batch processing computing environments (such as the cluster job scheduling and management system Slurm) with remote function calls has the following technical shortcomings:

[0004] Disconnect between invocation and resource allocation: General remote calls are geared towards fixed services and cannot dynamically request batch processing resources based on the needs of a single task, such as CPU, memory, and GPU. They lack a mechanism for expressing and fulfilling resource requirements from the caller to the scheduler.

[0005] The scheduling efficiency of a large number of tasks versus a limited number of execution slots is contradictory: when the number of logical tasks N is much greater than the parallelism P, creating an array element for each task causes the scheduling record to grow with N; fixed task mapping causes resource idleness and tail waiting when the time consumption is uneven, while dynamic allocation is prone to duplicate acquisition or pollution of running information.

[0006] The return of intermediate information for long-running tasks is difficult: returning results only after the task is completed cannot reflect the progress; the direct connection between computing nodes and the calling end is affected by network isolation and connection interruption; high-frequency process information easily consumes resources, and there is a lack of reliable and controllable active return solutions.

[0007] The transmission path for payloads of different sizes is singular: transmitting large objects uniformly through the message system is prone to exceeding limits, increasing memory and network pressure; writing to external storage uniformly increases latency and metadata overhead for small objects; and dividing paths by type leads to inconsistent protocols and a lack of a unified discrimination and diversion mechanism.

[0008] Missing submission status confirmation and final record: Locally generated job identifiers cannot prove that the scheduler has accepted the job; historical status cannot be obtained after the calling end disconnects or restarts; when batch submission fails, a clear failure final status record cannot be generated for each logical task. Summary of the Invention

[0009] To address the aforementioned technical problems, the present invention provides a batch processing computing system with active backhaul and large / small load splitting, comprising: The calling end receives the unit to be executed, parameter sequence, working directory and resource requirements, organizes them into N logical tasks, generates job identifier T and logical task identifier i, and publishes task data and control requests. The submitting end verifies the control request, calculates the number of execution slots P, requests P array execution slots from the batch resource scheduler, generates a task description, and submits / cancels the task. At the execution end, each array execution slot scans from staggered starting positions and exclusively retrieves task data containing job identifier T, logical task identifier i, and payload. User functions are executed, and during execution, user functions can explicitly call the active callback interface to publish active callback records. The execution results and active callback execution payloads are split and written into process and final state records.

[0010] Preferably, the user function explicitly calls the process callback interface before returning or throwing an exception. The execution end reads the job identifier T and logical task identifier i from the current task running information, serializes the intermediate object, selects the execution payload, forms an active callback record carrying the job identifier T, logical task identifier i and unified payload, and writes it into the process and final state records.

[0011] Preferably, the system further includes a unified load structure, comprising mutually exclusive inline load branches and external load branches, to perform the load splitting.

[0012] Preferably, the execution terminal establishes an independent slot root directory for each array execution slot and an independent subdirectory for each acquired logical task identifier i. After the logical task ends, the slot root directory is restored, the running path of the task is removed, and the independent subdirectory of the logical task is deleted.

[0013] Preferably, the execution end provides array slot index j to P array execution slots, and the array slot index j and the logical task identifier i establish a dynamic relationship through runtime exclusive acquisition.

[0014] Preferably, the calling end compresses the working directory and uploads it to the external storage directory corresponding to the job identifier T.

[0015] Preferably, the calling end constructs a public-key encrypted control request, which includes job identifier, logical task, request parallelism, executing user, operating environment, and resource requirements.

[0016] Preferably, the submitting end uses an internally stored private key to decrypt and verify the control request.

[0017] This invention provides a batch processing computing system with active callback and large / small load splitting. The calling end receives the unit to be executed, parameter sequence, working directory, and resource requirements, organizes them into N logical tasks, generates a job identifier T and a logical task identifier i, and publishes task data and control requests. The submitting end verifies the control requests, calculates the number of execution slots P, requests P array execution slots from the batch processing resource scheduler, generates task descriptions, and submits / cancels them. The executing end scans each array execution slot from staggered starting positions and exclusively retrieves task data containing the job identifier T, logical task identifier i, and load, executes user functions, and supports explicit calls to the active callback interface during execution to publish active callback records. The system also writes the execution results and the load splitting of the active callback into process and final state records, solving the problems that arise when combining existing batch processing computing environments with remote function calls. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of a batch processing computing system architecture that features proactive data return and load balancing. Figure 2 A schematic diagram of a batch processing calculation method with active feedback and load splitting; Figure 3 This is a schematic diagram of the proactive feedback process during execution. Figure 4 A diagram illustrating the unified payload structure and transmission / reading process; Figure 5 A diagram illustrating the relationship between job status and result records, as well as a completion determination diagram; Figure 6 A schematic diagram of the final state compensation process for each logical task that fails to submit an array; Figure 7 A schematic diagram of the concurrency control process is provided for the runtime environment. Figure 8 A schematic diagram of the process for canceling security checks for a task. Detailed Implementation

[0019] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0020] The technical terms and their meanings in this invention are as follows: Units to be executed: Functions, scripts, program entry points, container entry points, WebAssembly (WASM) modules, or deployed execution entry point identifiers that can be resumed and executed on compute nodes.

[0021] Logical task: A single business calculation formed by the unit to be executed and a set of parameters. N sets of parameters correspond to N logical tasks.

[0022] Job Identifier T: An identifier generated by the calling end for a batch of remote calculations, used to associate the batch's submission request, logical task, running status, process feedback information, calculation results, and external storage objects. Job Identifier T is not equivalent to the job number generated by the batch scheduler.

[0023] Scheduler Job Number: The number returned by the batch processing scheduler after the submitting end successfully submits a resource request to the batch processing scheduler. It is used to query or cancel the actual batch processing job.

[0024] Logical task identifier i: an identifier used to distinguish logical tasks under the same job identifier T, which is an integer from 0 to N-1 in the example.

[0025] Job array execution slot j: The computing resource unit actually scheduled by the batch scheduler. The execution slot index j only identifies the allocated computing resource and does not correspond to the logical task identifier i; multiple logical tasks can be processed sequentially in the same execution slot.

[0026] Exclusive Acquisition Record: An occupancy record created atomically and exclusively in the shared coordination storage before the executor processes the logical task. Only executors that successfully create such a record can execute the corresponding logical task.

[0027] Active callback during execution: This refers to the operation where a user function actively sends progress, interim results, metrics, or other intermediate information before it has finished, rather than a completion callback triggered by the calling framework after the task is completed.

[0028] The unified load structure includes inline load branches and external load branches: a structure that can explicitly represent inline data or external storage references and recover the original data from the same read interface, performing load splitting.

[0029] Inline payload branch: Serialized data that is transmitted directly with the message.

[0030] External payload branch: Writes to external storage and only references the serialized data transmitted with the message to the storage location.

[0031] Job status and result records: Save information such as task creation, submission, submission failure, running, process return, success, failure, and cancellation by job identifier T, and allow message records or stored records to be read again within the retention period.

[0032] Reacquire job running status and results: The original calling process or other processes read the corresponding running status, process feedback information, success results or failure information from the history record based on the job identifier T.

[0033] Final state record: The final state record formed after a logical task is completed, fails, or is canceled, specifically including success, failure, or cancellation.

[0034] like Figure 1 As shown in the figure, an embodiment of the present invention provides a batch processing computing system with active backhaul and large / small load splitting, the system comprising: The calling end receives the unit to be executed, parameter sequence, working directory and resource requirements, organizes one unit to be executed and N sets of parameters into N logical tasks (i.e. task data plane), generates job identifier T for the whole batch of tasks, and assigns a logical task identifier i to each logical task. The logical task identifier independently represents the business task, publishes task data and control requests associated with T and i, and processes submission receipts, active callbacks and final results.

[0035] The job identifier T is associated with the batch call submission request, task data, historical running status, results, and external storage directory.

[0036] The logical task identifier i is associated with a set of parameters, running records, zero or more active callbacks, and a success, failure, or cancellation record.

[0037] The calling end is responsible for receiving the unit to be executed, the parameter sequence, and the resource requirements, generating T and i, preparing task data and control requests, and processing submission receipts, active callbacks, and final results.

[0038] The business environment that initiates remote calls and receives results has the following main inputs: the unit to be executed, the parameter sequence, the working directory, and resource requirements. The main outputs are task data, control requests, creation records, and local processing results. The resource responsibility is to declare the resource requirements of a single task and request parallelism. The task responsibility is to generate i with N sets of parameters and publish the task data. The payload responsibility is to serialize functions and parameters and distribute execution size. The recovery result and active callback are also included. The active callback responsibility is to receive, recover, and call the local process processing function. The security responsibility is to encrypt the submission and cancellation control commands. The environment and logs are packaged working directories and specified runtime environment.

[0039] Job status and result acquisition module: Located on the calling end, it reads the running status, process feedback information, success results and failure information according to the job identifier T, and calls the corresponding local processing functions.

[0040] Control message channel: Transmits encrypted batch commit and cancel commands from the calling end to the submitting end.

[0041] Task data channel: Transmits a task data packet containing T, i, the payload of the unit to be executed, and the parameter payload from the calling end to the executing end.

[0042] The submitting end receives and verifies the control request, calculates the number of execution slots P based on the number of tasks, the parallelism of the request, and the parallel limit of the submitting end, requests P job array execution slots from the batch processing scheduler, generates a batch processing task description, passes T, the total number of logical tasks N, and the number of execution slots P to the execution environment, and submits or cancels the job to the resource scheduler.

[0043] P = min(N, P_req, P_srv) The parallelism of the caller's request or computation is P_req, and the upper limit of the execution slots configured on the submitter is P_srv. The default P_req of the caller can also be determined by the smaller of the preset total CPU divided by the CPU requirement of a single task and the preset total memory divided by the memory requirement of a single task, and users are allowed to further reduce it.

[0044] The submitting end is responsible for receiving and verifying control requests, determining the number of execution slots P, generating batch task descriptions, and submitting or canceling jobs to the resource scheduler.

[0045] The batch resource scheduler receives the task description from the submitting end and schedules the execution slots according to the user, queue, and resource requirements.

[0046] The submitting end can access the service nodes of the batch processing scheduling control plane. The main input is the encrypted submit or cancel control command, and the main output is the batch processing task description, submit receipt, and submission failure handling record. The resource responsibility is to verify resource parameters, determine valid P, and request execution slots. The task responsibility is to convert a batch request into P batch processing execution slots. The payload responsibility is not to forward functions, parameters, results, or callback payloads. The callback responsibility is not to generate user function procedure callbacks. The security responsibility is to decrypt and verify the control command and verify the cancellation target. The environment and logs are declared in the task description as environment and log processing requirements.

[0047] The execution end runs in a batch processing computing environment. It is responsible for scanning and dynamically and exclusively retrieving logical tasks, restoring the units to be executed and their parameters, executing user functions, and writing process records such as running, active callback, success or failure, and final state records. It provides T, N, P and the initial array slot index j to each of the P job array execution slots. The array slot index j and the logical task i are dynamically related through runtime retrieval, and are not a one-to-one fixed mapping. The array slot index only identifies the execution resource. It scans the set of logical tasks in each job array execution slot from staggered starting positions, and competes for unretrieved logical tasks through the exclusive retrieval records corresponding to T and i in the shared storage. After completing one logical task, it continues to retrieve the next one, completing one batch remote execution.

[0048] The execution slot index j is associated with the array of execution slots actually started by the batch processing scheduler and the local execution directory of the execution end. After execution slot j successfully creates an exclusive pick-up record corresponding to job identifier T and logical task identifier i, the execution slot is responsible for executing logical task i; after the task is completed, the same execution slot can continue to pick up other logical tasks.

[0049] The execution end runs in a batch processing computing environment and is responsible for scanning and exclusively acquiring logical tasks, restoring the units to be executed and their parameters, executing user functions, and writing status and result records such as running, active callback, success or failure.

[0050] The execution end consists of compute nodes allocated by the batch scheduler. Its main inputs are T, N, P, task data, and execution environment information. Its main outputs are runtime records, callbacks, success or failure results, logs, and large result payloads. Its resource responsibility is to execute logical tasks using allocated resources; it does not handle cluster resource scheduling. Its task responsibility is to dynamically scan and exclusively claim i. One execution slot can process multiple tasks consecutively. Its payload responsibility is to recover functions and parameters. It performs size-based routing for results and callbacks. Its callback responsibility is to read T and i during user function execution, and to proactively publish callbacks after rate limiting. Its security responsibility is to access external storage as a task and delete temporary credentials after authentication. It initializes and reuses the runtime environment and logs, isolates task directories, and uploads execution logs.

[0051] Each functional module adopts a consistent protocol semantics, and the interaction information includes at least batch control information, status type, unified payload structure, job identifier, logical task identifier, and resource parameters. Each module completes sending, receiving, and recovery based on a unified data definition, thereby ensuring that control information, task data, job status, and results can be correctly associated.

[0052] The calling end does not directly manipulate the batch scheduler, the submitting end does not execute user functions and does not forward function parameters or return values, and the executing end is not responsible for defining the calling task or deciding on cluster-level resource allocation.

[0053] Status and Result Channels: The submitting or executing end writes records such as submission, execution, callback, success, failure, and cancellation, which are then read by the calling end.

[0054] Payload selection module: The calling end and the executing end generate inline payload or external storage reference based on the serialization length when sending the payload, and restore them uniformly when receiving.

[0055] Retrieval Coordination Storage: Provides the execution end with exclusive retrieval records corresponding to T and i.

[0056] External payload storage: Allows the calling and executing ends to save or read working directories, large payloads, environment archives, logs, and retrieval records.

[0057] User functions can explicitly call the active callback interface during execution on a compute node before generating a final return value. This interface obtains the job identifier T and logical task identifier i from the current task execution information, serializes intermediate objects, controls the transmission frequency and time window byte size, and then forms an active callback record independent of the success or failure final state. The execution unit, parameters, return value, and active callback data adopt a unified payload structure: small payloads are transmitted directly with the message, while large payloads are compressed and written to external storage, with a storage location reference carried in the message. The job execution status, process callback information, and results are saved according to the job identifier T, enabling the calling end to process active callbacks in execution in real time, and allowing the original calling process or other processes to retrieve historical execution status and results based on T.

[0058] Key Relationship: The number of logical tasks N is decoupled from the number of execution slots P in the job array z.

[0059] like Figure 2 As shown in the figure, the batch processing computing system with active backhaul and large / small load splitting provided by the embodiment of the present invention employs a batch processing computing method with active backhaul and large / small load splitting, the method including the following steps: In step S101, the calling end receives the function to be executed F, the parameter sequence [A0, A1, …, A(N-1)], the working directory, the runtime environment identifier, and the resource requirements for each task, such as the central processing unit (CPU), memory, and graphics processing unit (GPU).

[0060] In step S102, the calling end generates a job identifier T and assigns a logical task identifier i to each set of parameters. The combination of T and i uniquely identifies a logical task in this batch call.

[0061] In step S103, the calling end uses a serialization mechanism for resumable executable units to process F and each Ai. Since the deserialized object may contain executable content, the system only accepts trusted callers and trusted payloads.

[0062] In step S104, the load selection module applies a unified load selection to F and each Ai to generate an inline byte load or an external storage reference load.

[0063] In step S105, the calling end writes a creation record associated with the job identifier T and the logical task identifier i for each logical task, and sends task data containing the job identifier, logical task identifier, payload of the unit to be executed, and parameter payload to the task data channel.

[0064] Step S106: The calling end compresses the working directory and uploads it to the external storage directory corresponding to T. In this embodiment, the working directory is fixed to use external storage and does not participate in the threshold splitting of message payload.

[0065] Step S107: The calling end constructs a batch control request containing T, N, request parallelism, executing user, runtime environment, and resource requirements, and publishes it to the control message channel after encrypting it with the submitting end's public key.

[0066] In step S108, the submitting end obtains the control message, decrypts it using the private key stored only on the submitting end, and verifies the fields T, user, environment, number of tasks, parallelism, central processing unit (CPU), memory, graphics processing unit (GPU), notification, and authentication. The submitting end rejects unencrypted submit and cancel requests.

[0067] In step S109, the submitting end calculates P, generates a job array containing P execution slots, and submits a dynamic task description through the batch scheduling interface; this array contains only P actual execution slots, instead of first creating N array elements and then limiting the number of simultaneous runs.

[0068] Step S110: After successful submission, the submitting end publishes a submission success receipt containing T and the scheduler job identifier. The calling end has already established a status receiver corresponding to T before sending the control request, and only returns T to the upper layer after receiving the success receipt; if a submission failure receipt is received, an error is reported directly, and if no receipt is received after the waiting time limit, a timeout is reported.

[0069] Step S111: If the batch submission fails as a whole, the submission end first issues a submission failure receipt associated with T, and then issues a failure final state for each i according to the task list, so that both synchronous submission feedback and asynchronous batch completion judgment can end.

[0070] In step S112, the batch scheduler starts P array execution slots. The script provides each slot with T, N, P, and the initial array slot index j.

[0071] In step S113, the execution end establishes a local cache directory, environment directory, lock directory, execution slot directory, and log directory on the compute node according to the execution user. When multiple execution slots use the same environment for the first time, they compete for the file lock named after the environment identifier. Only the lock holder downloads and decompresses the environment, and writes a completion marker after the environment relocation is successful.

[0072] In step S114, when access to the authentication storage is required, the executing end obtains the temporary authentication materials controlled by the submitting end, completes identity initialization, and deletes the corresponding task execution information fields. Unauthenticated and authenticated clusters use different user root directories to prevent the executing end from implicitly falling back to an incorrect identity.

[0073] In step S115, the execution end starts an execution runtime process once in each execution slot. This process first creates or confirms the acquisition coordination area corresponding to T, and then enters the logical task scanning loop.

[0074] Step S116, for slot index j, the execution end calculates the scan start point: The initial scan position of slot j is the integer obtained by rounding down N×j / P.

[0075] The candidate order is scanned from start(j) to N-1, and then from 0 to start(j)-1. Different slots start from approximately uniformly distributed positions to reduce initial competition; each slot eventually scans the entire set, so faster slots can enter intervals that other slots have not yet claimed.

[0076] In step S117, the execution end first checks whether the collection record corresponding to T and i exists for candidate i, and then attempts to create the collection record in an atomic manner of "creating only if the target does not exist". Successful creation means that i has been acquired in the current slot; if it already exists or the competition fails, it means that other slots have already acquired it, and the current slot continues to scan.

[0077] In step S118, some external storage clients may report exclusive creation conflicts as general input / output (I / O) errors. At this point, the retrieval record is checked again: if the record already exists, it is treated as a contention failure; if it still does not exist, the error is reported as a real storage failure.

[0078] Step S119, exclusive acquisition indicates that logical task i has been acquired by an execution slot, used to avoid concurrent duplicate execution. In the basic implementation where the acquisition record does not contain leases, acquisition timeouts, or completion confirmations, the execution slot will not automatically release the acquisition record and retry if it expires after acquisition. Therefore, this method does not guarantee that the task will be executed exactly once after a failure.

[0079] Step S120: After receiving i, the execution terminal creates an independent subdirectory for i in the root directory of the slot, cleans up the old directory with the same name, and sets this directory as the current working directory and running path.

[0080] In step S121, the execution end writes T and i into the current task execution information and updates the task identifier exposed to the user-facing task to the current i. The execution slot index j has already been saved separately when entering the loop and no longer depends on the subsequently updated task execution information field.

[0081] In step S122, the execution end obtains the task packet from the task data channel by pressing T and i. After obtaining the matching packet, it releases the message receiving resources that are no longer in use to avoid long-term user function execution affecting the normal operation of the message channel.

[0082] In step S123, the execution end generates a running record associated with T, i, and the execution node, unpacks the working directory corresponding to T into the independent directory of i, parses and restores the unit to be executed and the parameter load, and then executes the corresponding task.

[0083] In step S124, when the function returns normally, the execution end serializes the result and performs unified payload selection again, and publishes the successful final state associated with T and i; when the function throws an exception, it publishes the failed final state containing exception information.

[0084] In step S125, after task i completes, the execution end restores the slot root directory, removes the running path of the task, deletes the independent directory of i, and continues scanning. Multiple logical tasks share the same execution end runtime process; therefore, the basic implementation isolates directories and task identifiers, but does not guarantee complete isolation of process-level global state or the state of loaded dependencies. An active callback is an active intermediate message during the execution of a remote task, not a completion callback after success / failure.

[0085] like Figure 3 As shown, the active callback steps during execution are as follows: In step S201, before the user function F returns or throws an exception, it may explicitly call the process callback interface 0 or more times, where D is the progress, stage result, indicator or custom intermediate object.

[0086] In step S202, the execution end's process callback interface serializes the intermediate object and reads T and i from the current task running information. The calling end does not expose the callback address to the execution end, but actively returns the status and result through the callback channel.

[0087] Step S203: The callback rate limiting state is based on the current (T, i). When the runtime process reused by the execution end switches from one i to another, the most recent sending time, the number of times it was suppressed, and the time window byte record of the previous task are reset to prevent different logical tasks from consuming each other's callback budget.

[0088] In step S204, the interface checks whether the time since the last actual transmission exceeds the configured minimum interval and removes historical byte records outside the statistical window; transmission continues only if the current payload is still below the window's cumulative byte limit after being added. The minimum interval, statistical window, and cumulative byte limit can all be configured or dynamically adjusted according to system load.

[0089] In step S205, if a callback is suppressed due to time or byte budget constraints, only the number of suppressed calls is accumulated. During the next actual transmission, the previously suppressed calls are carried over with the callback record and cleared. Thus, the calling end can identify merged intermediate updates, and the messaging system does not need to receive all high-frequency update records.

[0090] In step S206, intermediate objects that meet the sending conditions are selected with the same payload size as the unit to be executed, parameters, and results, forming an active callback record carrying T, i, the number of times suppressed, and a unified payload, and written to the status and result channels. The unit to be executed then continues to run; the active callback record does not carry out task completion notification.

[0091] In step S207, after the calling end reads the active callback based on the job identifier T, it recovers the intermediate data through the unified payload reading interface and calls the local processing function associated with the logical task identifier i and the number of times it was suppressed. At this time, the remote function can still continue to calculate; after the execution unit returns or an exception occurs, the system writes a success or failure record separately.

[0092] In step S208, the original calling process or other processes can retrieve the job running status and results from the history record based on the job identifier T. Reading the historical callback again will trigger the corresponding local processing function again. Since the basic implementation does not set an independent callback number or an automatic deduplication mechanism, local processing functions with external side effects should be designed to be repeatedly executed.

[0093] The current implementation does not add a separate sequence number or deduplication key to the active callback; duplicate attachments will re-execute the existing callback, and processing functions with side effects should consider idempotency themselves.

[0094] like Figure 4 As shown, the unified payload structure and sending and reading process are as follows: the unified payload structure contains at least two mutually exclusive branches: the inline payload directly contains serialized bytes; the external payload contains the address or resource identifier of the external storage object.

[0095] The function payload, parameter payload, result payload, and active callback payload on the sending side all call the same selection logic; the receiving side calls the same value retrieval interface, and the upper-layer service is not required to determine the actual transmission path.

[0096] Step S301: Obtain the serialized bytes and calculate the length.

[0097] Step S302: If the length is not greater than the configured threshold, an inline payload is generated and directly enters the task or status message. The threshold is used to reserve protocol and compression overhead before the single message limit of the messaging system, and can be configured according to the messaging system capability, network status, or payload type. The protection scope is not limited to a specific value.

[0098] Step S303: If the length is greater than the threshold, generate a formal object name and a temporary object name in the external payload directory partitioned by date, and write the original bytes into the temporary object after compression and binary encoding.

[0099] Step S304: After the write operation is complete, the temporary object is atomically renamed to the formal object, and then an external storage reference is sent in the message to prevent the receiving end from seeing the half-written object.

[0100] In step S305, if the receiving side obtains the inline payload, it directly returns the bytes; if it obtains the external payload, it opens the reference object, decompresses and restores the bytes, and then hands them over to the upper layer for deserialization.

[0101] The transmission paths for different data types are as follows:

[0102] Transmission thresholds can be configured or dynamically calculated based on messaging system limitations, network status, or payload type.

[0103] like Figure 5 As shown, each logical task i first generates a creation record, followed by a running record and zero or more active callbacks, ultimately resulting in a success, failure, or cancellation record. The batch scheduling layer separately stores the submission success or failure results. Active callbacks only represent intermediate information during execution and are not counted in the number of completed logical tasks.

[0104] Basic implementation: The number of created (C) tasks is equal to the number of successful / failed final states (T). Preferred implementation: After deduplication based on the unique identifier (subtask_id) of each parallel task in the job, each created subtask has a unique final state.

[0105] like Figure 6 As shown, the calling end establishes a receiver that filters by T and can read historical messages before sending the control request. The submitting end obtains the job identifier from the scheduler's submission interface and then publishes a submission success receipt; the calling end only returns T upon receiving this receipt. This mechanism resolves the ambiguity of "T being generated locally but the batch scheduler failing to accept it." The calling end has already written creation records for N logical tasks before sending the control request. If an array submission fails and only generates one batch error, the calling end will not be able to obtain the final status records corresponding to each of the N logical tasks. Therefore, the submitting end first sends the batch submission failure results, and then writes failure records for i=0 to N-1 respectively, ensuring that each created logical task has a clear final status.

[0106] The job status and result acquisition module uses a new reading identity, starting from the historical record position and processing only records where the job identifier matches the target job identifier T. Successful records have their result payloads recovered and handed over to the success processing function, failed records to the failure processing function, and actively returned (callback) records to the process information processing function.

[0107] The basic implementation uses the equality of the number of created logical tasks with the number of success and failure records as the criterion for the completion of all tasks, without automatically deduplicating records based on their unique identifiers. Therefore, it can be described as "being able to read from a historical location again," but should not be described as processing each message only once or automatically deduplicating the final state. A preferred implementation may add a unique record identifier, a record sequence number, and an idempotent state table.

[0108] like Figure 7As shown, environment archives are stored in external storage according to environment identifiers. When a compute node is used for the first time, an environment directory is created in a user-isolated local environment cache; multiple execution slots execute download, decompression, activation, and path relocation serially through environment file locks, and a ready flag is written after all operations are successful. The lock is released by the operating system when the process exits, avoiding long-term persistence of manually locked files.

[0109] This avoids duplicate downloads, concurrent overwriting, and worker processes reading incompletely decompressed runtime environments.

[0110] The execution end creates an independent slot root directory for each execution slot and an independent subdirectory for each claimed slot i. Slot-level standard output and error output are written to logs named according to T, job, and slot identifier; upon exiting the processor, it attempts to upload the logs to the external storage directory corresponding to T. If successful, the local logs and task directory are deleted; if it fails, the local logs are retained for troubleshooting.

[0111] For external storage requiring authentication, the calling end places the authentication materials in a control request encrypted with a public key. The submitting end decrypts the request and transmits it only to the executing end via controlled task execution information; the executing end deletes the temporary authentication field immediately after completing identity initialization. The public key sealing mechanism protects the confidentiality and integrity of the control request, but simply possessing the public key cannot prove the calling end's identity; the basic implementation should not claim to have complete calling end identity authentication.

[0112] like Figure 8 As shown, the calling end first finds the scheduler job identifier corresponding to T from the status history, and then sends a cancellation request in encrypted form. The submitting end verifies the job identifier format, queries the job owner and job name, and only cancels jobs whose names contain the system's identifier, reducing the risk of incorrect identifiers causing the accidental cancellation of other business jobs.

[0113] Optional confirmation semantics: The submission service publishes the final state of cancellation confirmation after the cancellation task (scancel) is successful or unsuccessful.

[0114] After sending a cancellation request, the calling end creates a cancellation request record. This record indicates that a cancellation request has been submitted, but it does not mean that the submitting end has confirmed the scheduler's cancellation operation was successful. The submitting end can further create a cancellation result record after the cancellation is successful or unsuccessful.

[0115] Example 1 1. N logical tasks are processed by P execution slots. The calling end submits a function F and multiple sets of parameters; the submitting end calculates P based on the request parallelism and the submitting end's upper limit, and submits only P array execution slots. The executing end determines the initial scan position in slot j based on the total number of tasks, slot index, and slot quantity. After successfully establishing an exclusive claim record corresponding to logical task i, it executes the corresponding task and continues scanning after completion. Ultimately, the state still represents all logical tasks, but the batch scheduler only maintains P array elements.

[0116] 2. Active callback during execution After an execution slot receives logical task i, it writes the job identifier T and logical task identifier i into the current task execution information. When the unit to be executed completes data loading or reaches a predetermined progress, it calls the active callback interface during execution. Calls that meet the sending conditions form an active callback record containing T, i, the number of suppressed attempts, and intermediate payloads, which is processed locally by the calling end immediately; a success record is formed separately after the unit to be executed returns. The execution slot then receives other logical tasks and resets the active callback sending control state for the new tasks.

[0117] 3. High-load process feedback If the serialized phased object generated by the execution unit exceeds the payload threshold, the execution end, when the byte budget allows for transmission, compresses the object, writes it to external storage, and atomically publishes the formal object. The active callback message only carries a reference to the external storage. Upon receiving it, the calling end restores the object through a unified interface. Therefore, the large active callback does not increase the single message capacity of the message broker.

[0118] 4. Handling Submission Failures The calling end has written N logical task creation records and task data, but the cluster refuses to schedule the submission due to partition or resource parameter errors. The submitting end sends a submission failure result so that the calling end can immediately receive the failure information; at the same time, it writes a failure record for each of the N logical tasks so that when the calling end retrieves the result again according to the job identifier T, it can confirm that all logical tasks have been completed.

[0119] 5. Reacquire job running status and results after connection interruption. If the original calling process exits during task execution, another calling process can retrieve previous running records, callbacks, and final status records from the historical location based on the job identifier T. The basic implementation will again call the local processing function corresponding to the historical callback; therefore, when writing to the database or sending notifications, this processing function should use the job identifier, logical task identifier, and business identifier to avoid duplicate side effects.

[0120] The key points for implementation are as follows: 1. Exclusive claim records do not include leases and completion markers; if the executor becomes invalid after claiming, the corresponding logical task may no longer be executed. Therefore, this method is used to avoid concurrent duplicate claims and is not equivalent to guaranteeing that the task will be executed exactly once.

[0121] 2. No automatic retry is set for logical tasks; after a single logical task fails, a failure record is generated, and the execution end continues to process other tasks without automatically deleting the corresponding exclusive claim record and re-executing the task.

[0122] 3. The execution end sets an independent directory and task identifier for each logical task, but multiple logical tasks can share runtime processes, dependency caches and global process state; when stronger isolation is required, independent processes, containers or execution sandboxes can be used.

[0123] 4. Active callback is not a task completion callback: it is explicitly called by the user function during execution; when the execution unit returns normally or an exception occurs, the system will generate a separate success or failure record.

[0124] 5. Local processing can be triggered again when retrieving historical active callbacks; when it is necessary to avoid duplicate side effects, idempotent processing can be performed based on job identifier, logical task identifier, business identifier, or record sequence number.

[0125] 6. The execution-side process callback interface does not generate a return record if a valid T or i cannot be read; this interface does not assume the responsibilities of the calling or submitting end.

[0126] 7. A unified payload is used for the unit to be executed, parameters, results, and active callbacks; the working directory and logs are stored externally, and control requests are transmitted through the control message channel.

[0127] 8. Determine whether all tasks have ended based on the number of records created for the logical tasks and the number of successful and failed records; if idempotent processing of records is required, record sequence numbers or deduplication information can be further set.

[0128] 9. The cancellation record issued by the calling end is used to notify that a cancellation request has been submitted, and does not indicate that the submitting end has confirmed that the batch job has been successfully cancelled; the confirmation semantics can be generated separately by the submitting end to form a cancellation result record.

[0129] 10. Public key encryption is used to protect the content of control requests, but it does not constitute complete sender authentication on its own; it can be combined with caller signatures or fine-grained authorization when needed.

[0130] 11. The submitting end receives control requests from the current read position of the control channel; when it is necessary to process backlogged requests during downtime, a persistent read position or acknowledgment mechanism can be used.

[0131] 12. Serialization of Executable Units is oriented towards trusted content: the executable object payload cannot be used with untrusted input; the Executable Units typically also need to be recoverable in the execution environment or the uploaded working directory.

[0132] 13. The job identifier T can be generated based on time information, or it can be combined with tenant identifiers, random numbers, or monotonic sequences to enhance uniqueness.

[0133] The following are alternative solutions: 1. Atomic retrieve records can be stored in distributed file systems, object stores, databases, coordination services, or key-value systems that support conditional writes, unique key insertions, compare-and-swap, or leases; 2. Circular scan can be replaced by random starting point, hash sharding, work stealing, central waiting queue or hierarchical priority scan, as long as the execution slot and logical task are not fixed one-to-one mapping; 3. Exclusive claim records can be recorded with lease agreements, heartbeats, and completion status, allowing the claimant to claim again after the lease expires; 4. The execution end can reuse processes, threads, containers, or execution sandboxes; if stronger isolation is required, a child process can be started for each i, while retaining the slot-level loop; 5. During execution, the required current job identifier and logical task identifier are actively returned, which can be stored in the task execution information field, thread local variables, coroutine local variables, or explicit task execution information objects; 6. The callback handler can be a synchronous function, an asynchronous function, an event listener, or a downstream message subscriber; 7. Fixed-size thresholds can be replaced with dynamic thresholds determined based on proxy limits, compression ratios, network congestion, load type, or real-time load; 8. External payloads can be deduplicated by content summary, so that the same payload of the unit to be executed is saved only once and shared by multiple logical tasks; 9. The historical status and results of a job can be saved in a message log, a record database, or an append-only storage object. Record sequence numbers, read positions, and idempotency tables can be set to allow continued reading from a specified position. 10. The function to be executed can be replaced with an execution unit version identifier plus a pre-deployment entry point, a container image entry point, a script, or a language-independent execution unit; 11. The cluster job scheduling management system (Slurm) can be replaced by other schedulers that can request P batch execution resources, and the array slots can also be represented by P homogeneous jobs in the same batch.

[0134] This invention connects RPC-style remote function calls with distributed resource allocation. The calling end submits the unit to be executed, parameters, and resource requirements. The submitting end transforms the resource requirements into a scheduling request from the cluster job scheduling management system (Slurm). The executing end executes the task in the computing resources actually allocated by the scheduler. The system internally completes serialization, scheduling submission, task transmission, status collection, and result and active callback delivery, so that the caller does not need to directly face the production and consumption process of the task queue.

[0135] The main advantages of the embodiments of the present invention are concentrated in the following three aspects: 1. Dynamic Task Acquisition: The submitting end only requests P execution slots, and the execution slot index j and the logical task identifier i are independent of each other; each execution end scans the task set from different starting points and dynamically acquires unexecuted logical tasks through cross-node atomic exclusive recording. After an execution slot completes its current task, it continues to acquire other tasks. 2. Active Callback During Execution: Intermediate information associated with the current job identifier T and logical task identifier i can be sent even before the user function returns; the calling end can process it immediately or retrieve the intermediate information again based on T after the connection is interrupted; 3. Unified message routing for different sizes: The execution unit, parameters, results and active callbacks use the same discriminative payload structure. The sending side selects whether to store the message inline or externally based on the size after serialization, and the receiving side recovers it through a unified interface.

[0136] The technical solution of this invention employs a language runtime that can serialize execution units, a persistent messaging system with historical retrieval capabilities, multi-node accessible external storage, and a resource scheduling system that supports batch job arrays. It can be replaced by other language runtimes, message middleware, object storage, shared file systems, database large object storage, or batch processing resource schedulers with corresponding functions. It converts batch remote function calls into batch job array execution slots with controlled quantity, and the execution slots dynamically pick up logical tasks, actively send back intermediate information during task execution, and select transmission paths according to the load size.

[0137] Beneficial effects 1. Reduce array size and control plane pressure: N logical tasks generate only P array execution slots. When N=100000 and P=200, the number of array elements is reduced from 100000 to 200, rather than simply limiting concurrency to 200.

[0138] 2. Improve execution slot utilization: After a faster execution slot completes its current task, it can continue to pick up tasks that have not yet started, without having to wait idly for long tasks corresponding to other slots due to fixed mapping.

[0139] 3. Reuse startup overhead: The same runtime process on the execution side processes multiple logical tasks sequentially, reducing the redundant overhead of runtime startup, execution environment activation and communication component exit waiting.

[0140] 4. Avoid concurrent duplicate claims: Atomic exclusive creation visible across nodes ensures that only one successful claimer exists for the same i at any given time.

[0141] 5. Maintain correct attribution under dynamic mapping: Update the current job identifier T and logical task identifier i before each execution, and reset the active callback control status by i to avoid incorrectly attributing intermediate information and final status to execution slot index j.

[0142] 6. Provides process observability during task execution: The caller can receive progress and stage objects before the function returns, without needing to poll logs or wait for the final result.

[0143] 7. Impact of Controlled Active Callback on the System: Feedback on time intervals, window byte budgets, and number of suppressed events suppresses high-frequency or high-volume events while preserving observability.

[0144] 8. Balancing small object transmission latency and large object capacity: Small payloads go directly into the messaging system, while large payloads are automatically transferred to external storage, avoiding message overload and reducing the overhead of writing all objects to external storage.

[0145] 9. Reduce reliance on continuous online access of the calling end: The job running status and results are saved according to the job identifier T. After the original calling process exits, other processes can still retrieve the active callback, success results and failure information according to T.

[0146] 10. Clearly distinguish between local task preparation completion and scheduler acceptance success: The caller confirms submission only after receiving the scheduler's successful submission result; when submission fails, the caller can immediately obtain both the batch failure result and the failure records of each of the N logical tasks.

[0147] The above effects are achieved through controlled execution slots, dynamic exclusive acquisition, current task running information, active feedback during execution, unified load, and re-readable job status and result records.

Claims

1. A batch processing computing system with active data return and large / small load splitting, characterized in that, include: The calling end receives the unit to be executed, parameter sequence, working directory and resource requirements, organizes them into N logical tasks, generates job identifier T and logical task identifier i. The load selection module serializes the parameters with logical task identifier i in the unit to be executed and parameter sequence, performs load selection, builds and publishes task data and control requests. The submitting end verifies the control request, calculates the number of execution slots P, requests P array execution slots from the batch resource scheduler, transmits the job identifier T, the total number of logical tasks N and the number of execution slots P to the execution environment, generates a task description and submits / cancels the job to the resource scheduler. At the execution end, array slot index j is provided to P array execution slots. The initial scan position is the integer obtained by rounding down N×j / P. Each array execution slot scans from staggered starting positions. Each array execution slot first checks whether the retrieval record corresponding to the job identifier T and logical task identifier i exists. If it does not exist, it retrieves the task data containing job identifier T, logical task identifier i, and load. The next array execution slot continues to scan and executes user functions. During the execution process, user functions can explicitly call the active return interface to read the job identifier T and logical task identifier i from the current task running information. The load selection module serializes the intermediate object and then performs load selection to form an active return record carrying job identifier T, logical task identifier i, and unified load, and writes it into the process and final state records to obtain the execution result. When selecting a load, if the serialized byte length is not greater than the configured threshold, an inline load branch is selected; if the serialized byte length is greater than the configured threshold, an external load branch that is mutually exclusive with the inline load branch is selected, and load splitting is performed.

2. The batch processing computing system with active backhaul and large / small load splitting as described in claim 1, characterized in that, The execution terminal establishes an independent slot root directory for each array execution slot and an independent subdirectory for each acquired logical task identifier i. After the logical task ends, the slot root directory is restored, the running path of the task is removed, and the independent subdirectory of the logical task is deleted.

3. The batch processing computing system with active backhaul and large / small load splitting as described in claim 1, characterized in that, The array slot index j and the logical task identifier i are dynamically related when the task is retrieved.

4. The batch processing computing system with active backhaul and large / small load splitting as described in claim 1, characterized in that, The calling end compresses the working directory and uploads it to the external storage directory corresponding to the job identifier T.

5. The batch processing computing system with active backhaul and large / small load splitting as described in claim 1, characterized in that, The calling end constructs a public-key encrypted control request, which includes job identifier, logical task, request parallelism, executing user, operating environment, and resource requirements.

6. The batch processing computing system with active backhaul and large / small load splitting as described in claim 5, characterized in that, The submitting end uses the internally stored private key to decrypt and verify the control request.

Images