Spark asynchronous task submission method and device and storage medium

Abstract

The invention discloses a Spark asynchronous task submission method and device and a storage medium, and relates to the technical field of the Internet, and the method comprises the steps: selecting an idle target session for a Spark asynchronous task from a session multiplexing pool based on task request parameters in the Spark asynchronous task, the session multiplexing pool comprising a plurality of Spark sessions created through a Live server; forwarding a plurality of execution statements to be executed in the Spark asynchronous task to the target session by using a target thread pool associated with the target session, and concurrently executing each execution statement based on a Spark context packaged in the target session; after it is detected that all the execution statements in the Spark asynchronous task are executed, the target session is recycled to the session multiplexing pool, the technical problem that in the prior art, the task execution efficiency is low due to the fact that sessions cannot be multiplexed in submission of the Spark task is solved, and the execution efficiency and the resource utilization rate of the Spark asynchronous task are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, and particularly relates to a Spark asynchronous task submission method, device and storage medium. BACKGROUND

[0002] In the Apache Spark big data processing ecology, Apache Livy can provide the client with the ability of asynchronously submitting Spark tasks, and its native working mode is to create and manage an independent Spark session for each task, and the creation of the session needs to go through the processes of SparkContext initialization and JVM startup, and the cold start delay is as high as 10-30 seconds, after the execution of a task, the current session is destroyed, and a new session is created. The session is created frequently, and the overhead is large, and the resource is wasted in the process of frequently creating / destroying the session, and the delay of the task is high.

[0003] The above content is only used to assist in understanding the technical solutions of the present application, and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a Spark asynchronous task submission method, device and storage medium, which aims to solve the technical problem of low task execution efficiency caused by the unrecycled session in the existing Spark task submission.

[0005] To achieve the above purpose, the present application provides a Spark asynchronous task submission method, which comprises: receiving a Spark asynchronous task submitted by a client, the Spark asynchronous task comprising a plurality of execution statements to be executed and task request parameters; selecting an idle target session for the Spark asynchronous task from a session reuse pool based on the task request parameters, the session reuse pool comprising a plurality of Spark sessions created through a Livy server, and the Spark session encapsulating an initialized Spark context; using a target thread pool associated with the target session to forward each execution statement in the Spark asynchronous task to the target session, and concurrently executing each execution statement based on the Spark context encapsulated in the target session; recycling the target session to the session reuse pool after detecting that all execution statements in the Spark asynchronous task are executed.

[0006] In an embodiment, the step of selecting an idle target session for the Spark asynchronous task from the session multiplex pool further comprises the steps of: reading session pool configuration parameters of the session multiplex pool, the session pool configuration parameters at least including an initial session number; based on the session pool configuration parameters, initiating a session pool creation request to the Livy server, pre-creating the initial session number of Spark sessions, and adding the successfully created Spark sessions to an available session queue of the session multiplex pool, completing the initialization of the session multiplex pool.

[0007] In an embodiment, the session pool configuration parameters of the session multiplex pool are obtained, and based on the session pool configuration parameters, the session idle time threshold and the minimum total number of sessions of the session multiplex pool are determined; periodically obtaining the idle time of the idle Spark sessions in the available session queue of the session multiplex pool, and destroying the Spark sessions whose idle time exceeds the session idle time threshold through the Livy server, until the number of Spark sessions in the session multiplex pool reaches the minimum total number of sessions.

[0008] In an embodiment, the abnormality detection parameters of each Spark session in the session multiplex pool are periodically collected, the abnormality detection parameters at least including Spark context state, heartbeat response time, historical execution failure number, and resource load parameter; For each of the abnormality detection parameters of the Spark sessions, the abnormality detection parameters of any one of the Spark sessions are identified; In the case where the abnormality detection parameters of any one of the Spark sessions are determined to exceed the corresponding parameter threshold range, the corresponding Spark session is confirmed as the abnormal session, and the abnormal session is destroyed through the Livy server.

[0009] In an embodiment, the session pool configuration parameters of the session multiplex pool are obtained, and based on the session pool configuration parameters, the maximum total number of sessions and the minimum total number of sessions of the session multiplex pool are determined; periodically collecting session-level load data corresponding to the session multiplex pool, the load data including the number of active Spark sessions, busy / idle time period, and task queue length associated with each of the Spark sessions in the used session queue of the session multiplex pool, the busy / idle time period being a peak period or a low period, the peak period and the low period being determined by the time period of historical load data; based on the task queue length associated with each of the Spark sessions, calculating an average value of the task queue length corresponding to the session multiplex pool; If the busy time period is the peak time, when the task queue length average value is greater than or equal to a first preset length, and the number of Spark sessions in the current session multiplexing pool is less than the maximum session total number, a Spark session is added through the Livy server until the number of Spark sessions reaches the maximum session total number. If the busy time period is the off-peak time, when the number of active Spark sessions continuously for a preset time period is less than the minimum session total number, an idle Spark session is destroyed through the Livy server until the number of Spark sessions reaches the minimum session total number.

[0010] In an embodiment, the step of forwarding each execution statement in the Spark asynchronous task to the target session by using the target thread pool associated with the target session comprises: each execution statement in the Spark asynchronous task is encapsulated into a corresponding executable task unit; each executable task unit is submitted to the target thread pool; each executable task unit is pushed to the target session in parallel by a worker thread in the target thread pool through a REST interface of the Livy server.

[0011] In an embodiment, the step of concurrently executing each execution statement based on the Spark context encapsulated in the target session comprises: based on the parallelism parameter of the target thread pool, a corresponding number of executors are assigned to each execution statement by the Spark context to drive synchronous and concurrent execution of each execution statement; the computing load of the Spark job triggered by each execution statement is detected periodically; when the computing load is detected to exceed a preset resource shortage threshold, the number of executors is increased until the number of executors reaches a maximum number of executors; when the idle time of an idle executor is detected to exceed a preset idle threshold, the idle executor is released until the number of executors reaches a minimum number of executors.

[0012] In an embodiment, the step of selecting an idle target session for the Spark asynchronous task from a session multiplexing pool based on the task request parameter comprises: the task request parameter is parsed to extract the task resource requirement of the Spark asynchronous task; Using the task resource requirements as a filtering condition, candidate idle sessions whose resource configurations meet the filtering conditions are selected from the available session queue of the session reuse pool. The available session queue contains multiple idle Spark sessions. Obtain the historical execution failure rate, idle time, heartbeat response time and resource load parameters of each candidate idle session, and calculate the health score of each candidate idle session based on the historical execution failure rate, idle time, heartbeat response time and resource load parameters; The candidate idle session corresponding to the one with the highest health score value is selected as the target session; If the available session queue is empty and the current number of Spark sessions in the session reuse pool has not reached the preset maximum total number of sessions, the session expansion process is triggered. A new Spark session is created through the Livy server, the Spark context of the new Spark session is initialized, and the new Spark session is used as the target session.

[0013] Furthermore, to achieve the above objectives, this application also proposes a Spark asynchronous task submission device, which includes: The task acquisition module is used to receive Spark asynchronous tasks submitted by the client. The Spark asynchronous tasks include multiple execution statements to be executed and task request parameters. The session selection module is used to select an idle target session for the Spark asynchronous task from the session reuse pool based on the task request parameters. The session reuse pool includes multiple Spark sessions created by the Livy server, and each Spark session encapsulates an initialized Spark context. The task execution module is used to forward each execution statement in the Spark asynchronous task to the target session using the target thread pool associated with the target session, and concurrently execute each execution statement based on the Spark context encapsulated within the target session. The session recycling module is used to recycle the target session to the session reuse pool after detecting that all the execution statements in the Spark asynchronous task have been executed.

[0014] In addition, to achieve the above objectives, this application also proposes a Spark asynchronous task submission device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the Spark asynchronous task submission method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the Spark asynchronous task submission method as described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the Spark asynchronous task submission method as described above.

[0017] The one or more technical solutions proposed in this application have at least the following technical effects: They receive Spark asynchronous tasks submitted by clients, where each Spark asynchronous task includes multiple execution statements to be executed and task request parameters; instead of creating a new Spark session for each task, they select an idle target session for the Spark asynchronous task from a pre-created and maintained session reuse pool based on the task request parameters. The target session has already been initialized by the Livy server, and its internal Spark context is ready for immediate use. Utilizing the target thread pool associated with the target session, each execution statement in the Spark asynchronous task is forwarded to the target session, and each execution statement is executed concurrently based on the Spark context encapsulated within the target session; after all execution statements in the Spark asynchronous task have been executed, the target session is recycled to the session reuse pool for reuse by subsequent tasks, avoiding the overhead of repeated creation. Through session reuse, concurrent submission and parallel execution of each execution statement in the Spark asynchronous task, and intelligent session recycling, the technical problem of low task execution efficiency caused by the inability to reuse sessions during Spark task submission in existing technologies is solved, further exploring the concurrency potential of a single session and significantly improving the execution efficiency and resource utilization of Spark asynchronous tasks. The Spark asynchronous task submission method provided in this application receives Spark asynchronous tasks containing multiple execution statements and selects a matching idle target session from a pre-built session reuse pool for the Spark asynchronous task, avoiding the repeated creation of already initialized Spark contexts and eliminating startup delays. Next, the target thread pool bound to the target session forwards the multiple execution statements to the target session, triggering parallel execution of the multiple statements within the target session's Spark context, fully releasing the cluster's computing power. Finally, after all execution statements have been completed, the target session is recycled back to the reuse pool, ensuring session reusability. Thus, through a closed-loop mechanism of session reuse, concurrent submission and parallel execution of each execution statement in a Spark asynchronous task, and cyclical session recycling, the overhead of frequent session start-stop is avoided, and the performance bottleneck of single-session serial execution is overcome, significantly improving task throughput and resource utilization efficiency, thereby solving the technical problem of low task execution efficiency caused by the inability to reuse sessions in existing Spark task submissions. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the Spark asynchronous task submission method in this application. Figure 2 This application provides a logical diagram illustrating the concurrent execution of multiple statements within a single Spark session. Figure 3 A schematic diagram illustrating the management logic of a session multiplexing pool provided in this application; Figure 4 This is a schematic diagram of the module structure of the Spark asynchronous task submission device according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware operating environment involved in the Spark asynchronous task submission method in this application embodiment. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application embodiment is as follows: receiving a Spark asynchronous task submitted by a client, the Spark asynchronous task includes multiple execution statements to be executed and task request parameters; based on the task request parameters, selecting an idle target session for the Spark asynchronous task from the session reuse pool, the session reuse pool includes multiple Spark sessions created through the Livy server, and each Spark session encapsulates an initialized Spark context; using the target thread pool associated with the target session, forwarding each execution statement in the Spark asynchronous task to the target session, and concurrently executing each execution statement based on the Spark context encapsulated inside the target session; after detecting that all execution statements in the Spark asynchronous task have been executed, reclaiming the target session to the session reuse pool.

[0024] In this embodiment, for ease of description, the Spark asynchronous task submission system will be used as the execution subject for the following explanation.

[0025] In the Apache Spark big data processing ecosystem, Apache Livy provides clients with the ability to asynchronously submit Spark tasks. Its native working mode involves creating and managing an independent Spark session for each task. Creating a session requires SparkContext initialization and JVM startup, resulting in a cold start latency of 10-30 seconds. After executing a task, the current session is destroyed, and a new session is created. Frequent session creation incurs significant overhead, leading to resource waste and high task latency during this process.

[0026] This application provides a solution that receives Spark asynchronous tasks containing multiple execution statements and selects a matching idle target session from a pre-built session reuse pool for the Spark asynchronous task, avoiding the repeated creation of already initialized Spark contexts and eliminating startup latency. Next, the target thread pool bound to the target session forwards the multiple execution statements to the target session, triggering parallel execution of the multiple statements within the target session's Spark context, fully releasing the cluster's computing power. Finally, after all execution statements have been completed, the target session is recycled back to the reuse pool, ensuring session reusability. Thus, through a closed-loop mechanism of session reuse, concurrent submission and parallel execution of individual execution statements in Spark asynchronous tasks, and cyclical session recycling, the overhead of frequent session start-stop is avoided, and the performance bottleneck of single-session serial execution is overcome, significantly improving task throughput and resource utilization efficiency, thereby solving the technical problem of low task execution efficiency caused by the inability to reuse sessions in existing Spark task submissions.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a Spark asynchronous task submission device capable of implementing the above functions. The following description uses the Spark asynchronous task submission system as an example to illustrate this embodiment and the subsequent embodiments.

[0028] Based on this, this application provides a Spark asynchronous task submission method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the Spark asynchronous task submission method of this application.

[0029] In this embodiment, the Spark asynchronous task submission method includes steps 101-104: Step 101: Receive the Spark asynchronous task submitted by the client. The Spark asynchronous task includes multiple execution statements to be executed and task request parameters.

[0030] Specifically, the client is the terminal / application that initiates the Spark task submission request, such as a big data platform frontend, data analysis tool, or API caller. The client is the initiator of the task request. Spark asynchronous tasks are non-blocking computation tasks based on the Spark framework. After submission, Spark asynchronous tasks do not require waiting for completion and can return immediately. They execute asynchronously in the background, and the results can be obtained through callbacks / polling. Multiple execution statements constitute the smallest unit of execution for a Spark task. These statements can be Spark Structured Query Language (SQL) statements or Spark computation code segments written in Scala / Python (such as data cleaning, aggregation, and analysis statements). Multiple execution statements work together to complete a complete business computation logic. Task request parameters are configuration information for Spark asynchronous tasks, including but not limited to task priority, task resource requirements (CPU / memory specifications), execution timeout, and task identifier. These parameters are the core basis for task request parameter session selection and task resource allocation.

[0031] In some embodiments, a REST API interface supporting multiple protocols is deployed in the Spark asynchronous task submission system as the task submission entry point. Clients submit task data (i.e., Spark asynchronous tasks) containing a list of execution statements and task request parameters through this interface. The protocol can be Hypertext Transfer Protocol (HTTP), Hypertext Transfer Protocol Secure (HTTPS), Remote Procedure Call (RPC), etc. The system first verifies the validity of the Spark asynchronous task execution request submitted by the client, including verifying the completeness of the request format, the syntax compliance of the execution statements (such as basic syntax checks for Spark SQL statements), and the completeness of required parameters (task identifier, task resource requirements). If the verification fails, a standardized error message is returned. Once the Spark asynchronous task verification passes, all execution statements and task request parameters to be executed in the Spark asynchronous task are parsed and extracted. The request parameters can also be mapped to a standardized internal system structure. Subsequently, a globally unique ID is assigned to the Spark asynchronous task and encapsulated as a task object. Simultaneously, an acceptance receipt containing the task ID is returned to the client, completing the task reception process, ready to trigger subsequent session matching and concurrent execution processes. By standardizing users' one-time batch analysis requirements, a structured prerequisite is provided for enabling session reuse and concurrent execution of multiple statements.

[0032] Step 102: Based on the task request parameters, select an idle target session for the Spark asynchronous task from the session reuse pool. The session reuse pool includes multiple Spark sessions created through the Livy server. Each Spark session encapsulates an initialized Spark context.

[0033] Specifically, the session reuse pool is a Spark session management container built on an object pool pattern. It pre-stores multiple healthy Spark sessions created and maintained by the Livy server to avoid the overhead of creating a new session for each task submission. The Livy server acts as an intermediary service connecting the client and the Spark cluster, responsible for the creation, destruction, and state management of Spark sessions. The Livy server is the core carrier for creating Spark sessions in the session reuse pool. A Spark session is an independent runtime environment for Spark task execution. Each Spark session corresponds to a unique execution context, and the Spark session is the basic unit for isolated execution of Spark tasks. An initialized Spark context is the core component of a Spark session, containing Spark cluster connection information, executor configuration, runtime environment, etc. After initialization, tasks can be executed directly without repeatedly establishing cluster connections. Idle target sessions are Spark sessions in the session reuse pool that have not been assigned tasks, have a normal Spark context state, and whose resources meet the current task's requirements.

[0034] In some embodiments, task request parameters are parsed to extract key information such as task resource requirements (CPU / memory specifications), task priority, and execution timeout, and session filtering rules are constructed. Then, the status of each Spark session in the session reuse pool is queried. This session reuse pool pre-stores multiple Spark sessions created and maintained by the Livy server, and each Spark session encapsulates an initialized Spark context. The system filters target sessions marked as "idle," with normal Spark context status and resource configuration meeting the filtering rules from the session reuse pool, marks the target sessions as "assigned," and binds them to the current Spark asynchronous task. Through a precise matching mechanism based on task request parameters, suitable idle Spark sessions are selected from the pre-built session reuse pool to provide an execution environment for subsequent statement forwarding, achieving session reuse to eliminate the cold start time of creating new sessions.

[0035] Optionally, the steps for selecting an idle target session for a Spark asynchronous task from the session reuse pool based on the task request parameters include: Parse the task request parameters to extract the task resource requirements of the Spark asynchronous task; Using task resource requirements as a filtering condition, candidate idle sessions whose resource configurations meet the filtering conditions are selected from the available session queue of the session reuse pool. The available session queue contains multiple idle Spark sessions. Obtain the historical execution failure rate, idle time, heartbeat response time and resource load parameters of each candidate idle session, and calculate the health score of each candidate idle session based on the historical execution failure rate, idle time, heartbeat response time and resource load parameters; Select the candidate idle session with the highest health score as the target session; If the available session queue is empty and the current number of Spark sessions in the session reuse pool has not reached the preset maximum total number of sessions, the session expansion process is triggered. A new Spark session is created through the Livy server, the Spark context of the new Spark session is initialized, and the new Spark session is used as the target session.

[0036] Specifically, task resource requirements are the specific computational resource requirements of the Spark asynchronous task extracted from the task request parameters, which may include key indicators such as the number of Executors, CPU cores, memory capacity, and parallelism. The available session queue is a collection of idle Spark sessions maintained in the session reuse pool, serving as the source for filtering candidate idle sessions. The historical execution failure rate is the ratio of the number of failed executions of historical statements to the total number of executions for the corresponding Spark session, reflecting its stability. Idle time is the idle time of the corresponding Spark session from the completion of the previous task and its return to the session reuse pool until the current filtering; excessive idle time may lead to abnormal session status. Heartbeat response time is the time it takes to receive a normal response after sending a heartbeat detection request to the Spark session, reflecting the connectivity between the Spark session and the Livy server and Spark cluster. Resource load parameters are indicators of the current resource utilization status of the session, including the remaining CPU cores, remaining memory capacity, and executor idle ratio, reflecting the resource redundancy of the session in handling new tasks.

[0037] For example, firstly, the task request parameters are parsed to extract the task resource requirements of the Spark asynchronous task (which may include key indicators such as the number of executors, the number of CPU cores, and memory capacity), and these requirements are transformed into standardized filtering rules. Then, the available session queue (which stores idle Spark sessions) in the session reuse pool is queried. Using the task resource requirements as the filtering condition, the resource configuration information of all Spark sessions in the available session queue is traversed to select candidate idle sessions whose resource configurations fully meet the task resource requirements. For each candidate idle session, the system retrieves multi-dimensional operational metrics, including the historical execution failure rate (e.g., the percentage of task execution failures in the past 24 hours), idle time (the idle time after being reclaimed from the previous task), heartbeat response time (the connection time between the session and the Livy server), and resource load parameters (remaining CPU / memory percentage). Based on preset rules, a single-dimensional score is determined for each operational metric. Each single-dimensional score is then weighted according to preset weights (e.g., 0.4 for failure rate, 0.25 for heartbeat response time, 0.25 for resource load, and 0.1 for idle time) to obtain a health score for each candidate session. The candidate session with the highest health score is selected as the target session, its status is updated to "assigned," and it is bound to the current Spark asynchronous task. If the available session queue is empty, the system checks whether the current number of Spark sessions in the session reuse pool has reached the preset maximum number of sessions. If the current number of Spark sessions in the session reuse pool has not reached the upper limit (i.e., the preset maximum number of sessions), the session expansion process is triggered. The session creation interface of the Livy server is called, passing in configuration parameters that match the task resource requirements. A new Spark session is created, and Spark context initialization is automatically completed (including cluster connection establishment, executor configuration loading, etc.). After initialization, the new Spark session is assigned as the target session to the Spark asynchronous task, and simultaneously added to the session reuse pool for management. If the current number of Spark sessions in the session reuse pool has reached the preset maximum number of sessions, the current Spark asynchronous task can be temporarily stored in the waiting queue until a free Spark session is released from the session reuse pool for re-selection. Under the premise of meeting task resource requirements, the healthiest and most efficient sessions are selected from the session reuse pool, and dynamic expansion is performed when resources are insufficient. This helps improve the success rate and performance of a single task, ensuring the system's elastic service capability under high load.

[0038] Step 103: Using the target thread pool associated with the target session, forward each execution statement in the Spark asynchronous task to the target session, and execute each execution statement concurrently based on the Spark context encapsulated within the target session.

[0039] Specifically, in this application, the system maintains a dedicated thread pool locally for each Spark session to concurrently process statement submission tasks sent to that Spark session, ensuring that submissions and executions between different Spark sessions are isolated from each other. The target thread pool is a dedicated thread scheduling component bound one-to-one with the target Spark session, containing a preset number of worker threads, which can be responsible for asynchronously forwarding execution statements, controlling the statement submission rate, and avoiding blocking caused by single-threaded forwarding.

[0040] In some embodiments, when it is confirmed that the target thread pool bound to the target session is active, the multiple execution statements contained in the Spark asynchronous task can be distributed to the idle worker threads of the target thread pool. The target thread pool forwards each execution statement asynchronously to the target session in a multi-threaded parallel manner, avoiding the blocking problem of single-threaded serial forwarding. After receiving multiple execution statements, the target session starts the execution process based on its internally initialized Spark context. The Spark context parses each execution statement and combines it with preset parallelism parameters to allocate the task to the associated executor cluster, enabling the simultaneous execution of multiple execution statements. At the same time, the execution status of each execution statement is fed back in real time to ensure that the execution process is controllable and effectively shorten the total execution time of the Spark asynchronous task. By using the dedicated thread pool of the target session and the concurrent execution of each execution statement, the performance bottleneck of single session being able to process statements serially in traditional solutions is solved, thereby improving the task throughput of a single session, effectively shortening the overall execution latency of the Spark asynchronous task, and fully releasing the parallel processing potential of cluster computing resources.

[0041] refer to Figure 2 , Figure 2 This diagram illustrates the logic of concurrent execution of multiple statements within a single Spark session. It clearly shows the running mode of execution statements in Spark session reuse scenarios. Multiple execution statements (execution statement 1 to execution statement n) are asynchronously forwarded through the thread pool associated with the same Spark session. Then, relying on the Spark context encapsulated within the Spark session, they send computation requests to the Spark cluster, ultimately achieving parallel execution of multiple execution statements in the Spark cluster.

[0042] Optionally, the steps of forwarding individual execution statements in a Spark asynchronous task to the target session using the target thread pool associated with the target session include: Encapsulate each execution statement in a Spark asynchronous task into a corresponding executable task unit; Submit each executable task unit to the target thread pool; By calling the REST interface of the Livy server through worker threads in the target thread pool, each executable task unit is pushed to the target session in parallel.

[0043] Specifically, an executable task unit is the smallest standardized execution carrier encapsulated from a single execution statement in a Spark asynchronous task. It can contain the execution statement, its unique identifier, execution timeout, parameter context, and other information, adapting to the Livy server-side interface call specification. The Livy server's REST interface is a standardized HTTP interface provided by the Livy service, such as POST / sessions / {sessionId} / statements. The REST interface is the core entry point for submitting execution statements to the Spark session and supports asynchronous / synchronous call modes. Worker threads are thread instances in the target thread pool that actually execute the task push operation. Each worker thread independently handles the forwarding process of one executable task unit, avoiding single-threaded blocking.

[0044] For example, the system iterates through all execution statements contained in a Spark asynchronous task, assigns a unique identifier to each statement, and encapsulates the statement text, execution timeout, parameter context, and other information into standardized executable task units conforming to the Livy server interface specification. Then, all encapsulated executable task units are submitted in batches to the target thread pool bound to the target session. The target thread pool allocates tasks according to a preset number of worker threads; idle public threads directly accept executable task units, and any unallocated executable task units are placed in a waiting queue to be automatically acquired after a thread is released. Next, worker threads in the target thread pool independently initiate requests, calling the statement submission REST interface provided by the Livy server to push the corresponding executable task units to the target session in parallel. Each public thread is responsible for the push process of one executable task unit, avoiding the blocking problem of single-threaded serial forwarding. During the push process, worker threads receive response information from the Livy server in real time, recording the push status (success / failure) of each executable task unit. A preset retry mechanism is triggered upon failure, and an exception is reported if the retry fails. After all executable task units have been pushed, the target thread pool summarizes the status information. By pushing multiple execution statements to the target session in parallel through the target thread pool, the limitations of traditional serial submission are broken. This allows multiple execution statements to be received by the Spark session almost simultaneously, laying the foundation for subsequent implementation of true concurrent execution based on dynamic resource allocation or scheduling strategies. This helps to improve the task throughput of a single session and the overall response efficiency of task execution.

[0045] Optionally, the steps of concurrently executing each statement based on the Spark context encapsulated within the target session include: Based on the parallelism parameter of the target thread pool, the Spark context is used to allocate a corresponding number of Executors to each execution statement, driving the synchronous and concurrent execution of each execution statement; Periodically check the computational load of Spark jobs triggered by each executed statement; When the computing load is detected to exceed the preset resource shortage threshold, the number of Executors is increased until the maximum number of Executors is reached. When the idle time of an idle Executor exceeds a preset idle threshold, the idle Executor is released until the number of Executors reaches the minimum number of Executors.

[0046] Specifically, the parallelism parameter is a core parameter for the number of concurrent executions of control statements preset in the target thread pool. It reflects the maximum number of statements executed simultaneously and is the core basis for allocating Executors in the Spark context of the target session. An Executor is a process in the Spark cluster responsible for actually executing computational tasks. Each Executor corresponds to independent CPU and memory resources, and is the hardware carrier for running execution statements. A Spark job is a collection of distributed computing tasks generated after a single execution statement is parsed by the Spark context. It can contain multiple computational stages and tasks. Computational load is an indicator of the resource consumption status of a Spark job, which can include Executor CPU utilization, memory utilization, etc. The resource stress threshold is a preset critical value for computational load, such as 80% CPU utilization and 85% memory utilization. The maximum number of Executors is the upper limit of the number of Executors configured in a single Spark session, used to avoid excessive expansion consuming cluster resources and affecting the operation of other Spark sessions. The idle threshold is a preset critical value for Executor idle time (e.g., 120 seconds). The minimum number of Executors is the lower limit of the number of Executors configured in a single Spark session. This is to ensure that subsequent new tasks do not need to re-request Executors, thus reducing cold start time.

[0047] For example, when multiple execution statements are concurrently submitted to the target session through the target thread pool, SparkDriver receives the execution statements and generates an independent Spark job for each statement. At this time, the Spark context reads the parallelism parameter of the target thread pool and requests the corresponding number of Executors from the cluster resource manager (such as YARN or Kubernetes) to support parallel computing. Meanwhile, during the execution of the execution statements, the ExecutorAllocationManager component in the system background periodically monitors the computational load of all running Spark jobs, including Executor CPU utilization, memory utilization, etc. If the computational load exceeds the preset resource pressure threshold, and the current number of Executors has not reached the maximum limit (i.e., the maximum number of Executors), a scaling request is submitted to the cluster resource manager. Executors are added in preset increments to alleviate resource pressure until the configured maximum number of Executors (spark.dynamicAllocation.maxExecutors) is reached. If the idle time of some Executors exceeds the idle threshold (e.g., spark.dynamicAllocation.executorIdleTimeout=120s), some idle Executors are released to prevent resource idleness until the remaining number of Executors is not less than the minimum number of Executors (spark.dynamicAllocation.minExecutors). Through this dynamic scaling mechanism of Executors, the Spark context can adaptively match actual computational needs, ensuring true synchronous concurrent execution of multiple statements while avoiding resource waste, significantly improving cluster resource utilization efficiency and task response performance.

[0048] Step 104: After detecting that all execution statements in the Spark asynchronous task have been completed, the target session is recycled to the session reuse pool.

[0049] In some embodiments, after submitting all execution statements, the status interfaces of each statement provided by the Livy server, such as GET / sessions / {sessionId} / statements / {statementId}, are continuously polled to monitor the execution status of each statement in real time. When it is detected that all execution statements contained in the Spark asynchronous task have entered the "available" or "error" termination state, i.e., no statement is in the "running" state, and all execution statements have been completed (including successful / failed termination), the Spark asynchronous task is considered to have been completed. Subsequently, the status of the target session can be updated from "occupied" to "idle", the binding relationship between the target session and the current Spark asynchronous task is cleared, and the session identifier and metadata (such as resource configuration, health indicators, last active time) of the target session are re-registered to the available session queue of the session reuse pool for subsequent task reuse. The session recycling mechanism realizes the cyclic reuse of Spark sessions, thereby avoiding the resource overhead and startup delay caused by frequent creation and destruction of Spark sessions, improving system resource utilization and task scheduling efficiency, while ensuring the cyclic use and lifecycle management of Spark sessions.

[0050] Based on the Spark asynchronous task submission method provided in this application, the method receives Spark asynchronous tasks submitted by clients. Each Spark asynchronous task includes multiple execution statements to be executed and task request parameters. Instead of creating a new Spark session for each task, an idle target session is selected from a pre-created and maintained session reuse pool based on the task request parameters. The target session has been initialized by the Livy server and its internal Spark context is ready for immediate use. Utilizing the target thread pool associated with the target session, each execution statement in the Spark asynchronous task is forwarded to the target session, and each execution statement is executed concurrently based on the Spark context encapsulated within the target session. After all execution statements in the Spark asynchronous task have been executed, the target session is recycled back to the session reuse pool for reuse by subsequent tasks, avoiding the overhead of repeated creation. Through session reuse, concurrent submission and parallel execution of each execution statement in the Spark asynchronous task, and intelligent session recycling, this method solves the technical problem of low task execution efficiency caused by the inability to reuse sessions in Spark task submission in existing technologies. Furthermore, it explores the concurrency potential of a single session, improving the execution efficiency and resource utilization of Spark asynchronous tasks.

[0051] In some embodiments, prior to the step of selecting an idle target session for a Spark asynchronous task from a session reuse pool, the method further includes: Read the session pool configuration parameters of the session reuse pool. The session pool configuration parameters include at least the initial number of sessions. Based on the session pool configuration parameters, a session pool creation request is sent to the Livy server to pre-create several initial Spark sessions, and each successfully created Spark session is added to the available session queue of the session reuse pool, thus completing the initialization of the session reuse pool.

[0052] Specifically, the session pool configuration parameters are a set of core parameters used to define the operating rules of the session reuse pool, serving as the basis for session pool initialization, scaling up and down, and resource management. The session pool configuration parameters include at least the initial number of sessions, the minimum total number of sessions, the maximum total number of sessions, and the session idle time threshold. The initial number of sessions is the number of Spark sessions pre-created during session reuse pool initialization, ensuring available sessions immediately upon pool startup, without needing to create sessions temporarily upon task submission. The minimum total number of sessions is the minimum number of sessions that the session reuse pool must maintain during operation, preventing excessive session reclamation during off-peak periods that could lead to a lack of available sessions for subsequent tasks. The maximum total number of sessions is the upper limit on the number of sessions the session reuse pool can support, preventing unlimited growth in the number of sessions and ensuring overall cluster stability. The session idle time threshold is the longest allowed time for a session to remain idle; idle sessions exceeding this threshold will be reclaimed and destroyed to prevent long-term resource idleness.

[0053] As an example, before startup or first use, the system first reads the session pool configuration parameters from the configuration file or configuration center. These parameters include at least the initial number of sessions (i.e., the number of Spark sessions to be pre-created during pool initialization). This initial number can be preset based on the Spark cluster's resource size and expected task concurrency. Then, based on this initial number, a batch session creation request is sent to the Livy server. This request carries key information such as the basic resource configuration required for the Spark session and Spark context initialization parameters, ensuring that the created Spark sessions meet the basic requirements for task execution. Upon receiving the session creation request, the Livy server creates Spark sessions one by one according to the initial number, completing processes such as JVM startup, SparkContext initialization, and cluster connection establishment. Once the session status becomes "active," the system verifies the availability of each session (e.g., normal heartbeat response and compliant resource configuration) and adds all successfully created Spark sessions to the available session queue of the session reuse pool. Simultaneously, each session is marked as "idle," and basic information such as session ID and creation time is recorded, completing the initialization of the session reuse pool. By pre-creating ready Spark sessions, available resources can be reserved in advance, avoiding the 10-30 second cold start time of creating temporary sessions when submitting tasks. This ensures that a suitable session can be quickly obtained after task submission, effectively improving task response speed and system throughput in high-concurrency scenarios. Consequently, the system's response speed and service capabilities are significantly improved, laying the foundation for efficient reuse, concurrent execution, and elastic scheduling in the future.

[0054] In some embodiments, the session pool configuration parameters of the session reuse pool are obtained, and the session idle time threshold and the minimum total number of sessions of the session reuse pool are determined based on the session pool configuration parameters. Periodically retrieve the idle time of idle Spark sessions in the available session queue of the session reuse pool, and destroy Spark sessions whose idle time exceeds the session idle time threshold through the Livy server, until the number of Spark sessions in the session reuse pool reaches the minimum total number of sessions.

[0055] Specifically, the system first reads the session pool configuration parameters from the configuration center or local configuration file, and then parses the session idle time threshold (e.g., 1000 seconds) and the minimum total number of sessions (e.g., 10) from these parameters. Next, a session idle recycling thread is started, which iterates through the available session queue in the session reuse pool at fixed intervals (e.g., every 100 seconds). The available session queue stores all Spark sessions currently in an idle state. For each idle Spark session in the available session queue, the system calculates its idle time, which is the difference between the current system time and the timestamp of the last time the Spark session was used. If the idle time of a Spark session exceeds the session idle time threshold, the Spark session is marked as an object to be destroyed. Before execution of destruction, the system determines whether the total number of Spark sessions in the current session reuse pool is greater than the minimum total number of sessions. Destruction is only allowed if this condition is met, ensuring that the session reuse pool always maintains basic capacity. Subsequently, the system can call the REST interface (DELETE / sessions / {sessionId}) provided by the Livy server to send a session termination request to the Livy server. Livy then notifies the resource manager (such as YARN or Kubernetes) to release the corresponding SparkApplication and its occupied computing resources. After the idle Spark session is destroyed, it is removed from the available session queue and internal metadata mapping. This achieves dynamic reclamation of long-unused session resources while ensuring minimum service capacity, effectively avoiding cluster resource waste and improving overall resource utilization efficiency and system elasticity.

[0056] In some embodiments, anomaly detection parameters of each Spark session in the session reuse pool are collected periodically. The anomaly detection parameters include at least Spark context state, heartbeat response time, number of historical execution failures, and resource load parameters. For each Spark session, identify the anomaly detection parameters for any Spark session. If any Spark session's anomaly detection parameters exceed the corresponding parameter threshold range, the corresponding Spark session is identified as an abnormal session and destroyed via the Livy server.

[0057] Specifically, the anomaly detection parameters are a multi-dimensional set of indicators used to determine the health status of a Spark session. These parameters are the core basis for identifying abnormal sessions and include at least Spark context state, heartbeat response time, historical execution failure count, and resource load parameters, comprehensively covering the core dimensions of session operation. Spark context state refers to the running state of the Spark context within the Spark session, such as normal, initialization failure, connection interruption, or resource exhaustion. The Spark context state directly determines whether the session has the basic capability to execute tasks. Heartbeat response time is the time taken to receive a normal response after sending a heartbeat detection request to the Spark session, reflecting the network connectivity and session activity between the Spark session and the Livy server and Spark cluster. Historical execution failure count is the cumulative number of non-business logic failures (such as session crashes or resource overflows) that occurred during task execution in this Spark session within a preset statistical period (e.g., 24 hours), reflecting the session's execution stability. Resource load parameters are indicators of the session's current resource usage status, such as CPU utilization, memory utilization, and Executor busy percentage, reflecting whether the session's resource carrying capacity is within a reasonable range. The parameter threshold range is a preset normal value range for each anomaly detection parameter, such as heartbeat response time ≤ 2000ms and CPU utilization ≤ 90%. If the value exceeds the threshold range, the anomaly detection parameter is determined to be abnormal.

[0058] As an example, the system starts a separate session exception monitoring thread, which iterates through all Spark sessions (including idle sessions and sessions with assigned tasks) in the session reuse pool at a preset period (e.g., once every minute). It comprehensively collects exception detection parameters for each Spark session, including at least the Spark context state, heartbeat response time, historical execution failure count, and resource load parameters. The Spark context state can be obtained by calling the Livy server's GET / sessions / {sessionId} interface, and can be normal, initialization failed, connection interrupted, or resource exhausted. The heartbeat response time can be obtained by sending a detection request to the Spark session and recording the response time. The historical execution failure count is calculated from the system logs based on the cumulative number of non-business logic failures within a preset period (e.g., 24 hours). Resource load parameters can be collected from the cluster resource monitoring module, including metrics such as CPU utilization and memory utilization. Subsequently, for each Spark session, each anomaly detection parameter is verified to ensure it meets preset threshold ranges. These parameters include a Spark context state of "normal," a heartbeat response time ≤2000ms, ≤3 historical execution failures, and CPU utilization ≤90%. If any anomaly detection parameter exceeds the corresponding preset threshold, the Spark session is immediately marked as an abnormal session. If the abnormal session is currently handling an active task, a task migration process can be triggered to transfer incomplete tasks to healthy sessions in the pool. If the abnormal session is idle, the Livy server's session destruction REST interface can be called directly, passing in the abnormal session ID to initiate a destruction request. This releases the JVM, Executor, and cluster resources it occupies. After destruction, the record is removed from the session list in the session reuse pool, ensuring that only healthy sessions reliably handling tasks remain in the session reuse pool, thus guaranteeing efficient and stable submission of Spark asynchronous tasks. By periodically collecting multi-dimensional anomaly detection parameters, session runtime anomalies can be identified accurately and promptly. Destroying abnormal sessions via the Livy server avoids task assignment to abnormal sessions, significantly reducing task failure rates. At the same time, it releases invalid resources occupied by abnormal sessions, avoids resource waste, improves the effective utilization rate of cluster resources, and ensures that healthy sessions are always retained in the session reuse pool, providing reliable support for subsequent task allocation.

[0059] In some embodiments, the session pool configuration parameters of the session multiplexing pool are obtained, and the maximum and minimum total number of sessions of the session multiplexing pool are determined based on the session pool configuration parameters; Periodically collect session-level load data corresponding to the session reuse pool. The load data includes the number of active Spark sessions, the busy / idle time period, and the task queue length associated with each Spark session in the used session queue in the session reuse pool. The busy / idle time period is either peak or off-peak, and the peak and off-peak periods are determined by the time period of historical load data. Calculate the average task queue length corresponding to the session reuse pool based on the task queue length associated with each Spark session. If the busy / idle time period is a peak period, when the average length of the task queue is greater than or equal to the first preset length, and the number of Spark sessions in the current session reuse pool is less than the maximum total number of sessions, Spark sessions are added through the Livy server until the number of Spark sessions reaches the maximum total number of sessions. If the busy / idle time period is a low-peak period, when the number of active Spark sessions remains below the minimum total number of sessions for a preset duration, idle Spark sessions will be destroyed through the Livy server until the number of Spark sessions reaches the minimum total number of sessions.

[0060] Specifically, session-level load data is key data reflecting the overall operational pressure and status of the session reuse pool, including the number of active Spark sessions, the busy / idle time period, and the task queue length. Session-level load data is the core input for scaling decisions. The number of active Spark sessions is the number of Spark sessions currently handling tasks (in the "assigned" state), directly reflecting the current task processing pressure. The busy / idle time period is a session pool operational cycle type based on historical load data, divided into peak periods (times with dense task submissions and high load) and off-peak periods (times with fewer tasks and low load). The used session queue is the set of Spark sessions in the session reuse pool that are in the "assigned" state and handling tasks. The task queue length associated with the used session queue reflects the task backlog within a single session. The task queue length is the total number of tasks waiting to be executed or currently executing in a single used Spark session, reflecting the task processing pressure of that Spark session. The first preset length is a pre-set threshold for the length of the task queue (e.g., 10). When the average length of the task queue reaches or exceeds the first preset length, it indicates that the number of Spark sessions in the current session reuse pool cannot meet the task processing requirements, thus triggering expansion.

[0061] As an example, session pool configuration parameters are obtained from the session reuse pool's configuration center or local configuration file. The maximum total number of sessions (the upper limit allowed by the session pool) and the minimum total number of sessions (the minimum number that must always be maintained) are extracted and determined, clarifying the boundary constraints for scaling up and down. Simultaneously, historical load data is loaded to divide the time periods of task-intensive peak periods and task-sparse off-peak periods as a basis for determining busy / idle status. Then, a load monitoring thread is started to collect session-level load data at a preset interval (e.g., every 2 minutes). Session-level load data includes the number of active Spark sessions, the busy / idle time period, and the task queue length. The number of currently active Spark sessions (the total number of sessions with assigned tasks) is counted. Based on the current time, the busy / idle time period is matched, and the used session queue is traversed to obtain the task queue length associated with each Spark session (the total number of tasks waiting and executing). Based on the collected task queue lengths associated with each Spark session, the average task queue length corresponding to the session reuse pool is calculated, comprehensively reflecting the degree of task backlog within the session reuse pool. If the current time is determined to be a peak period, and the average task queue length is greater than or equal to the first preset length (e.g., 10), while the current number of sessions has not reached the maximum total number of sessions, Spark sessions can be created in batches through the Livy server, and Spark context initialization can be completed. This scaling continues until the number of sessions reaches the upper limit or the average task queue length drops. If the current time is determined to be an off-peak period, and the number of active Spark sessions remains below the minimum total number of sessions for a preset duration (e.g., 10 minutes), idle Spark sessions in the available session queue are traversed, and redundant idle Spark sessions are destroyed in batches through the Livy server until the number of Spark sessions drops to the minimum total number of sessions. After scaling up or down, the total number of sessions in the session reuse pool and the status of the available / used session queues are updated in real time to ensure that the number of sessions accurately matches the task load, balancing task processing efficiency and cluster resource utilization. By combining busy / idle time periods and real-time task queue load, the number of Spark sessions is dynamically adjusted. During peak periods, automatic scaling up reduces queuing latency, and during off-peak periods, scaling down saves resources, thereby improving cluster resource utilization efficiency while ensuring task response performance, achieving intelligent elastic scaling of the session reuse pool.

[0062] refer to Figure 3 , Figure 3This diagram illustrates the management logic of the session reuse pool, showcasing its complete execution logic. The session reuse pool's configuration center manages session pool parameters (including at least the initial number of sessions, minimum total number of sessions, maximum total number of sessions, and session idle time threshold), and creates Spark sessions through the Livy server. The session reuse pool contains an available session queue (idle sessions) and a used session queue (sessions carrying tasks), and is equipped with a session idle recycling thread (cleaning up timed-out idle sessions) and a session exception monitoring thread (identifying and destroying abnormal sessions). When a client initiates a task request, an idle Spark session is retrieved from the available session queue in the session reuse pool to execute the task, and the Spark session carrying the task is added to the used session queue. After the task is completed, the Spark session is recycled back to the available session queue.

[0063] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the Spark asynchronous task submission method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0064] This application also provides a Spark asynchronous task submission device; please refer to [reference needed]. Figure 4 Spark asynchronous task submission mechanisms include: The task acquisition module 401 is used to receive Spark asynchronous tasks submitted by the client. The Spark asynchronous task includes multiple execution statements to be executed and task request parameters. The session selection module 402 is used to select an idle target session for the Spark asynchronous task from the session reuse pool based on the task request parameters. The session reuse pool includes multiple Spark sessions created by the Livy server. Each Spark session encapsulates an initialized Spark context. The task execution module 403 is used to forward each execution statement in the Spark asynchronous task to the target session using the target thread pool associated with the target session, and to concurrently execute each execution statement based on the Spark context encapsulated inside the target session. The session recycling module 404 is used to recycle the target session to the session reuse pool after detecting that all executed statements in the Spark asynchronous task have been completed.

[0065] The Spark asynchronous task submission device provided in this application, employing the Spark asynchronous task submission method in the above embodiments, can solve the technical problem of low task execution efficiency caused by the inability to reuse sessions in existing Spark task submissions. Compared with the prior art, the beneficial effects of the Spark asynchronous task submission device provided in this application are the same as those of the Spark asynchronous task submission method provided in the above embodiments, and other technical features in the Spark asynchronous task submission device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0066] This application provides a Spark asynchronous task submission device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to execute the Spark asynchronous task submission method in Embodiment 1 above.

[0067] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the Spark asynchronous task submission device in the embodiments of this application. The Spark asynchronous task submission device in the embodiments of this application may include, but is not limited to, mobile terminals such as laptops and in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The Spark asynchronous task submission device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0068] like Figure 5As shown, the Spark asynchronous task submission device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the Spark asynchronous task submission device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the Spark asynchronous task submission device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a Spark asynchronous task submission device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented alternatively.

[0069] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0070] The Spark asynchronous task submission device provided in this application, employing the Spark asynchronous task submission method described in the above embodiments, can solve the technical problem of low task execution efficiency caused by the inability to reuse sessions in existing Spark task submissions. Compared with the prior art, the beneficial effects of the Spark asynchronous task submission device provided in this application are the same as those of the Spark asynchronous task submission method provided in the above embodiments, and other technical features of this Spark asynchronous task submission device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0071] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0072] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0073] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the Spark asynchronous task submission method in the above embodiments.

[0074] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0075] The aforementioned computer-readable storage medium may be included in the Spark asynchronous task submission device; or it may exist independently and not be assembled into the Spark asynchronous task submission device.

[0076] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the Spark asynchronous task submission device, the Spark asynchronous task submission device: receives a Spark asynchronous task submitted by a client, the Spark asynchronous task including multiple execution statements to be executed and task request parameters; selects an idle target session for the Spark asynchronous task from the session reuse pool based on the task request parameters, the session reuse pool including multiple Spark sessions created through the Livy server, each Spark session encapsulating an initialized Spark context; forwards each execution statement in the Spark asynchronous task to the target session using the target thread pool associated with the target session, and concurrently executes each execution statement based on the Spark context encapsulated within the target session; and after detecting that all execution statements in the Spark asynchronous task have been executed, reclaims the target session back to the session reuse pool.

[0077] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0078] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0079] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0080] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described Spark asynchronous task submission method. This solves the technical problem of low task execution efficiency caused by the inability to reuse sessions in existing Spark task submission methods. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the Spark asynchronous task submission method provided in the above embodiments, and will not be elaborated upon here.

[0081] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the Spark asynchronous task submission method described above.

[0082] The computer program product provided in this application can solve the technical problem of low task execution efficiency caused by the inability to reuse sessions in existing Spark task submissions. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the Spark asynchronous task submission method provided in the above embodiments, and will not be repeated here.

[0083] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A Spark asynchronous task submission method, characterized in that, The Spark asynchronous task submission method includes: Receive Spark asynchronous tasks submitted by the client, wherein the Spark asynchronous task includes multiple execution statements to be executed and task request parameters; Based on the task request parameters, an idle target session is selected from the session reuse pool for the Spark asynchronous task. The session reuse pool includes multiple Spark sessions created by the Livy server, and each Spark session encapsulates an initialized Spark context. Using the target thread pool associated with the target session, each execution statement in the Spark asynchronous task is forwarded to the target session, and each execution statement is executed concurrently based on the Spark context encapsulated within the target session; Once the execution of all the statements in the Spark asynchronous task has been completed, the target session will be recycled to the session reuse pool.

2. The Spark asynchronous task submission method as described in claim 1, characterized in that, Before the step of selecting an idle target session from the session reuse pool for the Spark asynchronous task, the method further includes: Read the session pool configuration parameters of the session reuse pool, wherein the session pool configuration parameters include at least the initial number of sessions; Based on the session pool configuration parameters, a session pool creation request is initiated to the Livy server to pre-create several Spark sessions of the initial session, and each successfully created Spark session is added to the available session queue of the session reuse pool to complete the initialization of the session reuse pool.

3. The Spark asynchronous task submission method as described in claim 1, characterized in that, Also includes: Obtain the session pool configuration parameters of the session reuse pool, and determine the session idle time threshold and minimum total number of sessions of the session reuse pool based on the session pool configuration parameters; The idle time of the Spark sessions in the available session queue of the session reuse pool is periodically obtained, and the Spark sessions whose idle time exceeds the session idle time threshold are destroyed by the Livy server until the number of Spark sessions in the session reuse pool reaches the minimum total number of sessions.

4. The Spark asynchronous task submission method as described in claim 1, characterized in that, Also includes: Periodically collect anomaly detection parameters for each Spark session in the session reuse pool. These anomaly detection parameters include at least Spark context state, heartbeat response time, historical execution failure count, and resource load parameters. For each Spark session, the anomaly detection parameters are identified for any Spark session. If it is determined that the anomaly detection parameter of any Spark session exceeds the corresponding parameter threshold range, the corresponding Spark session is identified as the abnormal session, and the abnormal session is destroyed through the Livy server.

5. The Spark asynchronous task submission method as described in claim 1, characterized in that, Also includes: Obtain the session pool configuration parameters of the session multiplexing pool, and determine the maximum and minimum total number of sessions for the session multiplexing pool based on the session pool configuration parameters; Periodically collect session-level load data corresponding to the session reuse pool. The load data includes the number of active Spark sessions, the busy / idle time period, and the task queue length associated with each Spark session in the used session queue of the session reuse pool. The busy / idle time period is either a peak period or an off-peak period, and the peak and off-peak periods are determined by the time period of historical load data. Based on the task queue length associated with each Spark session, calculate the average task queue length corresponding to the session reuse pool; If the busy / idle time period is the peak period, when the average length of the task queue is greater than or equal to the first preset length, and the number of Spark sessions in the current session reuse pool is less than the maximum total number of sessions, Spark sessions are added through the Livy server until the number of Spark sessions reaches the maximum total number of sessions. If the busy / idle time period is the off-peak period, when the number of active Spark sessions remains below the minimum total number of sessions for a preset duration, the idle Spark sessions are destroyed by the Livy server until the number of Spark sessions reaches the minimum total number of sessions.

6. The Spark asynchronous task submission method as described in claim 1, characterized in that, The step of forwarding each execution statement in the Spark asynchronous task to the target session using the target thread pool associated with the target session includes: Each execution statement in the Spark asynchronous task is encapsulated into a corresponding executable task unit; Each of the executable task units is submitted to the target thread pool; The executable task units are pushed to the target session in parallel by calling the REST interface of the Livy server through worker threads in the target thread pool.

7. The Spark asynchronous task submission method as described in claim 1, characterized in that, The step of concurrently executing each of the execution statements based on the Spark context encapsulated within the target session includes: Based on the parallelism parameter of the target thread pool, the Spark context is used to allocate a corresponding number of Executors to each of the execution statements, driving each of the execution statements to execute synchronously and concurrently. Periodically check the computational load of the Spark jobs triggered by each of the executed statements; When the computing load is detected to exceed a preset resource shortage threshold, the number of Executors is increased until the maximum number of Executors is reached. When the idle time of an idle Executor exceeds a preset idle threshold, the idle Executor is released until the number of Executors reaches the minimum number of Executors.

8. The Spark asynchronous task submission method as described in claim 1, characterized in that, The step of selecting an idle target session for the Spark asynchronous task from the session reuse pool based on the task request parameters includes: Parse the task request parameters to extract the task resource requirements of the Spark asynchronous task; Using the task resource requirements as a filtering condition, candidate idle sessions whose resource configurations meet the filtering conditions are selected from the available session queue of the session reuse pool. The available session queue contains multiple idle Spark sessions. Obtain the historical execution failure rate, idle time, heartbeat response time and resource load parameters of each candidate idle session, and calculate the health score of each candidate idle session based on the historical execution failure rate, idle time, heartbeat response time and resource load parameters; The candidate idle session corresponding to the one with the highest health score value is selected as the target session; If the available session queue is empty and the current number of Spark sessions in the session reuse pool has not reached the preset maximum total number of sessions, the session expansion process is triggered. A new Spark session is created through the Livy server, the Spark context of the new Spark session is initialized, and the new Spark session is used as the target session.

9. A Spark asynchronous task submission device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the Spark asynchronous task submission method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the Spark asynchronous task submission method as described in any one of claims 1 to 8.

Images