A Spark job submission method and system for light clients

Abstract

Disclosed is a Spark job submission method and system for a light client, comprising a dynamic proxy receiving a job processing request sent by a front-end user and selecting a first back-end service to process the request; the first back-end service writing the request to a message queue, notifying the front-end user that the request was successfully submitted and entering a scheduling queue for processing; a second back-end service that is idle in the cluster retrieves the request from the message queue, uses a Spark-driven process to call a set executor to schedule and execute the request, and regularly writes the processing progress, results, and sampled data to a NoSQL database; after receiving a successful request submission message, the front-end user sends a request to the dynamic proxy to obtain real-time progress and processing results. The dynamic proxy selects a third back-end service to process the request based on the request information, retrieves the real-time progress and processing results from the database, and returns them to the front-end user. This application can reduce the resource usage burden of edge node servers, reduce the frequency of failures, and efficiently achieve the interactive effect of light clients.

Claims

1. A Spark job submission method for a light client, characterized in that: include: S1: In response to the dynamic proxy HAProxy receiving a job processing request sent by a front-end user, selecting a first back-end service to process the job processing request according to a load balancing strategy; S2: The first backend service writes the job processing request into a message queue and notifies the frontend user that the job processing request has been successfully submitted and entered into a scheduling queue for processing; S3: Another idle second backend service in the YARN cluster obtains the job processing request from the message queue. The second backend service uses the Spark driver process to call the executor in the cluster to schedule and execute the job processing request, writes the processing progress to the NoSQL database at regular intervals, and writes the processing results and sampled data to the NoSQL database. S4: After receiving the information that the job processing request has been successfully submitted from S2, the front-end user sends a request to the dynamic proxy HAProxy to obtain the real-time progress and processing results of the job. The dynamic proxy HAProxy selects a third back-end service for processing based on the request information and the load strategy, obtains the real-time progress and processing results of the job from the NoSQL database, and returns them to the front-end user through the dynamic proxy HAProxy. Direct interactive access between the front-end user and the YARN cluster is shielded by a dynamic proxy service, wherein the dynamic proxy service specifically includes: Start the Launcher service on the YARN cluster edge node server, start and maintain one or more lightweight Web application services, and configure the Launcher to monitor and ensure that the one or more lightweight Web application services can always run in the YARN cluster; The successfully started lightweight web application service will register its IP address and listening window to the HAProxy backend configuration through the HAProxy dynamic configuration interface; The HAProxy dynamic configuration interface dynamically reloads HAProxy according to the latest configuration, provides a unified proxy service to the outside world, and forwards Spark related requests to the lightweight Web application services distributed in the YARN cluster for processing.

2. The Spark job submission method for a light client according to claim 1, characterized in that: Also includes: In response to the dynamic proxy HAProxy receiving a job cancellation request sent by a front-end user, the dynamic proxy HAProxy selects a second back-end service according to the load balancing strategy to obtain the job execution progress and the back-end service information where the job is currently running from the NoSQL database, performs a cancellation operation according to the job progress status, and feeds back the job cancellation result to the front-end user through the dynamic proxy HAProxy.

3. The Spark job submission method for a light client according to claim 2, characterized in that: The cancel operation according to the job progress status includes: In response to the absence of the job execution progress, deleting the job information in the message queue; In response to the completion of the job execution progress, marking the job as cancelled; In response to the incomplete execution progress of the job, a request to cancel the job is forwarded to the first backend service where the job is running for processing. The first backend service submits a job cancellation instruction to the cluster through the Spark driver process, changes the job execution progress in the NoSQL database to cancel, marks the job as canceled, and feeds back the cancellation result to the second backend service.

4. The Spark job submission method for a light client according to claim 3, characterized in that: Starting the Launcher service on the YARN cluster edge node server specifically includes: The Launcher service periodically obtains the lightweight Web application service backend service interface currently configured by the HAProxy through the HAProxy dynamic configuration interface; The Launcher service communicates with the lightweight Web application service to check the validity of all backend services, and in response to the number N of currently valid lightweight Web application services being less than the number M of services that actually need to be started, starts the lightweight Web application service of MN through the Cluster mode of Spark onYarn; The Launcher service deletes the invalid lightweight Web application service from the HAProxy configuration through the HAProxy dynamic configuration interface.

5. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 4 is implemented.

6. A Spark job submission system for a light client, capable of executing the Spark job submission method for a light client according to any one of claims 1 to 4, characterized in that: The system comprises: A lightweight web application service with an internally integrated Spark execution engine driver process. It runs within a YARN cluster using Spark on Yarn's Cluster mode and schedules executors to complete Spark job execution requests. Dynamic proxy HAProxy, deployed on the edge server node of the YARN cluster, proxies Spark job operation requests sent by front-end users and sends them to the lightweight Web application services distributed on the back-end YARN cluster for corresponding processing according to the load balancing strategy; Message queue: After receiving the Spark job submission request, the lightweight Web application service directly writes it into the lightweight message queue service and queues it for processing; The NoSQL database is used to save and update the Spark job execution progress, and save the sample data or result data after the job is completed, and display it to the front-end user.

7. The Spark job submission system for light clients according to claim 6, characterized in that: It also includes an HAProxy dynamic configuration interface. After the lightweight Web application service is successfully started, the IP address and port it monitors are dynamically registered in the HAProxy proxy configuration through the HAProxy dynamic configuration interface. The HAProxy dynamic configuration interface automatically reloads the HAProxy service according to the configuration changes, allowing the front-end user to access all back-end services without perception.

8. The Spark job submission system for light clients according to claim 7, characterized in that: It also includes a Launcher service for monitoring and ensuring that one or more lightweight Web application services can always run in the YARN cluster; and dynamically deleting invalid lightweight Web application service interfaces from the HAProxy dynamic configuration interface.

9. The Spark job submission system for light clients according to claim 8, characterized in that: In response to the number N of currently valid lightweight Web application services being less than the number M of services that actually need to be started, the lightweight Web application service MN is started through the Cluster mode of Spark on Yarn.

Images