PySpark big data task management method and system and storage medium

Abstract

The invention provides a PySpark big data task management method and system and a storage medium, and the method comprises the steps: firstly receiving a PySpark task code submitted by a user; then analyzing the operation environment dependency information to generate a structured environment list; checking whether a corresponding cache mirror image exists in a mirror image warehouse or not according to the environment list; when the cache mirror image does not exist, constructing a dynamic mirror image; when the PySpark task is scheduled and executed, a corresponding task exclusive container mirror image is called from the mirror image warehouse, and a container is created based on the task exclusive container mirror image to serve as an environment when the PySpark task runs; in the process of executing the PySpark task, capturing a running log in real time, and transmitting the running log to a big data development platform in a streaming manner for centralized storage, indexing and display; through declarative environment construction, intelligent resource scheduling and full-link log integration, the environment management efficiency can be effectively improved, the resource utilization rate is optimized, and the operation and maintenance debugging experience is improved.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a PySpark big data task management method, system, and storage medium. Background Technology

[0002] Alibaba Cloud DataWorks is a mainstream big data development platform, while PySpark is a programming interface provided for Python developers. It enables interaction with Spark's core components through Py4J and supports functions such as structured data processing, stream computing, and machine learning. PySpark is widely used in Alibaba Cloud DataWorks for developing big data projects because it combines the ease of use of Python with the high performance of Spark.

[0003] However, PySpark has the following core pain points when developing big data projects: 1. Complex environment configuration: Python virtual environment and third-party library dependencies need to be manually maintained, resulting in poor reusability across tasks; 2. Inefficient resource scheduling: Static resource allocation leads to insufficient cluster utilization or resource contention; 3. Difficult development and debugging: Logs are scattered across the DataWorks console and Spark cluster, resulting in low efficiency in fault location; Therefore, overall, the existing technology has relatively low efficiency in managing big data tasks with PySpark. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art. Therefore, the purpose of this invention is to propose a PySpark big data task management method, system, and storage medium.

[0005] This invention proposes a PySpark big data task management method, comprising: receiving PySpark task code submitted by a user, wherein the code contains runtime environment dependency information defined in declarative syntax, the dependency information including at least a base image identifier, a Python version, and a list of third-party libraries; parsing the runtime environment dependency information to generate a structured environment list; checking whether a corresponding cached image exists in the image repository based on the environment list; when the cached image does not exist, constructing a dynamic image, including: pulling a base image based on the base image identifier, using the base image as the parent image, executing an installation command according to the list of third-party libraries to generate a task-specific container image, and pushing the task-specific container image to the image repository; when scheduling and executing a PySpark task, retrieving the corresponding task-specific container image from the image repository, and creating a container based on the task-specific container image as the runtime environment for the PySpark task; during the execution of the PySpark task, capturing runtime logs in real time, and streaming the runtime logs to a big data development platform for centralized storage, indexing, and display.

[0006] According to an embodiment of the PySpark big data task management method of the present invention, the method first receives PySpark task code submitted by a user. The code contains runtime environment dependency information defined in declarative syntax. The dependency information includes at least a base image identifier, a Python version, and a list of third-party libraries. Then, the method parses the runtime environment dependency information to generate a structured environment list. Next, based on the environment list, the method checks whether a corresponding cached image exists in the image repository. If no cached image exists, the method constructs a dynamic image, including: pulling a base image based on the base image identifier, using the base image as the parent image, executing an installation command according to the list of third-party libraries, generating a task-specific container image, and storing the task-specific container image in the repository. The container image is pushed to the image repository; when scheduling and executing PySpark tasks, the corresponding task-specific container image is retrieved from the image repository, and a container is created based on the task-specific container image as the runtime environment for the PySpark task; during the execution of the PySpark task, the runtime logs are captured in real time and streamed to the big data development platform for centralized storage, indexing, and display; in this way, through declarative environment construction, intelligent resource scheduling, and end-to-end log integration, the pain points of PySpark environment management, resource allocation, and operation and maintenance debugging in DataWorks are solved, which can effectively improve environment management efficiency, optimize resource utilization, and improve the operation and maintenance debugging experience.

[0007] In addition, the PySpark big data task management method according to embodiments of the present invention may also have the following additional technical features: Furthermore, when constructing the dynamic image, a layered inheritance strategy is adopted, including: the first layer is the base image layer, which provides the operating system, Java runtime environment, Spark and Python basic environment; the second layer is the project common dependency layer, which is used to install common third-party libraries shared by multiple tasks within the project group based on the base image layer; the third layer is the task-specific dependency layer, which is used to install the specific third-party libraries declared by the current task and their versions based on the base image layer or the project common dependency layer. In order to efficiently provide a customized and isolated environment for each task, a container layering model is adopted. First, a general base layer shared by all tasks is built, then the common dependency layer shared by the project group is superimposed, and finally the specific dependency layer unique to each task is injected, resulting in a lightweight, efficient and highly isolated task runtime image. That is, efficiency is improved by sharing the base layer, and the customization and isolation of the environment are achieved by superimposing independent layers. This "layered construction and layered reuse" mechanism is the key technical foundation for achieving "declarative automatic environment construction" that can ensure environment consistency and isolation while maintaining extremely high efficiency.

[0008] Furthermore, before executing the PySpark task, the method further includes: performing resource prediction, specifically including: performing AST parsing on the PySpark task code to extract code structure features; combining the task's historical execution feature data to form an input feature vector; inputting the input feature vector into a pre-trained resource prediction model to obtain the predicted resource demand level; allocating appropriate computing resources to the task and selecting the corresponding execution cluster based on the resource demand level; and using this machine learning-based intelligent resource scheduling decision-making method to transform resource allocation from manual experience-based decision-making to data-driven automated prediction by analyzing code features and historical performance, which helps to improve the overall cluster utilization and task execution efficiency.

[0009] Furthermore, the code structure features include one or more of the following: AST depth, AST complexity, types of operations included, and estimated data processing volume. The types of operations include data shuffle operations and machine learning library call operations. This facilitates in-depth analysis of the code and enables more accurate and earlier prediction of the task's computational load than simply based on the number of lines of code or manual labels.

[0010] Furthermore, the resource prediction model is a decision tree model or a random forest model, and the pre-training includes training based on historical task characteristics and actual resource consumption data; the resource demand levels include at least three levels: low, medium, and high, with different levels corresponding to different computing resource quotas; this helps to automate and accurately allocate resources. The resource prediction model is based on a large amount of data, which is far more scientific than manual judgment, and can avoid resource waste and prevent task failures caused by insufficient resources, thereby improving the overall cluster utilization and task execution efficiency.

[0011] Furthermore, the step of allocating appropriate computing resources and selecting the corresponding execution cluster for the task based on the resource requirement level includes: selecting a serverless MaxCompute cluster when the resource requirement level is low; or selecting an EMR standard CPU cluster when the resource requirement level is medium; or selecting an EMR GPU cluster when the resource requirement level is high, or when machine learning library call operations are detected in the code. This helps to automate and refine resource allocation, thereby improving overall cluster utilization and task execution efficiency.

[0012] Furthermore, the real-time capture of runtime logs includes: configuring a logging framework and SocketAppender on the Driver and Executor nodes of the Spark task; sending locally generated log events to a specified log collection agent in real time via the SocketAppender in the form of a network stream; thereby capturing runtime logs in real time, creating a prerequisite for centralized log management.

[0013] Furthermore, the process of streaming logs to the big data development platform includes: receiving log streams from multiple tasks through the log collection agent; parsing and enriching the logs, adding task identifiers and metadata; and writing the processed log data in batches into the log service integrated into the big data development platform. This streaming transmission of logs to the big data development platform helps to achieve centralized log management.

[0014] To address the aforementioned problems, this invention also proposes a PySpark big data task management system, comprising: a receiving module for receiving PySpark task code submitted by a user, the code containing runtime environment dependency information defined in declarative syntax, the dependency information including at least a base image identifier, a Python version, and a list of third-party libraries; a parsing module for parsing the declarative syntax and generating a structured environment list; a checking module for checking whether a corresponding cached image exists in the image repository based on the environment list; a building module for building a dynamic image when the cached image does not exist, including: pulling a base image based on the base image identifier, using the base image as the parent image, executing an installation command according to the list of third-party libraries, generating a task-specific container image, and pushing the task-specific container image to the image repository; a scheduling module for pulling the corresponding task-specific container image from the image repository when scheduling the execution of a PySpark task, and creating a container based on the task-specific container image as the runtime environment for the task; and a log service module for capturing runtime logs in real time during the execution of the PySpark task, and streaming the logs to a big data development platform for centralized storage, indexing, and display.

[0015] According to an embodiment of the PySpark big data task management system of the present invention, the PySpark big data task management method of the above embodiment is executed. First, the system receives PySpark task code submitted by the user. The code contains runtime environment dependency information defined in declarative syntax. The dependency information includes at least a base image identifier, a Python version, and a list of third-party libraries. Then, the system parses the runtime environment dependency information to generate a structured environment list. Next, based on the environment list, the system checks whether a corresponding cached image exists in the image repository. If no cached image exists, a dynamic image is constructed, including: pulling a base image based on the base image identifier, using the base image as the parent image, executing installation commands according to the list of third-party libraries, and generating a task-specific image. It uses container images and pushes task-specific container images to the image repository. When scheduling and executing PySpark tasks, it retrieves the corresponding task-specific container image from the image repository and creates a container based on the task-specific container image as the runtime environment for the PySpark task. During the execution of PySpark tasks, it captures runtime logs in real time and streams them to the big data development platform for centralized storage, indexing, and display. In this way, through declarative environment construction, intelligent resource scheduling, and end-to-end log integration, it solves the pain points of environment management, resource allocation, and operation and maintenance debugging of PySpark in DataWorks, which can effectively improve environment management efficiency, optimize resource utilization, and improve the operation and maintenance debugging experience.

[0016] To address the aforementioned problems, the present invention also proposes a computer-readable storage medium storing a PySpark big data task management program, wherein the PySpark big data task management program, when executed by a processor, implements the PySpark big data task management method as described in the first aspect embodiment of the present invention.

[0017] According to embodiments of the present invention, when a PySpark big data task management program stored on a computer-readable storage medium is executed by a processor, it performs the PySpark big data task management method described above. First, it receives PySpark task code submitted by a user. The code contains runtime environment dependency information defined in declarative syntax. The dependency information includes at least a base image identifier, a Python version, and a list of third-party libraries. Then, it parses the runtime environment dependency information to generate a structured environment list. Next, it checks whether a corresponding cached image exists in the image repository based on the environment list. When no cached image exists, it constructs a dynamic image, including: pulling a base image based on the base image identifier, using the base image as the parent image, and executing the PySpark big data task management method according to the list of third-party libraries. The system executes installation commands to generate a task-specific container image and pushes it to the image repository. When scheduling and executing PySpark tasks, it retrieves the corresponding task-specific container image from the image repository and creates a container based on the image image as the runtime environment for the PySpark task. During the execution of PySpark tasks, it captures runtime logs in real time and streams them to the big data development platform for centralized storage, indexing, and display. Through declarative environment construction, intelligent resource scheduling, and end-to-end log integration, it solves the pain points of PySpark environment management, resource allocation, and operation and maintenance debugging in DataWorks, effectively improving environment management efficiency, optimizing resource utilization, and enhancing the operation and maintenance debugging experience.

[0018] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a PySpark big data task management method according to an embodiment of the present invention; Figure 2 This is an overall architecture diagram of a PySpark big data task management method according to a specific embodiment of the present invention; Figure 3 This is a schematic diagram of a layered container image according to a specific embodiment of the present invention; Figure 4 This is a schematic diagram of intelligent resource scheduling decision-making according to a specific embodiment of the present invention; Figure 5 This is a schematic diagram of a log collection framework according to a specific embodiment of the present invention; Figure 6This is a structural block diagram of a PySpark big data task management system according to an embodiment of the present invention.

[0020] Figure label: 100 - PySpark Big Data Task Management System; 110 - Receiving Module; 120 - Parsing Module; 130 - Checking Module; 140 - Building Module; 150 - Scheduling Module; 160 - Log Service Module. Detailed Implementation

[0021] The embodiments of the present invention are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. The embodiments of the present invention are described in detail below.

[0022] In the native DataWorks PySpark environment, existing technologies require users to resolve the dependencies of each task's required third-party Python libraries (such as pandas, numpy, sklearn, etc.). This manifests itself in the following ways: (1) Manual operation: Developers must log in to the associated EMR (Elastic MapReduce) cluster via SSH (Secure Shell) or use an initialization script to install each compute node one by one using the pip install command; (2) Repetitive work: If multiple tasks require the same library (e.g., all use pandas), this installation process needs to be repeated on the cluster associated with each task. Even if the clusters are the same, all dependencies need to be reinstalled after task migration or cluster reset; (3) Environment conflicts: Different tasks may depend on different versions of the same library (e.g., task A requires pandas 1.5 and task B requires pandas 2.0). These cannot be isolated in the global environment, and version conflicts will cause task failures; (4) Poor reproducibility: The lack of a declarative, versioned dependency management mechanism makes it difficult to quickly and accurately rebuild the task runtime environment, which brings great difficulties to troubleshooting and task migration. Therefore, in summary, the preparation of the DataWorks PySpark environment in the existing technology relies entirely on manual operation, which is inefficient and cannot guarantee the consistency and isolation of the environment.

[0023] To address the above problems, this invention provides a PySpark big data task management method, system, and storage medium, as described below. Figures 1-6 This invention describes a PySpark big data task management method, system, and storage medium according to embodiments of the present invention.

[0024] Figure 1 This is a flowchart of a PySpark big data task management method according to an embodiment of the present invention. Figure 1 As shown, a PySpark big data task management method according to an embodiment of the present invention includes the following steps: Step S1: Receive the PySpark task code submitted by the user. The code contains runtime environment dependency information defined in declarative syntax. The dependency information includes at least the base image identifier, Python version, and a list of third-party libraries.

[0025] In a specific implementation, developers do not need to perform manual operations. They only need to declare the required base image, Python version, third-party libraries and their versions in the PySpark task code through a specific SDK (Software Development Kit). This is equivalent to an environment "recipe" for the task. Specifically, this can be implemented using a dependency declaration resolution unit. The functions of this dependency declaration resolution unit include: receiving the user's PySpark task code; and the implementation method includes: an SDK interface that provides a Python library (such as dataworks_env) that allows users to define the environment through an intuitive class (such as PySparkEnv).

[0026] Specifically, according to the PySpark big data task management method of the present invention, the method first receives PySpark task code submitted by the user. The code contains runtime environment dependency information defined in declarative syntax. The dependency information includes at least the base image identifier, Python version and third-party library list. This declaratively defines the PySpark task code, thereby clarifying the user's environment dependencies and facilitating subsequent processing.

[0027] Step S2: Parse the runtime environment dependency information and generate a structured environment list.

[0028] In a specific embodiment, after receiving the task, the runtime environment dependency information is parsed to generate a structured environment list. Specifically, this can be achieved using a dependency declaration parsing unit. The function of this dependency declaration parsing unit also includes: parsing and verifying the environment dependencies declared by the user in a specific format in the PySpark task code; the implementation method includes: the declaration content includes: parsing the parameters passed by the user: (1) base_image: the base Docker image (e.g., dataworks / pyspark:3.3), which specifies the versions of Spark and Java; (2) python: the required Python interpreter version (e.g., 3.10); (3) packages: the list of required Python packages (e.g., 'pandas==2.0, scikit-learn'), which supports installation from PyPI, and also supports specifying additional index sources or installing WHL files via URL; (5) requirements_file: Alternatively, allowing the user to specify a requirements.txt file in the project, and the system will read and install all dependencies in the file; the output includes: the parsing unit converts the user's declaration into a structured, machine-readable "environment manifest" (usually a JSON or YAML file), which explicitly lists all the packages to be installed and their exact versions. This manifest is the sole basis for subsequent image building.

[0029] Specifically, according to the PySpark big data task management method of the present invention, the runtime environment dependency information is then parsed to generate a structured environment list, thereby parsing the runtime environment dependency information to generate a structured environment list, which facilitates subsequent corresponding processing based on the environment list.

[0030] Step S3: Based on the environment list, check if the corresponding cached image exists in the image repository.

[0031] In a specific implementation, the system checks whether a corresponding cached image exists in the image repository based on the environment list. Specifically, if a corresponding cached image is not found in the image repository, it is used directly.

[0032] Specifically, according to the PySpark big data task management method of the present invention, the corresponding cached image is checked in the image repository based on the environment list, so as to determine the subsequent processing method of the image based on whether the corresponding cached image is in the image repository.

[0033] Step S4: When no cached image exists, build a dynamic image, including: pulling the base image based on the base image identifier, using the base image as the parent image, executing the installation command according to the list of third-party libraries, generating a task-specific container image, and pushing the task-specific container image to the image repository.

[0034] In a specific implementation, when no cached image exists, a layered inheritance container image building strategy is adopted. Its core logic lies in improving efficiency by sharing a base layer and achieving environment customization and isolation by stacking independent layers. Specifically, building a dynamic image includes: pulling a base image based on a base image identifier, using the base image as the parent image, executing installation commands according to a list of third-party libraries, generating a task-specific container image, and pushing the task-specific container image to the image repository.

[0035] In a specific implementation, layered image technology is, for example, a Docker layer. Specifically, the first layer is a standardized, minimal PySpark base image (e.g., dataworks / pyspark:3.3); the second layer is a new container image dedicated to this task, created by running the pip install command based on the packages parameter in the declaration, which installs the required libraries onto the base image; this image is then pushed and stored in the platform's private image repository.

[0036] Specifically, according to the PySpark big data task management method of the present invention, when no cached image exists, a dynamic image is constructed, including: pulling a base image based on a base image identifier, using the base image as the parent image, executing an installation command according to a list of third-party libraries, generating a task-specific container image, and pushing the task-specific container image to an image repository. This allows for the construction of a new container image dedicated to the task, and pushing the container image to the image repository facilitates subsequent calls, thereby contributing to high efficiency during subsequent calls.

[0037] Step S5: When scheduling and executing a PySpark task, retrieve the corresponding task-specific container image from the image repository, and create a container based on the task-specific container image as the runtime environment for the PySpark task.

[0038] In a specific implementation, when a task is scheduled and executed, the scheduler does not start the Spark process directly on the bare metal. Instead, it quickly starts a container based on a pre-built image. Specifically, all the code for the task runs in this container environment, thus obtaining a completely isolated and fully dependent execution environment.

[0039] Specifically, according to the PySpark big data task management method of the present invention, when scheduling and executing a PySpark task, the corresponding task-specific container image is retrieved from the image repository, and a container is created based on the task-specific container image as the environment for running the PySpark task. In this way, strong isolation between task environments is achieved through containerization, and dependency conflicts are completely resolved. Through image layering and caching, tasks with the same dependencies share the base layer, and only the differences need to be newly created, which greatly improves the construction efficiency and storage efficiency, and realizes "cross-task reuse".

[0040] Step S6: During the execution of the PySpark task, capture the running logs in real time and stream the running logs to the big data development platform for centralized storage, indexing and display.

[0041] In specific embodiments, unified monitoring of log forwarding and centralized processing is achieved through log generation and real-time capture (data source and collection), log relay and preprocessing (data transmission and proxy), log persistence and indexing (data storage and organization), and unified visualization and querying of logs (data consumption and display).

[0042] Specifically, according to the PySpark big data task management method of the present invention, during the execution of PySpark tasks, the running logs are captured in real time and the running logs are streamed to the big data development platform for centralized storage, indexing and display. In this way, by building a real-time log pipeline, the log barrier between the Spark computing environment and the DataWorks operation and maintenance platform is broken down, and the operation and maintenance monitoring interface is unified.

[0043] Therefore, the PySpark big data task management method according to embodiments of the present invention first receives PySpark task code submitted by a user. The code contains runtime environment dependency information defined in declarative syntax. The dependency information includes at least a base image identifier, a Python version, and a list of third-party libraries. Then, the runtime environment dependency information is parsed to generate a structured environment list. Next, based on the environment list, it is checked whether a corresponding cached image exists in the image repository. When no cached image exists, a dynamic image is constructed, including: pulling a base image based on the base image identifier, using the base image as the parent image, executing installation commands according to the list of third-party libraries, generating a task-specific container image, and then... The system pushes container images to the image repository; when scheduling and executing PySpark tasks, it retrieves the corresponding task-specific container image from the image repository and creates a container based on the task-specific container image as the runtime environment for the PySpark task; during the execution of PySpark tasks, it captures runtime logs in real time and streams them to the big data development platform for centralized storage, indexing, and display; in this way, through declarative environment construction, intelligent resource scheduling, and end-to-end log integration, it solves the pain points of environment management, resource allocation, and operation and maintenance debugging of PySpark in DataWorks, which can effectively improve environment management efficiency, optimize resource utilization, and improve the operation and maintenance debugging experience.

[0044] In one embodiment of the present invention, when building the dynamic image in step S4, a layered inheritance strategy is adopted, including: the first layer is the base image layer, which is used to provide the operating system, Java runtime environment, Spark and Python basic environment; the second layer is the project common dependency layer, which is used to install the common third-party library shared by multiple tasks in the project group based on the base image layer; the third layer is the task-specific dependency layer, which is used to install the specific third-party library and its version declared by the current task based on the base image layer or the project common dependency layer.

[0045] In a specific implementation, when building a dynamic image, efficiency can be improved by sharing a base layer, and environment customization and isolation can be achieved by overlaying independent layers. Specifically, this includes establishing a unified foundation (base image), solidifying project-level common dependencies (project image), and injecting task-specific environments (task image).

[0046] In a specific implementation, the purpose of establishing a unified foundation (base image) is to provide all PySpark tasks with an absolutely consistent, stable, and repetitive underlying runtime environment. Specifically, establishing a unified foundation (base image) is achieved by building a common layer called the base image. This image contains a minimal operating system, a pre-defined version of the JDK (required for Spark to run), a pre-defined version of the Python interpreter, and core Spark dependencies. The advantages of establishing a unified foundation (base image) are: this layer is shared by all projects and environments; it only needs to be built once and stored in a repository, and can be pulled and used by all subsequent tasks, avoiding the need to repeatedly install these large, fundamental components for each task, thus greatly saving storage space and build time.

[0047] In a specific implementation, the purpose of solidifying project-level common dependencies (project images) is to reuse a common toolchain and dependency libraries at the team or project level, further reducing repetitive work. Specifically, solidifying project-level common dependencies (project images) is implemented as follows: a project image is built using a base image as the parent image. This layer "adds" and installs "common dependency packages" (such as internally developed frameworks, specific versions of database connectors, etc.) that may be used by multiple tasks within the project team. The advantages of solidifying project-level common dependencies (project images) are: it achieves project-level environment reuse; for tasks requiring these common packages, there is no need to build from the bottom layer, but can be directly based on the project image, significantly improving the efficiency of environment preparation.

[0048] In a specific implementation, the purpose of injecting a task-specific environment (task image) is to meet the unique dependency requirements of a single PySpark task, achieve customization of the final environment, and ensure isolation. Specifically, the implementation of injecting a task-specific environment (task image) is as follows: The final task image is built using the project image (or directly based on the base image if the project has no common dependencies) as the parent image. This layer installs specific third-party library versions declared in the task code (e.g., pandas==2.0) through an "injection" operation. The advantages of injecting a task-specific environment (task image) are: it achieves customization and isolation of the final environment. Each task has its own unique layer specifically for storing its specific dependencies, which ensures: Environment isolation: Even if task A and task B depend on different versions of the same library, they will not conflict because they exist in different image layers; Efficient build: Docker's layered mechanism means that if only the top-level specific dependency changes, only this thinnest layer needs to be rebuilt and stored, and all lower layers can reuse the cache.

[0049] Specifically, according to the PySpark big data task management method of this invention, a layered inheritance strategy is adopted when building a dynamic image, including: the first layer is the base image layer, which provides the operating system, Java runtime environment, Spark and Python basic environment; the second layer is the project common dependency layer, which is used to install common third-party libraries shared by multiple tasks within the project group based on the base image layer; the third layer is the task-specific dependency layer, which is used to install the specific third-party libraries declared by the current task and their versions based on the base image layer or the project common dependency layer. In order to efficiently provide a customized and isolated environment for each task, a container layering model is adopted. First, a general base layer shared by all tasks is built, then the common dependency layer shared by the project group is superimposed, and finally the specific dependency layer unique to each task is injected, resulting in a lightweight, efficient and highly isolated task runtime image. That is, efficiency is improved by sharing the base layer, and the customization and isolation of the environment are achieved by superimposing independent layers. This "layered construction and layered reuse" mechanism is the key technical foundation for achieving "declarative automatic environment construction" that can ensure environment consistency and isolation while maintaining extremely high efficiency.

[0050] In one embodiment of the present invention, before executing the PySpark task, the method further includes: performing resource prediction, specifically including: performing AST parsing on the PySpark task code to extract code structure features; combining the historical execution feature data of the task to form an input feature vector; inputting the input feature vector into a pre-trained resource prediction model to obtain the predicted resource demand level; and allocating appropriate computing resources to the task and selecting the corresponding execution cluster according to the resource demand level.

[0051] In specific embodiments, resource demand can be predicted by combining an Abstract Syntax Tree (AST) with historical data features, including AST parsing, historical feature extraction, and resource allocation decision-making. Specifically, intelligent resource scheduling decisions can be made based on machine learning. The core logic lies in transforming resource allocation from manual experience-based decision-making to data-driven automated prediction by analyzing code features and historical performance. This includes multi-dimensional data input (input layer), deep code feature extraction (feature engineering layer), and model-based prediction decision-making (intelligent decision-making layer).

[0052] In a specific embodiment, the purpose of multi-dimensional data input (input layer) is to provide comprehensive and objective decision-making basis for the resource prediction model, eliminating the subjectivity and inaccuracy of relying solely on human experience. Specifically, the multi-dimensional data input (input layer) is implemented by receiving input data from two main dimensions: current task code: the PySpark script to be submitted for execution; and historical task data: execution data of past tasks accumulated by the platform, including its code characteristics, actual resource usage (such as CPU, memory consumption, and execution time). The advantage of multi-dimensional data input (input layer) is that it combines the static characteristics of the code itself with the dynamic results of historical execution, providing the model with a deep and practical analytical foundation.

[0053] In a specific implementation, the purpose of deep code feature extraction (feature engineering layer) is to transform abstract code text into quantifiable feature metrics that characterize its computational complexity. Specifically, the implementation of deep code feature extraction (feature engineering layer) is as follows: AST parsing: The system performs lexical and syntactic analysis on the submitted PySpark code, converting it into an Abstract Syntax Tree (AST). The AST accurately expresses the logical structure of the code in a structured, tree-like form; Feature extraction: Based on the AST, the system extracts a series of key static features, such as: AST depth / complexity: reflecting the logical nesting depth and structural complexity of the code; Operation type identification: identifying whether the code contains Shuffle operations (such as groupBy, join), machine learning operations (such as MLlib library calls), data read / write operations, etc. Different types of operations have vastly different resource (CPU, network, memory) requirements; Data source / data volume estimation: by parsing the data source path or combining metadata information, the scale of the data to be processed is initially estimated. The advantages of deep code feature extraction (feature engineering layer) are: through deep code analysis, the computational load of the task can be predicted more accurately and earlier than simply based on the number of lines of code or manual labels.

[0054] In a specific embodiment, the purpose of model-based predictive decision-making (intelligent decision layer) is to automatically output the optimal resource allocation and cluster selection scheme based on the extracted features. Specifically, the implementation of model-based predictive decision-making (intelligent decision layer) is as follows: Feature matrix construction: Align the static features extracted from the current task with the features of similar tasks in historical data and their actual resource consumption labels to construct a feature vector for model inference; Decision tree model inference: Use a pre-trained decision tree (or random forest, etc.) model to infer the feature matrix. This model establishes a mapping relationship between "code features and resource requirements" by learning patterns in historical data; Decision output: The model generates the final prediction result; Resource tiers: Divide the task into preset resource tiers such as Low, Medium, and High, each tier corresponding to specific configurations such as the number of CPU cores and memory size; Cluster selection: Based on resource requirements and task characteristics (such as whether a GPU is needed), recommend the most suitable execution cluster (e.g., MaxCompute for lightweight tasks, EMR standard cluster for general medium tasks, and EMR GPU cluster for heavy machine learning tasks). The advantages of model-based predictive decision-making (intelligent decision layer) are: it realizes the automation and precision of resource allocation; model decision-making is based on a large amount of data, which is far more scientific than human judgment; it can avoid resource waste and prevent task failure due to insufficient resources, thereby improving the overall cluster utilization and task execution efficiency.

[0055] Specifically, according to embodiments of the present invention, the PySpark big data task management method further includes, before executing a PySpark task, performing resource prediction, specifically including: performing AST parsing on the PySpark task code to extract code structure features; combining historical execution feature data of the task to form an input feature vector; inputting the input feature vector into a pre-trained resource prediction model to obtain the predicted resource demand level; allocating appropriate computing resources to the task and selecting the corresponding execution cluster according to the resource demand level; thereby using a machine learning-based intelligent resource scheduling decision-making method, by analyzing code features and historical performance, transforming resource allocation from manual experience-based decision-making to data-driven automated prediction, which helps to improve the overall cluster utilization and task execution efficiency.

[0056] In one embodiment of the present invention, the code structure features include one or more of the following: AST depth, AST complexity, types of operations included, and estimated data processing volume, wherein the types of operations include data shuffle operations and machine learning library call operations.

[0057] Specifically, according to the PySpark big data task management method of the present invention, the code structure features include one or more of the following: AST depth, AST complexity, types of operations included, and estimated data processing volume. The types of operations include data shuffle operations and machine learning library call operations. This facilitates in-depth analysis of the code and enables more accurate and earlier prediction of the computational load of the task than simply based on the number of lines of code or manual labels.

[0058] In one embodiment of the present invention, the resource prediction model is a decision tree model or a random forest model, and the pre-training includes training based on historical task features and actual resource consumption data; the resource demand levels include at least three levels: low, medium and high, with different levels corresponding to different computing resource quotas.

[0059] Specifically, according to the PySpark big data task management method of the present invention, the resource prediction model is a decision tree model or a random forest model, and the pre-training includes training based on historical task characteristics and actual resource consumption data; the resource demand levels include at least three levels: low, medium, and high, with different levels corresponding to different computing resource quotas; this helps to achieve automated and accurate resource allocation. The resource prediction model is based on a large amount of data, which is far more scientific than manual judgment, and can avoid resource waste and prevent task failure due to insufficient resources, thereby improving the overall cluster utilization and task execution efficiency.

[0060] In one embodiment of the present invention, allocating appropriate computing resources to a task and selecting a corresponding execution cluster according to the resource requirement level includes: selecting a serverless MaxCompute cluster when the resource requirement level is low; or selecting an EMR standard CPU cluster when the resource requirement level is medium; or selecting an EMR GPU cluster when the resource requirement level is high, or when machine learning library call operations are detected in the code.

[0061] In specific implementations, when resource requirements are low (e.g., lightweight tasks with simple ASTs (depth < 5) and no complex data exchange (Shuffle operations < 3), a MaxCompute cluster is selected. Specifically, MaxCompute is a serverless data warehouse and computing platform. Users do not need to manage the cluster; they only pay for the actual amount of data computed and stored. For lightweight tasks, its cost is far lower than maintaining a persistent Spark cluster. MaxCompute eliminates the time required to apply for and start ECUs (Compute Units), making it suitable for handling "quick and easy" tasks such as data filtering, simple aggregation, and small-scale data synchronization. MaxCompute's fully managed service eliminates the operational burden of cluster maintenance, security patching, and resource allocation. Therefore, for simple tasks, choosing MaxCompute is the most economical and convenient solution while meeting performance requirements.

[0062] In specific implementations, when resource requirements are medium (e.g., tasks with relatively complex code logic (5 ≤ AST depth < 8) or medium-sized tasks requiring the processing of massive amounts of data (1TB), the EMR (Elastic MapReduce) standard cluster is selected. Specifically, EMR provides managed Hadoop / Spark clusters, offering powerful and customizable distributed computing capabilities. It can efficiently handle complex ETL (extract, transform, load) tasks requiring numerous shuffles and multi-step iterations. EMR users have complete control over the cluster size (number of nodes) and machine type (CPU / memory ratio), and can scale vertically or horizontally according to task load, achieving a balance between performance and cost. EMR natively supports the complete Spark ecosystem, allowing easy use of all components such as Spark SQL, Streaming, and GraphX. Therefore, for most general-purpose, medium-complexity distributed computing tasks, the EMR standard cluster is the primary choice, achieving the best balance between performance, flexibility, and cost.

[0063] In specific embodiments, when resource requirements are high (e.g., the corresponding task characteristics are: highly computationally intensive tasks involving machine learning (MLlib) operations), the EMR GPU cluster is selected. Specifically, GPUs (Graphics Processing Units) have thousands of cores, making them ideal for performing the large number of matrix operations and parallel computations in machine learning training and inference. Using GPUs can reduce the execution time of such tasks from hours to minutes. The EMR GPU cluster comes pre-installed with necessary GPU drivers and libraries such as CUDA and cuDNN, and provides management and scheduling functions for GPU resources, providing a ready-to-use hardware acceleration environment for machine learning tasks. The EMR GPU cluster physically isolates resource-intensive machine learning tasks from ordinary CPU computing tasks into different clusters, avoiding resource contention and ensuring the computing performance and data security of both. Therefore, for computationally intensive tasks such as machine learning, selecting the EMR GPU cluster is the only specialized choice that can meet their extreme computing needs and significantly improve performance.

[0064] Specifically, according to the PySpark big data task management method of the present invention, when the resource demand level is low, a serverless MaxCompute cluster is selected; when the resource demand level is medium, an EMR standard CPU cluster is selected; when the resource demand level is high, or when machine learning library call operations are detected in the code, an EMR GPU cluster is selected. This helps to automate and refine resource allocation, thereby improving the overall cluster utilization and task execution efficiency.

[0065] In one embodiment of the present invention, step S6, which captures runtime logs in real time, includes: configuring a log framework and a SocketAppender on the Driver and Executor nodes of the Spark task; and sending locally generated log events to a specified log collection agent in real time via the SocketAppender in the form of a network stream.

[0066] In a specific implementation, the purpose of log generation and real-time capture (data source and collection) is to obtain complete and real-time log data from the source, laying the foundation for subsequent centralized processing. Specifically, the implementation of log generation and real-time capture (data source and collection) is as follows: Generating native logs: During operation, the Spark Driver and Executor generate detailed runtime logs, error messages, and performance metrics through their built-in Log4j logging framework; Forwarding via SocketAppender: A key modification is made to Spark's Log4j configuration. Instead of using the default appender that writes to local files, a SocketAppender is configured. This appender sends log events in real-time, in streaming form via a network socket connection, rather than writing them to the local disk of the computing cluster. The advantages of log generation and real-time capture (data source and collection) are: it decouples logs from their generation location, avoiding the need for users or operations personnel to log to remote, potentially destroyed EMR cluster nodes to search for log files, thus creating a prerequisite for centralized log management.

[0067] Specifically, according to the PySpark big data task management method of the present invention, a log framework and SocketAppender are configured on the Driver and Executor nodes of the Spark task; the locally generated log events are sent to the specified log collection agent in real time in the form of network streams through SocketAppender; thereby capturing runtime logs in real time and creating a prerequisite for centralized log management.

[0068] In one embodiment of the present invention, step S6, which transmits logs to a big data development platform in a streaming manner, includes: receiving log streams from multiple tasks through a log collection agent; parsing and enriching the logs, adding task identifiers and metadata; and writing the processed log data in batches into the log service integrated by the big data development platform.

[0069] In a specific implementation, the purpose of log forwarding and preprocessing (data transmission and proxy) is to reliably receive log streams and perform necessary preprocessing to make them suitable for storage and retrieval. Specifically, the implementation of log forwarding and preprocessing (data transmission and proxy) is as follows: Log Agent Reception: A Socket listener is started in a resident log agent (usually an independent service process) to continuously receive log streams from the SocketAppenders of various Spark tasks; Parsing and Forwarding: The log agent can parse, filter, and enrich the logs (e.g., adding task IDs, project names, and other metadata), and then forward the structured log data in batches to the downstream persistent storage system. The advantages of log forwarding and preprocessing (data transmission and proxy) are: the log agent acts as an efficient "log router," decoupling log production and consumption. On the one hand, this reduces the I / O pressure on Spark task nodes; on the other hand, it provides log preprocessing capabilities and unifies the data format to a format supported by the downstream system.

[0070] In a specific implementation, the purpose of log persistence and indexing (data storage and organization) is to transform massive, streaming log data into data assets that can be stored long-term and queried at high speed. Specifically, the implementation of log persistence and indexing (data storage and organization) is as follows: Structured storage: The log broker writes the processed logs to DataWorksLogstore (or similar log services, such as Alibaba Cloud SLS), a high-throughput, low-latency distributed storage service designed specifically for log scenarios, capable of efficiently storing massive amounts of log data; Indexing: While storing, the Logstore service automatically or according to configuration indexes key fields in the logs (such as timestamps, log levels, task IDs, error message keywords, etc.). The advantages of log persistence and indexing (data storage and organization) are: through professional log services for storage and indexing, log data is no longer a "sleeping archive," but rather becomes a structured, quickly searchable database, enabling efficient subsequent investigation and analysis.

[0071] In a specific implementation, the purpose of unified log visualization and querying (data consumption and presentation) is to provide developers and administrators with a unified and convenient interface for viewing and analyzing logs, ultimately solving the problem of "difficult debugging." Specifically, the unified log visualization and querying (data consumption and presentation) is implemented as follows: Visual querying: The DataWorks console (management interface) is integrated with the underlying Logstore and Elasticsearch (search and analysis engine). Users can directly use query statements on the DataWorks web interface to retrieve logs of all tasks in real time, just like using a search engine. The advantages of unified log visualization and querying (data consumption and presentation) are: it achieves fundamental unification of the operation and maintenance monitoring interface, eliminating the need for users to switch back and forth between the DataWorks console and the EMR cluster. All logs, regardless of which underlying cluster the task runs on, are now centrally aggregated within the DataWorks platform, supporting full-text search, multi-dimensional filtering, and contextual browsing, greatly improving the efficiency of fault location and operational monitoring.

[0072] Specifically, according to the PySpark big data task management method of the present invention, log streams from multiple tasks are received through a log collection agent; the logs are parsed and enriched, and task identifiers and metadata are added; the processed log data is written in batches to the log service integrated into the big data development platform; thereby transmitting the logs to the big data development platform in a streaming manner, which helps to achieve centralized management of logs.

[0073] The PySpark big data task management method of the present invention will be further described below with reference to a specific embodiment. In this specific embodiment, a PySpark big data task management method is provided: Figure 2 This is an overall architecture diagram of a PySpark big data task management method according to a specific embodiment of the present invention, as shown below. Figure 2 As shown in this specific embodiment, the PySpark big data task management method includes declarative environment construction, intelligent resource scheduling, and end-to-end log integration.

[0074] In this specific embodiment, to address the problems of inefficiency and inability to guarantee the consistency and isolation of the environment due to complete reliance on manual operation in the prior art, a combination of declarative dependency management and containerization technology is adopted to fundamentally automate this process.

[0075] In this specific embodiment, declarative definition means that developers do not need to operate manually. They only need to declare the required base image, Python version, third-party libraries and their versions in the PySpark task code through a specific SDK. This is equivalent to an environment "recipe" for the task.

[0076] In this specific embodiment, dynamic image building refers to the platform receiving a task and parsing the declaration: First, it checks if a cached image that meets the declaration requirements exists. If so, it uses it directly; otherwise, it starts an automated image building process based on layered image technology (such as Docker layers). First layer: A standardized, minimal PySpark base image (e.g., dataworks / pyspark:3.3).

[0077] The second layer: Based on the packages parameter in the declaration, the pip install command is run to install the required libraries onto the base image, forming a new container image dedicated to this task.

[0078] This image was pushed and stored in the platform's private image repository.

[0079] In this specific embodiment, when the task is scheduled and executed, the scheduler does not start the Spark process directly on the bare metal. Instead, it quickly starts a container based on the image built above. All the code of the task runs in this container environment, thereby obtaining a completely isolated and fully dependent execution environment.

[0080] In this specific embodiment, the combination of declarative dependency management and containerization technology addresses the issues of automating environment preparation and isolating environments. By replacing manual operations with "declaration," the environment configuration process is transformed from the developer's responsibility to an automated service of the platform. Through containerization, strong isolation between task environments is achieved, completely resolving dependency conflicts. Through image layering and caching, tasks with the same dependencies share the base layer, and only the differences need to be rebuilt, greatly improving build efficiency and storage efficiency, and achieving "cross-task reuse."

[0081] In this specific embodiment, the combination of declarative dependency management and containerization technology adopts the logic of "manual configuration is inefficient and error-prone - declarative definition separates intent from implementation - containerization technology is used to automatically and isolatedly build the environment - obtain efficient, consistent, isolated and reproducible environment management capabilities", which has the following advantages: (1) Improve development efficiency: developers are freed from tedious environment operation and maintenance, and only need to focus on business logic. The environment preparation time is reduced from hours / minutes to seconds (pulling images); (2) Ensure environment consistency: eliminate the problem of "it works well here", and development, testing and production environments can achieve complete consistency by declaring the same "recipe"; (3) Achieve environment isolation: different tasks and different versions of dependencies do not interfere with each other, which improves the overall stability and reliability of the platform; (4) Enhance reproducibility: the dependencies of tasks are defined in plain text and versioned, and the operating environment at that time can be accurately reproduced at any time, which is convenient for auditing and troubleshooting; (5) Optimize resource utilization: the image layering and sharing mechanism reduces the repeated occupation of disk space.

[0082] In this specific embodiment, the core task of declarative environment automatic construction is to automatically and efficiently transform the user's "description" (declaration) of the environment into a fully isolated, ready-to-use runtime container environment; its workflow and technical details are as follows: The dependency declaration resolution unit is the starting point and input interface of the entire process; its function is to receive, parse, and verify the environment dependencies declared by the user in a specific format in the PySpark task code; its implementation includes: SDK Interface: Provides a Python library (such as dataworks_env) that allows users to define environments through an intuitive class (such as PySparkEnv).

[0083] Declaration content: Parses user-provided parameters: base_image: The base Docker image (e.g., dataworks / pyspark:3.3), specifying the versions of Spark and Java; python: The required Python interpreter version (e.g., 3.10); packages: A list of required Python packages (e.g., 'pandas==2.0, scikit-learn'). Installation from PyPI is supported, as well as specifying additional index sources or installing WHL files via URL; requirements_file: Alternatively, allows the user to specify a requirements.txt file in the project, which the system will read and install all dependencies within that file.

[0084] Output: The parsing unit converts the user's declaration into a structured, machine-readable "environment manifest" (usually a JSON or YAML file) that explicitly lists all packages to be installed and their exact versions. This manifest is the sole basis for subsequent image building.

[0085] Figure 3 This is a schematic diagram of a layered container image according to a specific embodiment of the present invention, as shown below. Figure 3 As shown in the figure, in this specific embodiment, the diagram illustrates a container image building strategy based on layered inheritance. Its core logic lies in improving efficiency by sharing a base layer and achieving environment customization and isolation by stacking independent layers; the process is as follows: (1) Establish a unified foundation (base image): Its purpose is to provide an absolutely consistent, stable and repetitive underlying runtime environment for all PySpark tasks. Its implementation is to build a common layer called the base image, which contains a minimal operating system, a pre-defined version of JDK (required for Spark to run), a pre-defined version of the Python interpreter, and core Spark dependencies. Its advantages are that this layer is shared by all projects and environments. It only needs to be built once and stored in the repository, and can be pulled and used by all subsequent tasks. This avoids repeatedly installing these large and basic components for each task, which greatly saves storage space and build time.

[0086] (2) Solidify project-level common dependencies (project image): Its purpose is to reuse a common toolchain and dependency library at the team or project level, further reducing repetitive work; its implementation method is to build a project image with the base image as the parent image. This layer will "add" the installation of "common dependency packages" that may be used by multiple tasks within the project team (such as the company's self-developed framework, a specific version of the database connector, etc.); its advantage is that it realizes project-level environment reuse. For tasks that need these common packages, there is no need to start building from the bottom layer. It can be directly based on the project image, which significantly improves the efficiency of environment preparation.

[0087] (3) Injecting a task-specific environment (task image): Its purpose is to meet the unique dependency requirements of a single PySpark task, realize the customization of the final environment, and ensure isolation. The implementation method is to build the final task image with the project image (or directly based on the base image if the project has no common dependencies) as the parent image. This layer installs the specific third-party library version declared in the task code (such as pandas==2.0) through the "injection" operation. Its advantages include: It realizes the customization and isolation of the final environment. Each task has its own unique layer to store its specific dependencies. This makes: Environment isolation: Even if task A and task B depend on different versions of the same library, they will not conflict with each other because they exist in different image layers; Efficient build: Docker's hierarchical mechanism means that if only the specific dependency of the top layer changes, only the thinnest layer needs to be rebuilt and stored, and the lower layers can all reuse the cache.

[0088] In this specific embodiment, the container image building strategy based on layered inheritance adopts the logic of "in order to efficiently provide a customized and isolated environment for each task - adopting a container layering model - first building a common base layer shared by all tasks - then superimposing a common dependency layer shared by the project team - finally injecting a specific dependency layer unique to each task - ultimately obtaining a lightweight, efficient, and highly isolated task runtime image". This mechanism of "layered building and layer reuse" is the key technical foundation for achieving "declarative automatic environment building" that can ensure both environment consistency and isolation while maintaining extremely high efficiency.

[0089] Figure 4 This is a schematic diagram of intelligent resource scheduling decision-making according to a specific embodiment of the present invention, such as... Figure 4 As shown in this specific embodiment, a resource prediction model workflow is provided, including: AST syntax tree parsing, historical feature extraction, and resource tier decision-making. Figure 4 This paper describes a machine learning-based intelligent resource scheduling decision-making method. Its core logic lies in transforming resource allocation from manual experience-based decision-making to data-driven automated prediction by analyzing code characteristics and historical performance. The process is as follows: (1) Multi-dimensional data input (input layer): Its purpose is to provide a comprehensive and objective basis for decision-making for resource prediction models, and to eliminate the subjectivity and inaccuracy of relying solely on human experience; its implementation method is: the system receives input data in two dimensions: current task code: the PySpark script to be submitted for execution; historical task data: the execution data of past tasks accumulated by the platform, including its code characteristics, actual resource usage (such as CPU, memory consumption, execution time, etc.); its advantage is that it combines the static characteristics of the code itself with the dynamic results of historical operation, providing the model with an in-depth and practical analytical foundation.

[0090] (2) Deep code feature extraction (feature engineering layer): Its purpose is to transform abstract code text into quantifiable feature indicators that characterize its computational complexity. Its implementation methods include: AST parsing: The system performs lexical and syntactic analysis on the submitted PySpark code and converts it into an abstract syntax tree (AST). The AST accurately expresses the logical structure of the code in a structured, tree-like form. Feature extraction: Based on the AST, the system extracts a series of key static features, such as: AST depth / complexity: Reflects the logical nesting depth and structural complexity of the code. Operation type identification: Identifies whether the code contains Shuffle operations (such as groupBy, join), machine learning operations (such as MLlib library calls), data read and write operations, etc. Different types of operations have huge differences in resource (CPU, network, memory) requirements. Data source / data volume estimation: By parsing the data source path or combining metadata information, the scale of the data to be processed is initially estimated. Its advantage is that through deep code parsing, the computational load of the task can be predicted more accurately and earlier than simply based on the number of lines of code or manual labels.

[0091] (3) Model-based prediction and decision-making (intelligent decision layer): Its purpose is to automatically output the optimal resource allocation and cluster selection scheme based on the extracted features; its implementation methods include: Feature matrix construction: Align the static features extracted from the current task with the features of similar tasks in historical data and their actual resource consumption labels to construct feature vectors for model inference; Decision tree model inference: Use a pre-trained decision tree (or random forest, etc.) model to infer the feature matrix. This model establishes a mapping relationship between "code features and resource requirements" by learning patterns in historical data; Decision output: The model generates the final prediction results: Resource level: Divide the task into preset resource levels such as Low, Medium, and High. Each level corresponds to specific configurations such as the number of CPU cores and memory size; Cluster selection: Recommend the most suitable execution cluster (e.g., MaxCompute for lightweight tasks, EMR standard cluster for general medium tasks, EMR) based on resource requirements and task characteristics (e.g., whether GPU is needed). GPU clusters are used for heavy machine learning tasks. The logic is as follows: they automate and refine resource allocation, and model decisions are based on a large amount of data, which is far more scientific than human judgment. This can avoid resource waste and prevent task failures caused by insufficient resources, thereby improving the overall cluster utilization and task execution efficiency.

[0092] Example table of feature combinations, resource tiers, and cluster types

[0093] In this specific embodiment, as shown in the table above, an example table of feature combinations, resource tiers, and cluster types is provided, specifically: MaxCompute is suitable for lightweight tasks with a low AST (depth < 5) and no complex data exchange (Shuffle operations < 3). Its advantages include: Cost-effectiveness: MaxCompute is a serverless data warehouse and computing platform. Users do not need to manage a cluster; they only pay for the actual amount of data computed and stored. For lightweight tasks, its cost is far lower than maintaining a persistent Spark cluster. Fast delivery: It eliminates the time required to apply for and start ECUs (Compute Units), making it suitable for "quick and easy" tasks such as data filtering, simple aggregation, and small-scale data synchronization. Simplified management: It is a fully managed service, eliminating the burden of cluster maintenance, security patching, and resource allocation. Therefore, for simple tasks, choosing MaxCompute is the most economical and convenient solution while meeting performance requirements.

[0094] EMR standard clusters are suitable for tasks with medium-level complexity, such as those with relatively complex code logic (5 ≤ AST depth < 8) or those requiring the processing of massive amounts of data (1TB). Their advantages include: Performance and flexibility: EMR (Elastic MapReduce) provides a managed Hadoop / Spark cluster with powerful, customizable distributed computing capabilities, efficiently handling complex ETL (extraction, transformation, and loading) tasks requiring extensive shuffling and multi-step iterations; Controllable resources: Users have complete control over the cluster's size (number of nodes) and machine type (CPU / memory ratio), and can scale vertically or horizontally according to task load, achieving a balance between performance and cost; Complete ecosystem: Natively supports the complete Spark ecosystem, allowing easy use of all components such as Spark SQL, Streaming, and GraphX. Therefore, for most general-purpose, medium-complexity distributed computing tasks, the EMR standard cluster is the primary choice that achieves the best balance between performance, flexibility, and cost.

[0095] EMR GPU clusters are suitable for high-performance, computationally intensive tasks involving machine learning (MLlib) operations. Their benefits include: Hardware acceleration: GPUs (Graphics Processing Units) with thousands of cores are ideal for performing the massive matrix operations and parallel computations required for machine learning training and inference, reducing execution time from hours to minutes; Professional support: EMR GPU clusters come pre-installed with essential GPU drivers and libraries such as CUDA and cuDNN, and provide management and scheduling capabilities for GPU resources, offering a ready-to-use hardware acceleration environment for machine learning tasks; Resource isolation: Physically isolating resource-intensive machine learning tasks from ordinary CPU computing tasks into different clusters avoids resource contention and ensures the computational performance and data security of both. Therefore, for computationally intensive tasks such as machine learning, choosing an EMR GPU cluster is the only professional option that can meet their extreme computational demands and significantly improve performance.

[0096] Figure 5 This is a schematic diagram of a log collection framework according to a specific embodiment of the present invention, as shown below. Figure 5 As shown in this specific embodiment, a unified monitoring method based on log forwarding and centralized processing is provided. Its core logic lies in building a real-time log pipeline to break down the log barriers between the Spark computing environment and the DataWorks operations and maintenance platform, thereby achieving a unified operations and maintenance monitoring interface. The process is as follows: (1) Log generation and real-time capture (data source and collection): Its purpose is to obtain complete and real-time log data from the source, laying the foundation for subsequent centralized processing; its implementation methods include: generating native logs: During the operation of Spark Driver and Executor, detailed running logs, error messages and performance indicators will be generated through its built-in Log4j logging framework; forwarding through SocketAppender: a key modification is made to Spark's Log4j configuration. Instead of using the default Appender that writes to local files, a SocketAppender is configured. This Appender will send log events in real time and in streaming form through a network Socket connection, instead of writing them to the local disk of the computing cluster; its advantage is that it decouples the logs from the place of generation, avoiding the need for users or operations personnel to log in to remote EMR cluster nodes that may be destroyed at any time to search for log files, creating a premise for centralized log management.

[0097] (2) Log forwarding and preprocessing (data transmission and proxy): Its purpose is to reliably receive log streams and perform necessary preprocessing to make them suitable for storage and querying. Its implementation includes: Log agent reception: In a resident log agent (usually an independent service process), a Socket listener is started to continuously receive log streams sent from the SocketAppenders of various Spark tasks; Parsing and forwarding: The log agent can parse, filter, enrich (e.g., add task ID, project name and other metadata) the logs, and then forward the structured log data in batches to the downstream persistent storage system. Its logic is: The log agent acts as an efficient "log router", decoupling log production and consumption. On the one hand, it reduces the I / O pressure of Spark task nodes, and on the other hand, it provides the ability to preprocess logs and unifies the data format to the format supported by the downstream system.

[0098] (3) Log persistence and indexing (data storage and organization): The purpose is to transform massive, streaming log data into data assets that can be stored for a long time and queried at high speed. The implementation methods include: structured storage: the log broker writes the processed logs to DataWorks Logstore (or other similar log services, such as Alibaba Cloud SLS), which is a high-throughput, low-latency distributed storage service designed specifically for log scenarios, capable of efficiently storing massive log data; indexing: while storing, the Logstore service will automatically or according to the configuration index the key fields in the log (such as timestamp, log level, task ID, error message keywords, etc.); its advantage is that by storing and indexing through professional log services, the log data is no longer a "sleeping archive", but becomes a structured database that can be quickly retrieved, providing the possibility for subsequent efficient investigation and analysis.

[0099] (4) Unified visualization and query of logs (data consumption and display): Its purpose is to provide developers and administrators with a unified and convenient interface to view and analyze logs, and ultimately solve the problem of "difficult debugging"; its implementation method is: visualization query: the DataWorks console (management interface) is integrated with the underlying Logstore and Elasticsearch (search analysis engine). Users can directly use query statements on the DataWorks Web interface to retrieve the logs of all tasks in real time, just like using a search engine; its advantages are: it realizes the fundamental unification of the operation and maintenance monitoring interface. Users no longer need to switch back and forth between the DataWorks console and the EMR cluster. All logs, regardless of which underlying cluster the task runs on, are now centrally gathered in the DataWorks platform, supporting full-text search, multi-dimensional filtering and contextual browsing, which greatly improves the efficiency of fault location and operation monitoring.

[0100] In this specific embodiment, the PySpark big data task management method, after experimental verification, can improve the construction efficiency by 5 times through declarative environment construction and dependency isolation; reduce cluster costs by 35% and reduce task queuing by 60% through intelligent resource scheduling; and accelerate fault location by 70% through end-to-end log integration.

[0101] In summary, the PySpark big data task management method according to embodiments of the present invention first receives PySpark task code submitted by a user. This code contains runtime environment dependency information defined using declarative syntax. The dependency information includes at least a base image identifier, a Python version, and a list of third-party libraries. Then, the runtime environment dependency information is parsed to generate a structured environment list. Next, based on the environment list, the system checks if a corresponding cached image exists in the image repository. If no cached image exists, a dynamic image is constructed, including: pulling a base image based on the base image identifier, using the base image as the parent image, executing installation commands according to the list of third-party libraries, generating a task-specific container image, and then... Dedicated container images are pushed to the image repository; when scheduling and executing PySpark tasks, the corresponding task-specific container image is retrieved from the image repository, and a container is created based on the task-specific container image as the runtime environment for the PySpark task; during the execution of PySpark tasks, runtime logs are captured in real time and streamed to the big data development platform for centralized storage, indexing, and display; in this way, through declarative environment construction, intelligent resource scheduling, and end-to-end log integration, the pain points of PySpark environment management, resource allocation, and operation and maintenance debugging in DataWorks are solved, which can effectively improve environment management efficiency, optimize resource utilization, and improve the operation and maintenance debugging experience.

[0102] Further embodiments of the present invention also disclose a PySpark big data task management system. Figure 6 This is a structural block diagram of a PySpark big data task management system according to an embodiment of the present invention, as follows: Figure 6 As shown, the PySpark big data task management system 100 includes: a receiving module 110, a parsing module 120, an inspection module 130, a building module 140, a scheduling module 150, and a log service module 160.

[0103] Specifically, the receiving module 110 is used to receive PySpark task code submitted by the user. The code contains runtime environment dependency information defined in declarative syntax. The dependency information includes at least the base image identifier, Python version, and a list of third-party libraries.

[0104] Parsing module 120 is used to parse declarative syntax and generate a structured environment manifest.

[0105] The inspection module 130 is used to check whether the corresponding cached image exists in the image repository based on the environment list.

[0106] The build module 140 is used to build a dynamic image when no cached image exists, including: pulling a base image based on the base image identifier, using the base image as the parent image, executing installation commands according to the list of third-party libraries, generating a task-specific container image, and pushing the task-specific container image to the image repository.

[0107] The scheduling module 150 is used to pull the corresponding task-specific container image from the image repository when scheduling and executing PySpark tasks, and to create a container based on the task-specific container image as the runtime environment for the task.

[0108] The log service module 160 is used to capture runtime logs in real time during the execution of PySpark tasks and stream the logs to the big data development platform for centralized storage, indexing and display.

[0109] In one embodiment of the present invention, the building module 140 adopts a layered inheritance strategy when building a dynamic image, including: a first layer is a base image layer, which provides an operating system, a Java runtime environment, and basic environments for Spark and Python; a second layer is a project common dependency layer, which is used to install common third-party libraries shared by multiple tasks within the project group based on the base image layer; and a third layer is a task-specific dependency layer, which is used to install specific third-party libraries and their versions declared by the current task based on the base image layer or the project common dependency layer.

[0110] In one embodiment of the present invention, before executing the PySpark task, the scheduling module 150 is further configured to perform resource prediction, specifically including: performing AST parsing on the PySpark task code to extract code structure features; combining the historical execution feature data of the task to form an input feature vector; inputting the input feature vector into a pre-trained resource prediction model to obtain the predicted resource demand level; and allocating appropriate computing resources to the task and selecting the corresponding execution cluster according to the resource demand level.

[0111] In one embodiment of the present invention, the code structure features include one or more of the following: AST depth, AST complexity, types of operations included, and estimated data processing volume, wherein the types of operations include data shuffle operations and machine learning library call operations.

[0112] In one embodiment of the present invention, the resource prediction model is a decision tree model or a random forest model, and the pre-training includes training based on historical task features and actual resource consumption data; the resource demand levels include at least three levels: low, medium and high, with different levels corresponding to different computing resource quotas.

[0113] In one embodiment of the present invention, the scheduling module 150 allocates appropriate computing resources to the task and selects the corresponding execution cluster according to the resource demand level, including: when the resource demand level is low, selecting a serverless MaxCompute cluster; or, when the resource demand level is medium, selecting an EMR standard CPU cluster; or, when the resource demand level is high, or when machine learning library call operations are detected in the code, selecting an EMR GPU cluster.

[0114] In one embodiment of the present invention, the log service module 160 captures runtime logs in real time, including: configuring a log framework and SocketAppender on the Driver and Executor nodes of the Spark task; and sending locally generated log events to a specified log collection agent in real time via SocketAppender in the form of a network stream.

[0115] In one embodiment of the present invention, the log service module 160 transmits logs to the big data development platform in a streaming manner, including: receiving log streams from multiple tasks through a log collection agent; parsing and enriching the logs, adding task identifiers and metadata; and writing the processed log data in batches into the log service integrated by the big data development platform.

[0116] According to an embodiment of the present invention, the PySpark big data task management system 100 executes the PySpark big data task management method described above. First, it receives PySpark task code submitted by a user. The code contains runtime environment dependency information defined in declarative syntax. The dependency information includes at least a base image identifier, a Python version, and a list of third-party libraries. Then, it parses the runtime environment dependency information to generate a structured environment list. Next, based on the environment list, it checks whether a corresponding cached image exists in the image repository. If no cached image exists, it constructs a dynamic image, including: pulling a base image based on the base image identifier, using the base image as the parent image, executing installation commands according to the list of third-party libraries, and generating the task. Dedicated container images are used, and these images are pushed to the image repository. When scheduling and executing PySpark tasks, the corresponding task-specific container image is retrieved from the image repository, and a container is created based on this image to serve as the runtime environment for the PySpark task. During the execution of PySpark tasks, runtime logs are captured in real time and streamed to the big data development platform for centralized storage, indexing, and display. Through declarative environment construction, intelligent resource scheduling, and end-to-end log integration, this approach addresses the pain points of environment management, resource allocation, and operation and maintenance debugging in PySpark within DataWorks, effectively improving environment management efficiency, optimizing resource utilization, and enhancing the operation and maintenance debugging experience.

[0117] A further embodiment of the present invention discloses a computer-readable storage medium storing a PySpark big data task management program, which, when executed by a processor, implements the PySpark big data task management method as described in any of the above embodiments of the present invention.

[0118] According to embodiments of the present invention, when a PySpark big data task management program stored on a computer-readable storage medium is executed by a processor, it performs the PySpark big data task management method described above. First, it receives PySpark task code submitted by a user. The code contains runtime environment dependency information defined in declarative syntax. The dependency information includes at least a base image identifier, a Python version, and a list of third-party libraries. Then, it parses the runtime environment dependency information to generate a structured environment list. Next, it checks whether a corresponding cached image exists in the image repository based on the environment list. When no cached image exists, it constructs a dynamic image, including: pulling a base image based on the base image identifier, using the base image as the parent image, and executing the PySpark big data task management method according to the list of third-party libraries. The system executes installation commands to generate a task-specific container image and pushes it to the image repository. When scheduling and executing PySpark tasks, it retrieves the corresponding task-specific container image from the image repository and creates a container based on the image image as the runtime environment for the PySpark task. During the execution of PySpark tasks, it captures runtime logs in real time and streams them to the big data development platform for centralized storage, indexing, and display. Through declarative environment construction, intelligent resource scheduling, and end-to-end log integration, it solves the pain points of PySpark environment management, resource allocation, and operation and maintenance debugging in DataWorks, effectively improving environment management efficiency, optimizing resource utilization, and enhancing the operation and maintenance debugging experience.

[0119] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0120] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A PySpark big data task management method, characterized in that, include: Receive PySpark task code submitted by the user, the code containing runtime environment dependency information defined in declarative syntax, the dependency information including at least the base image identifier, Python version and a list of third-party libraries; Parse the runtime environment dependency information to generate a structured environment list; Based on the environment list, check if the corresponding cached image exists in the image repository; When the cached image does not exist, a dynamic image is constructed, including: pulling the base image based on the base image identifier, using the base image as the parent image, executing the installation command according to the third-party library list, generating a task-specific container image, and pushing the task-specific container image to the image repository; When scheduling and executing a PySpark task, the corresponding task-specific container image is retrieved from the image repository, and a container is created based on the task-specific container image as the runtime environment for the PySpark task. During the execution of the PySpark task, runtime logs are captured in real time and streamed to a big data development platform for centralized storage, indexing, and display.

2. The PySpark big data task management method according to claim 1, characterized in that, When constructing the dynamic image, a hierarchical inheritance strategy is adopted, including: The first layer is the base image layer, which provides the operating system, Java runtime environment, Spark and Python basic environment; The second layer is the project common dependency layer, which is used to install common third-party libraries shared by multiple tasks within the project group based on the base image layer; The third layer is the task-specific dependency layer, which is used to install specific third-party libraries and their versions declared by the current task based on the base image layer or the project's public dependency layer.

3. The PySpark big data task management method according to claim 1, characterized in that, Before executing the PySpark task, the process also includes: performing resource prediction, specifically including: Perform AST parsing on the PySpark task code to extract code structure features; By combining the historical execution feature data of the task, an input feature vector is formed; The input feature vector is input into a pre-trained resource prediction model to obtain the predicted resource demand level. Based on the resource requirement level, allocate appropriate computing resources to the task and select the corresponding execution cluster.

4. The PySpark big data task management method according to claim 3, characterized in that, The code structure features include one or more of the following: AST depth, AST complexity, types of operations included, and estimated data processing volume. The types of operations include data shuffle operations and machine learning library call operations.

5. The PySpark big data task management method according to claim 3, characterized in that, The resource prediction model is a decision tree model or a random forest model, and the pre-training includes training based on historical task features and actual resource consumption data. The resource demand tiers include at least three tiers: low, medium, and high, with each tier corresponding to a different computing resource quota.

6. The PySpark big data task management method according to claim 3 or 5, characterized in that, The step of allocating appropriate computing resources to the task and selecting the corresponding execution cluster according to the resource requirement level includes: When resource requirements are low, choose a serverless MaxCompute cluster; or, When resource requirements are at a medium level, select the EMR standard CPU cluster; or... When resource requirements are high, or when machine learning library calls are detected in the code, select the EMR GPU cluster.

7. The PySpark big data task management method according to claim 1, characterized in that, The real-time capture of runtime logs includes: Configure the logging framework and SocketAppender on the Driver and Executor nodes of the Spark task; The SocketAppender sends locally generated log events to the specified log collection agent in real time as a network stream.

8. The PySpark big data task management method according to claim 7, characterized in that, The process of streaming logs to the big data development platform includes: The log collection agent receives log streams from multiple tasks. The logs are parsed and enriched, and task identifiers and metadata are added. The processed log data is written in batches to the log service integrated into the big data development platform.

9. A PySpark big data task management system, characterized in that, include: The receiving module is used to receive PySpark task code submitted by users. The code contains runtime environment dependency information defined in declarative syntax. The dependency information includes at least the base image identifier, Python version, and a list of third-party libraries. The parsing module is used to parse the declarative syntax and generate a structured environment manifest; The inspection module is used to check whether a corresponding cached image exists in the image repository based on the environment list; The build module is used to build a dynamic image when the cached image does not exist. The build module includes: pulling a base image based on the base image identifier, using the base image as the parent image, executing an installation command according to the third-party library list, generating a task-specific container image, and pushing the task-specific container image to the image repository. The scheduling module is used to pull the corresponding task-specific container image from the image repository when scheduling and executing a PySpark task, and create a container based on the task-specific container image as the runtime environment for the task. The log service module is used to capture runtime logs in real time during the execution of the PySpark task and stream the logs to the big data development platform for centralized storage, indexing, and display.

10. A computer-readable storage medium storing a PySpark big data task management program, wherein the PySpark big data task management program, when executed by a processor, implements the PySpark big data task management method as described in any one of claims 1-8.

Images