Space science data processing production flow orchestration and resource scheduling system and method

Abstract

The application relates to a space science data processing production flow arrangement and resource scheduling system and method, and relates to the technical field of resource scheduling.The system comprises a container management module, a workflow design arrangement module and a resource scheduling module.The container management module is used for encapsulating space science data processing algorithms based on containerization technology, generating a standard container image and a container configuration file; the workflow design arrangement module is used for generating a workflow description file; the resource scheduling module is used for allocating computing resources to a workflow and scheduling multiple data processing tasks based on resource monitoring information and the workflow description file; and the workflow execution monitoring module is used for collecting execution information in a workflow execution process, and displaying the execution information and resource use of the workflow based on a visual display interface.The application realizes the technical effects of effectively encapsulating space science data processing algorithms, designing a workflow, allocating resources and monitoring.

Description

Technical Field

[0001] This invention relates to the field of space science data processing and production technology, and in particular to a space science data processing production process orchestration and resource scheduling system and method. Background Technology

[0002] The space station mission of China's manned space program will conduct multidisciplinary space science experiments, covering multiple scientific research fields such as space life sciences and human research, microgravity physics, space astronomy and earth sciences, and new space technologies and applications. The scientific data and engineering parameters generated by these experiments are diverse and complex, with varied data types, complex formats, and different data processing logics. Furthermore, the deployment and management of the processing, analysis, and operational environments are extremely complex. As space science research continues to deepen, this data is increasing dramatically. How to efficiently and accurately process and produce this data has become a key factor in promoting the development of space science. Effective processing and production of this data will not only provide researchers with more accurate research evidence but also accelerate the progress of space science research, providing stronger support for human exploration of the universe.

[0003] In traditional data processing, traditional virtualization technology is often used to solve the problem of software environment dependency. This technology creates independent hardware environments and dynamically deploys the required software stack.

[0004] However, when faced with the complex challenges of heterogeneous data, parallel execution of multiple tasks, and traceability of the processing in space science data processing, existing technologies have limited capabilities in handling heterogeneous data, coordinating parallel execution of multiple tasks, and ensuring traceability of the processing process. This results in low data processing efficiency, difficulty in error localization, and low resource utilization. Moreover, in terms of the reliability and security of data processing, the lack of end-to-end monitoring and automated fault-tolerance mechanisms makes it difficult to guarantee the continuity and stability of mission data processing during the long-term on-orbit operation of the space station. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a system and method for arranging and scheduling production processes and resources for space science data processing, aiming to solve at least one of the above-mentioned technical problems.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: In a first aspect, this application provides a space science data processing production workflow orchestration and resource scheduling system, which adopts the following technical solution: A space science data processing production workflow orchestration and resource scheduling system, comprising: The container management module is used to acquire space science data processing algorithms, encapsulate the space science data processing algorithms based on containerization technology, and generate standard container images and container configuration files. The workflow design and orchestration module is used to generate a workflow description file in response to the user's trigger action on the graphical drag-and-drop function of components in the visual workflow design interface. The workflow description file represents the data processing process composed of multiple container images or programs. The resource scheduling module is used to acquire resource monitoring information in real time, and based on the resource monitoring information and the workflow description file, allocate computing resources to the workflow and schedule multiple data processing tasks. The workflow execution monitoring module is used to collect execution information during the workflow execution process and display the execution information and resource usage of the workflow based on a visual display interface. The execution information includes execution logs, performance indicators and execution status.

[0007] The beneficial effects of this invention are as follows: The container management module encapsulates space science data processing algorithms in a containerized manner, achieving standardization and isolation of the data processing environment, and improving the system's portability and maintainability; the workflow design and orchestration module generates workflow description files through a visual design interface, reducing the complexity of space science data processing and improving the efficiency of workflow orchestration; the resource scheduling module allocates computing resources and schedules multiple tasks based on real-time resource monitoring information, realizing dynamic allocation and efficient utilization of computing resources, and supporting the parallel execution of large-scale data processing tasks; the workflow execution monitoring module collects and displays workflow execution information and resource usage, realizing transparent management of the workflow execution process, facilitating problem localization and performance optimization.

[0008] Based on the above technical solution, the present invention can be further improved as follows.

[0009] Furthermore, the container management module includes: A container image repository building unit, used to build container image repositories; A containerization encapsulation tool unit is used to identify the language type of the space science data processing algorithm, generate a standard container image based on the language type of the space science data processing algorithm and a pre-set template library, generate a container configuration file corresponding to the standard container image based on the dependency relationship, and store the standard container image and container configuration file in the container image repository. The security scanning unit is used to obtain uploaded container images from the container image repository, perform security vulnerability assessments on the operating system layer, middleware layer, and application layer of the container images based on a layered scanning strategy and a vulnerability database, and obtain the vulnerability level of the container images; based on the vulnerability level of the container images and the corresponding handling strategy, determine the remediation recommendations for the container images. The image optimization unit is used to analyze the hierarchical structure of the container image based on the repair suggestions and hierarchical analysis algorithm of the container image, and obtain the hierarchical analysis results; based on the multi-stage build technology and the hierarchical analysis results of the container image, optimize the container image, generate the optimized container image and store it in the image repository; The container runtime environment adaptation unit is used to select the container image runtime environment based on the deployment environment characteristics of the container image.

[0010] The beneficial effects of adopting the above-mentioned further solutions are as follows: The various units within the container management module work collaboratively to achieve containerized encapsulation and management of space science data processing algorithms. The container image repository building unit constructs a repository, facilitating the storage and management of container images; the containerized encapsulation tool unit can generate and store standard container images and configuration files based on algorithm language types and pre-set templates, improving encapsulation efficiency and standardization; the security scanning unit performs security assessments on uploaded images and determines remediation suggestions, ensuring image security; the image optimization unit optimizes images based on remediation suggestions and hierarchical analysis, reducing image size and improving performance; the container runtime environment adaptation unit selects a suitable runtime environment based on the deployment environment, ensuring compatibility across different computing environments, thereby improving system portability and maintainability, and providing a stable and secure foundation for subsequent workflow design and resource scheduling.

[0011] Furthermore, the container image repository building unit is also used for: In response to a user's trigger action for the container image upload function, the system obtains the container image sent by the user, as well as the name, type, provider, image description, version number, version description, and program of the container image, and stores the container image in the container image repository. In response to a user's trigger action for updating the container image version, obtain the version number, version description, and updated program of the container image to be updated, and update the container image in the workflow template based on the version number, version description, and updated program of the container image to be updated.

[0012] The beneficial effects of adopting the above-mentioned further solutions are: the container image repository building unit supports the uploading, downloading, version management, and metadata recording of container images, and provides image signing and integrity verification; after an image version is uploaded, the image version in the workflow template can be updated through the "update workflow image version" operation, and different versions of the executable program can also be selected during the workflow configuration process. This meets the operational needs of different users for container images, facilitates the containerized encapsulation and use of space science data processing algorithms, and improves the maintainability and flexibility of the system.

[0013] Furthermore, the workflow design and orchestration module includes: The workflow step library management unit is used to set up a workflow step library, obtain the functions of creating, editing, deleting, copying, importing, exporting or managing historical versions of workflows set by users, set up and manage the processing steps corresponding to common space science data business processing flows; wherein, the workflow step library includes multiple data processing workflows; The graphical designer unit is used to design the interface and user permissions based on the visual workflow, obtain the drag-and-drop component function set by the user, and generate a workflow description file of algorithm steps in real time based on JSON. The template management unit is used to store standardized workflow templates; it is also used to obtain user-defined workflow template import or export functions; and it is also used to obtain user-defined workflow template overriding or workflow creation functions. The workflow verification unit is used to perform compliance verification on the workflow description file based on static analysis technology. The compliance verification includes dependency loop detection, parameter compliance verification, and resource constraint verification. The dependency loop detection is used to identify circular dependencies in the workflow description file. The parameter compliance verification is used to verify the parameter data types and value ranges in the workflow description file. The resource constraint verification is used to evaluate the matching degree between the resource requirements of the workflow description file and the available resources. The version management unit is used to record the version history of each workflow configuration, and also to obtain the workflow editing, saving or version switching functions set by the user.

[0014] The beneficial effects of adopting the above-mentioned further solutions are as follows: The step library management unit sets up a workflow step library, which can classify and manage commonly used data processing steps and support various operation functions of the workflow, making it convenient to flexibly manage workflows and set algorithm steps; The graphical designer unit, based on a visual interface and user permissions, allows users to generate workflow description files with multi-level dependencies in real time by dragging and dropping components, reducing the complexity of spatial science data processing and improving the efficiency of process orchestration; The template management unit stores standardized templates and supports the functions of importing, exporting, overriding, and creating workflow templates, facilitating the sharing and reuse of workflow configurations between different projects; The workflow verification unit performs compliance verification of workflow description files based on static analysis technology, including dependency loop detection, parameter compliance verification, and resource constraint verification, which can effectively improve the reliability of workflow configuration and execution success rate; The version management unit records the version history of workflow configurations and supports the functions of editing, saving, and switching versions, facilitating version control of workflows.

[0015] Furthermore, the step library management unit is also used for: Obtain the step parameters set by the user, including step name, step description, execution image configuration, program execution required parameter configuration, execution priority, concurrency mode configuration, execution trigger condition, program execution environment configuration, memory, CPU, and GPU; The beneficial effect of adopting the above-mentioned further solution is that by having the user define a complete set of parameters for each step, the consistency of the execution environment, the controllability of resources, and the predictability of behavior in each step of the workflow are ensured.

[0016] Furthermore, the graphical designer unit is also used for: Get the component pull, drag, connect, and associate functions set by the user, and combine the components into a complete workflow; The system provides a user-defined visualization configuration function for algorithm steps and execution environment, including step names, execution image versions, and resource parameters. After the user completes the workflow design, a workflow description file is generated and saved. The workflow description file is a description language supported by the workflow engine. The system provides a real-time preview function based on user settings, and visualizes the steps and execution logic of the business process based on workflow description files.

[0017] The beneficial effects of adopting the above-mentioned further solutions are as follows: The graphical designer unit supports users in performing component pull, drag, connect, and associate operations to combine components into a complete workflow, reducing the difficulty of workflow design and improving design efficiency; it supports users in visually configuring algorithm steps and execution environment, allowing users to intuitively set up the workflow, improving the accuracy and convenience of configuration; after the user completes the workflow design, the workflow description file is converted into a description language supported by the workflow engine and saved, ensuring that the workflow can be executed correctly; and it provides a real-time preview function so that users can understand the logical structure of the workflow in a timely manner, facilitating the adjustment and optimization of the workflow design.

[0018] Furthermore, the resource scheduling module includes: The scheduling strategy unit is used to obtain resource monitoring information in real time, select the optimal scheduling strategy from multiple scheduling strategies based on the resource monitoring information and workflow description file, and dynamically allocate computing resources based on the optimal scheduling. The multiple scheduling strategies include FIFO strategy, priority scheduling strategy and backfilling scheduling strategy. The resource management unit monitors the resource usage of each node, calculates resource allocation results based on the resource usage of each node, and calls the SLURM cluster management system through the Nextflow workflow engine to perform resource allocation; the resource usage includes CPU, memory, disk and network metrics. The parallel control unit is used to parse the task dependencies in the workflow description file and perform sequence management, dynamically adjust the parallelism based on the resource usage of each node, implement fault tolerance and fault recovery mechanisms, and coordinate the pause, resume, restart or terminate operations of multiple workflows in response to user control commands. The quota management unit is used to manage resource quotas based on resource monitoring information and resource requirements defined in the workflow description file, so that resource allocation conforms to preset quota limits.

[0019] The beneficial effects of adopting the above-mentioned further solutions are as follows: the scheduling strategy unit can dynamically allocate computing resources by selecting the optimal strategy from multiple scheduling strategies in real time based on resource monitoring information and workflow description files, achieving efficient utilization of computing resources and parallel execution of large-scale data processing tasks; the resource management unit can monitor the resource usage of each node and call the SLURM cluster management system through the Nextflow workflow engine to perform resource allocation, ensuring the rationality and accuracy of resource allocation; the parallel control unit is the core component of the resource scheduling module, undertaking the coordination and control responsibilities when multiple tasks are executed in parallel. In the space science data processing scenario, this unit ensures that a large number of computing tasks can run efficiently and reliably in parallel through a refined task scheduling and resource allocation mechanism. The quota management unit can manage resource quotas, ensuring that resource allocation conforms to preset quota limits, guaranteeing the fairness and controllability of resource allocation.

[0020] Furthermore, the scheduling strategy unit is also used for: Obtain the resource scheduling mode set by the user, which includes node sharing mode and node non-sharing mode; Obtain the scheduling parameters of the user-set concurrency strategy, and determine the execution strategy for job concurrency scheduling. The scheduling parameters include file mode, folder mode, and waterfall mode.

[0021] The beneficial effects of adopting the above-mentioned further solutions are: the scheduling strategy unit can obtain the resource scheduling mode of shared or non-shared nodes set by the user, and can clarify the computing node usage strategy; it can obtain the concurrent scheduling parameters of the user-set file mode, folder mode, waterfall mode and other concurrency strategy to determine the job concurrent scheduling execution strategy, and can combine with other job configuration parameters to complete the flexible arrangement of workflow, realize resource allocation and concurrent job scheduling, efficiently utilize computing resources, and meet the needs of high-performance computing.

[0022] Furthermore, the workflow execution monitoring module includes: The log collection unit is used to collect workflow execution logs, performance indicators and execution status in real time based on full-link tracking of workflow logs. The execution logs include start time, end time and execution status. The performance monitoring unit is used to monitor the resource usage of the system's computing nodes and provide early warnings. The resource usage includes CPU, memory, disk, and network metrics. The visualization unit provides a visual monitoring interface, which displays workflow execution information and resource usage. The alarm management unit is used to issue alarms for workflows in abnormal states.

[0023] The beneficial effects of adopting the above-mentioned further solutions are as follows: the log collection unit collects execution logs, performance indicators, and execution status in real time based on the full-link tracking of workflow logs, which can realize transparent management of the workflow execution process and facilitate problem localization; the performance monitoring unit monitors the resource usage of system computing nodes and provides early warning prompts, which can promptly detect abnormal resource usage; the visualization display unit provides a visual monitoring interface to display workflow execution information and resource usage, which can intuitively present the workflow execution progress; and the alarm management unit alarms for workflows in abnormal states, which can facilitate timely handling of anomalies and ensure the reliability and stability of the system.

[0024] Secondly, this application provides a method for orchestrating and scheduling resources for space science data processing production processes, employing the following technical solution: A method for orchestrating and scheduling resources in space science data processing production processes includes: Obtain space science data processing algorithms, encapsulate the space science data processing algorithms based on containerization technology, and generate standard container images and container configuration files; In response to a user's action to trigger the graphical drag-and-drop function of components in the visual workflow design interface, a workflow description file is generated, which represents the data processing process composed of multiple container images or programs. Real-time acquisition of resource monitoring information; based on the resource monitoring information and the workflow description file, allocation of computing resources to the workflow and scheduling of multiple data processing tasks. Collect execution information during the workflow execution process, and display the workflow execution information and resource usage through a visual display interface. The execution information includes execution logs, performance indicators, and execution status.

[0025] Thirdly, this application provides an electronic device that adopts the following technical solution: An electronic device includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in the second aspect, a method for orchestrating and scheduling space science data processing workflows and resources.

[0026] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium storing a computer program that can be loaded by a processor and executed by a method for orchestrating and scheduling space science data processing workflows as described in the second aspect.

[0027] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application.

[0028] In summary, compared with the prior art, the beneficial effects of the present invention are as follows: (1) Visual workflow design reduces the complexity of space science data processing and improves the efficiency of workflow orchestration; (2) By adopting containerization technology, the standardization and isolation of the data processing environment are realized, which improves the portability and maintainability of the system; (3) Through intelligent resource scheduling, the dynamic allocation and efficient utilization of computing resources are realized, supporting the parallel execution of large-scale data processing tasks; (4) Provides end-to-end monitoring capabilities, enabling transparent management of the workflow execution process, which facilitates problem localization and performance optimization; This invention is particularly suitable for providing system support for the processing and analysis of large-scale space science data such as space station scientific experiments, as well as the ever-changing operational needs of the space station. It has good application prospects and promotional value. Attached Figure Description

[0029] Figure 1 A schematic diagram of a space science data processing production process orchestration and resource scheduling system provided in one embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a container management module provided in one embodiment of the present invention; Figure 3 A flowchart illustrating the container image upload logic of the container image repository building unit provided in one embodiment of the present invention; Figure 4 A flowchart illustrating the logic of a container image repository building unit performing container image version upload, provided in one embodiment of the present invention; Figure 5 A logical diagram illustrating the execution of workflow design and orchestration steps by a step library management unit according to an embodiment of the present invention; Figure 6 A logic flowchart of the graphical designer unit performing the setup steps according to an embodiment of the present invention. Figure 7 A logical flowchart of the workflow execution process of the resource scheduling module provided in one embodiment of the present invention; Figure 8A logical flowchart illustrating the monitoring steps performed by a workflow execution monitoring module according to an embodiment of the present invention; Figure 9 A logical flowchart of a visualization unit for displaying computing node resource information is provided in one embodiment of the present invention. Figure 10 A visualization unit provided in one embodiment of the present invention displays a logical flowchart of the workflow execution progress; Figure 11 A flowchart illustrating a method for orchestrating and scheduling resources for space science data processing production processes, provided as an embodiment of the present invention; Figure 12 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0032] like Figure 1 As shown in the figure, this application provides a space science data processing production process orchestration and resource scheduling system, which mainly includes: The container management module is used to acquire space science data processing algorithms, encapsulate the space science data processing algorithms based on containerization technology, and generate standard container images and container configuration files. The workflow design and orchestration module is used to generate a workflow description file in response to the user's trigger action on the graphical drag-and-drop function of components in the visual workflow design interface. The workflow description file represents the data processing process composed of multiple container images or programs. The resource scheduling module is used to acquire resource monitoring information in real time, and based on the resource monitoring information and the workflow description file, allocate computing resources to the workflow and schedule multiple data processing tasks. The workflow execution monitoring module is used to collect execution information during the workflow execution process and display the execution information and resource usage of the workflow based on a visual display interface. The execution information includes execution logs, performance indicators and execution status.

[0033] The container management module encapsulates space science data processing algorithms in a containerized manner, achieving standardization and isolation of the data processing environment and improving the system's portability and maintainability. The workflow design and orchestration module generates workflow description files through a visual design interface, reducing the complexity of space science data processing and improving the efficiency of workflow orchestration. The resource scheduling module allocates computing resources and schedules multiple tasks based on real-time resource monitoring information, achieving dynamic allocation and efficient utilization of computing resources and supporting the parallel execution of large-scale data processing tasks. The workflow execution monitoring module collects and displays workflow execution information and resource usage, achieving transparent management of the workflow execution process and facilitating problem localization and performance optimization.

[0034] The technical solution of this invention will be described in detail below, using a practical deployment case of the business support platform system designed and developed based on this system. This embodiment takes the production of data products for the three modules of the space station as a typical application scenario.

[0035] Optional, such as Figure 2 As shown, the container management module includes a container image repository building unit, a containerization packaging tool unit, a security scanning unit, an image optimization unit, and a container runtime environment adaptation unit.

[0036] Among them, the container image repository building unit is used to build container image repositories; A containerization encapsulation tool unit is used to identify the language type of the space science data processing algorithm, generate a standard container image based on the language type of the space science data processing algorithm and a pre-set template library, generate a container configuration file corresponding to the standard container image based on the dependency relationship, and store the standard container image and container configuration file in the container image repository. The security scanning unit is used to obtain uploaded container images from the container image repository, perform security vulnerability assessments on the operating system layer, middleware layer, and application layer of the container images based on a layered scanning strategy and a vulnerability database, and obtain the vulnerability level of the container images; based on the vulnerability level of the container images and the corresponding handling strategy, determine the remediation recommendations for the container images. The image optimization unit is used to analyze the hierarchical structure of the container image based on the repair suggestions and hierarchical analysis algorithm of the container image, and obtain the hierarchical analysis results; based on the multi-stage build technology and the hierarchical analysis results of the container image, optimize the container image, generate the optimized container image, and feed it back to the image repository. The container runtime environment adaptation unit is used to select the container image runtime environment based on the deployment environment characteristics of the container image.

[0037] In this embodiment, the business support platform provides complete container image repository management functionality, supporting both Docker and Singularity container runtime environments to ensure compatibility in heterogeneous computing environments. Through the containerization packaging tools provided by the platform, data processing programs can be packaged into standard container images, and container configuration files can be automatically generated. The container image repository management functionality includes application container image management and application container image version management.

[0038] In this embodiment of the application, the container image repository supports the uploading, downloading, version management and metadata recording of container images, and provides image signing and integrity verification; The container image repository building unit is used to respond to user actions triggering the container image upload function, obtain the container image sent by the user, along with the container image's name, type, provider, image description, version number, version notes, and program, and store the container image in the container image repository. For example, users can upload a container image of a space science data processing algorithm through the system's provided interface.

[0039] The container image repository building unit is used to respond to user-triggered actions for updating container image versions. It obtains the version number, version description, and updated program of the container image to be updated, and updates the container image in the workflow template based on the version number, version description, and updated program of the container image to be updated.

[0040] For example, Figure 3 The flowchart illustrates the logic for uploading container images to the container image repository building unit, as follows: Figure 3 The container image upload process is as follows: After logging in, users on the business support platform are provided with an interactive application container image upload interface based on their personal permissions. System administrators and operators can upload container images to any category, while mission administrators and operators of the three modules of the space station can upload container images to a specified category. Users need to define the upload image driver type as data-driven / order-driven or both when uploading. In addition to the image itself, the upload also needs to include descriptive information about the image attributes, such as the image name, image provider, and image execution instructions.

[0041] The image repository supports version updates for executable images and records and queries historical versions of executable images. Users can upload different version image packages through the interactive interface provided by the system. After the image version is uploaded, the image version in the workflow template can be updated via the "Update Workflow Image Version" operation, or different versions of the executable can be selected during the workflow configuration process. The logical flowchart of the container image repository build unit for uploading container image versions is as follows: Figure 4 As shown.

[0042] The containerization packaging tool unit provides containerization tools to package data processing programs into standard container images, supporting multiple programming languages ​​and runtime environments, and automatically generating container configuration files. This unit offers a standardized program packaging solution, supporting multiple programming languages ​​such as Java, Python, and C++, and includes built-in best practice templates for different languages, such as JVM parameter optimization templates for Java applications and dependency management templates for Python environments. Through intelligent analysis of the project structure, it can automatically detect dependencies and generate a complete container configuration including the base image, runtime environment, and dependent libraries. Automated script tools can also be used to achieve containerization packaging, improving packaging efficiency.

[0043] The security scanning unit performs security scanning and vulnerability detection on container images, automating security assessments of uploaded images. The security scanning mechanism establishes a comprehensive image security assurance system, employing a layered scanning strategy for in-depth testing of uploaded container images. The system integrates a CVE (Common Vulnerability Exposure) database, covering three layers: operating system, middleware, and application. It conducts comprehensive security checks on the basic software packages, dependency libraries, and configuration files at each layer. The scanning engine combines static and dynamic analysis, detecting not only known vulnerabilities but also potential security risks.

[0044] The security scanning unit is also used to classify vulnerabilities into four levels—critical, high-risk, medium-risk, and low-risk—based on a security level assessment system, and to formulate corresponding handling strategies for each level. The scan results generate detailed security reports, including vulnerability descriptions, impact scope, and remediation recommendations. For discovered security issues, the platform provides automatic remediation suggestions, such as recommending secure base image versions and providing patch installation scripts. Through deep integration with image repositories, a complete closed-loop security management system—"scan-report-remediation-verification"—is achieved.

[0045] The image optimization unit provides container image optimization capabilities, automatically analyzing image layers, removing redundant dependencies, and reducing image size. This unit improves container image performance and efficiency through multi-dimensional analysis and technical means. It employs deduplication techniques to eliminate duplicate data and reduce image size; and utilizes a minimum base image strategy to remove unnecessary system components. Incremental updates can also be used to optimize images, reducing optimization time.

[0046] The container runtime environment adaptation unit selects the container image runtime environment based on the deployment environment characteristics of the container image. The business support platform supports both Docker and Singularity, two mainstream container runtime environments, ensuring compatibility across different computing scenarios. For high-performance computing scenarios, the platform prioritizes the Singularity runtime, whose root-privilege-free feature is more suitable for cluster environments, and its deep integration with the host file system facilitates data interaction. For development and testing environments, the Docker runtime is used, leveraging its rich ecosystem resources and convenient debugging tools. A more suitable container runtime environment can also be selected based on different hardware environments.

[0047] These units work together to achieve containerized encapsulation and management of space science data processing algorithms, ensuring the security, compatibility, and efficiency of container images.

[0048] Optionally, the workflow design and orchestration module includes a step library management unit, a graphical designer unit, a template management unit, a workflow verification unit, and a version management unit.

[0049] The step library management unit is used to set up a workflow step library, obtain the functions of creating, editing, deleting, copying, importing, exporting or managing historical versions of workflows set by users, set up and manage the processing steps corresponding to common space science data business processing flows; wherein, the workflow step library includes multiple data processing workflows; The graphical designer unit is used to design the interface and user permissions based on the visual workflow, obtain the drag-and-drop component function set by the user, and generate a workflow description file of algorithm steps in real time based on JSON. The template management unit is used to store standardized workflow templates; it is also used to obtain user-defined workflow template import or export functions; and it is also used to obtain user-defined workflow template overriding or workflow creation functions. The workflow verification unit is used to perform compliance verification on the workflow description file based on static analysis technology. The compliance verification includes dependency loop detection, parameter compliance verification, and resource constraint verification. The dependency loop detection is used to identify circular dependencies in the workflow description file. The parameter compliance verification is used to verify the parameter data types and value ranges in the workflow description file. The resource constraint verification is used to evaluate the matching degree between the resource requirements of the workflow description file and the available resources. The version management unit is used to record the version history of each workflow configuration, and also to obtain the workflow editing, saving or version switching functions set by the user.

[0050] In this embodiment, a web-based visual workflow design interface is provided, which uses a graphical drag-and-drop method to construct space science data processing workflows. Through the workflow design module of the business support platform, users can intuitively define data processing workflows containing multi-level dependencies, and the system generates JSON-based workflow description files in real time.

[0051] The step library management unit is also used to obtain user-set step parameters, including step name, step description, execution image configuration, program execution parameter configuration, execution priority, concurrency mode configuration, execution trigger condition, program execution environment configuration, memory, CPU, and GPU.

[0052] Figure 5 This is a logical diagram illustrating how the step library management unit executes workflow design and orchestration steps, such as... Figure 5 As shown, when a user needs to create a new workflow to complete the production of data products for the three-cabin task, the step library management unit executes the following steps: After logging in, users of the business support platform are provided with an interactive workflow step-adding interface based on their personal permissions. After adding a step, users can create a workflow. Adding a step involves setting multiple parameters, including step name, step description, execution image configuration, program execution required parameter configuration, execution priority, concurrency mode configuration, execution trigger condition, program execution environment configuration, memory, CPU, and GPU. The execution image configuration is a required field for adding the execution image name and image version; the program execution required parameter configuration is also a required field, including input, output, and other parameters; the concurrency mode configuration is optional, including waterfall, single-point, fixed, random, and multi-point modes.

[0053] The business support platform provides a graphical designer unit that allows users to define data processing flows with multi-level dependencies by dragging and dropping components. The designer generates workflow description files based on JSON in real time.

[0054] The graphical designer unit is also used for: Get the component pull, drag, connect, and associate functions set by the user, and combine the components into a complete workflow; The system provides a user-defined visualization configuration function for algorithm steps and execution environment, including step names, execution image versions, and resource parameters. After the user completes the workflow design, a workflow description file is generated and saved. The workflow description file is a description language supported by the workflow engine. The system provides a real-time preview function based on user settings, and visualizes the steps and execution logic of the business process based on workflow description files.

[0055] The logic flowchart of the setup steps performed by the graphical designer unit is as follows: Figure 6 As shown, after logging in, users of the business support platform are provided with an interactive workflow operation interface based on their personal permissions. After adding steps, users can create workflows. In the newly created workflow operation interface, users can drag and drop steps into the workflow operation interface by pulling, dragging, connecting, and associating them to form a complete workflow process. At the same time, users can configure step information and execution environment. After creation, the business support platform converts the workflow into a description language supported by the workflow engine and saves it.

[0056] The template management unit facilitates the sharing and reuse of workflow configurations across different projects. It also allows for the creation of a workflow template community, enabling users to share and exchange templates.

[0057] Users can export existing workflows from the interactive workflow interface provided by the system. The exported workflow configuration information is stored locally as a file. When new business needs require reusing an existing workflow template, users can update or create workflows by importing files through the interactive workflow interface. Imported files must comply with the relevant requirements of the business support platform. After importing a file, users can choose to overwrite the current workflow or create a new one. After importing, the information of the newly imported workflow is stored in the database.

[0058] The workflow verification unit provides workflow verification functionality, using static analysis techniques to check the rationality and completeness of workflow configurations. This functionality employs static analysis to comprehensively verify the configuration before workflow execution. It analyzes dependencies between steps using a dependency cycle detection algorithm and identifies circular dependencies using a directed acyclic graph (DAG) model, ensuring the rationality of the process logic. Parameter compliance verification covers data type verification, value range checks, and business rule matching, achieving automated verification through a rule engine. Resource constraint verification comprehensively evaluates the matching degree between CPU, memory, and storage resource requirements and available platform resources, preventing runtime resource conflicts. These verification measures generate detailed verification reports and provide intelligent remediation suggestions, effectively improving the reliability of workflow configurations and execution success rates.

[0059] The version management unit supports workflow version management, automatically generating a new version with each modification, retaining historical version records, and supporting version rollback and difference comparison. The business support platform provides workflow version management functionality. After editing the current workflow version, users on the business support platform can click the save button. The system will prompt whether to save the workflow as a new version; clicking save will save the new version of the workflow. After saving the current workflow version, users can switch between different versions of the workflow in the version drop-down list and perform operations such as editing and executing the workflow.

[0060] These units work together to enable the visual design and orchestration of space science data processing workflows, improving the efficiency and accuracy of workflow design.

[0061] Optionally, the resource scheduling module includes a scheduling strategy unit, a resource management unit, a parallel control unit, and a quota management unit.

[0062] The scheduling strategy unit is used to obtain resource monitoring information in real time, select the optimal scheduling strategy from multiple scheduling strategies based on the resource monitoring information and workflow description file, and dynamically allocate computing resources based on the optimal scheduling. The multiple scheduling strategies include FIFO strategy, priority scheduling strategy and backfilling scheduling strategy. The resource management unit monitors the resource usage of each node, calculates resource allocation results based on the resource usage of each node, and calls the SLURM cluster management system through the Nextflow workflow engine to perform resource allocation; the resource usage includes CPU, memory, disk and network metrics. The parallel control unit is used to parse the task dependencies in the workflow description file and perform sequence management, dynamically adjust the parallelism based on the resource usage of each node, implement fault tolerance and fault recovery mechanisms, and coordinate the pause, resume, restart or terminate operations of multiple workflows in response to user control commands. The quota management unit is used to manage resource quotas based on resource monitoring information and resource requirements defined in the workflow description file, so that resource allocation conforms to preset quota limits.

[0063] In this embodiment, the business support platform calls the SLURM cluster management system through the Nextflow workflow engine to realize intelligent scheduling of computing resources. It supports multiple scheduling algorithms such as FIFO, priority scheduling, and backfilling scheduling, and can manage no less than 200 workflow instances at the same time.

[0064] The workflow is executed through the workflow execution engine, which uses the configuration information of units such as the scheduling strategy unit, parallel control unit, and resource management unit to execute the workflow.

[0065] The workflow control business support platform allows users to resume, pause / restart, and terminate workflows through the workflow execution management interface. The workflow execution engine executes the workflow based on the user's commands in the visual interface. When a workflow is paused / terminated, the current step continues running, stopping and releasing resources at the next step. When a workflow is resumed, the execution engine sends a Resume statement to restore the original execution. When a workflow is restarted, the workflow execution engine re-executes the workflow based on the configuration information in the system's scheduling strategy unit, parallel control unit, and resource management unit.

[0066] Figure 7 This is a logical flowchart of the workflow execution process of the resource scheduling module, such as... Figure 7 As shown, when the workflow process is executed to complete the production business of the three-cabin task data products, the resource scheduling module generates a workflow execution engine scheduling script according to the workflow definition. The workflow execution engine automatically generates a job script according to the resources (CPU, GPU, memory, nodes, etc.) allocated in the scheduling script. The workflow engine submits the job and waits for the job scheduling system to execute the job. The workflow engine monitors in real time and returns job running details.

[0067] The scheduling strategy unit is also used for: Obtain the resource scheduling mode set by the user, which includes node sharing mode and node non-sharing mode; Obtain the scheduling parameters of the user-set concurrency strategy, and determine the execution strategy for job concurrency scheduling. The scheduling parameters include file mode, folder mode, and waterfall mode.

[0068] Edit workflow processes based on business production needs, clarify computing node usage strategies by setting resource scheduling modes including node sharing mode and node non-sharing mode; ensure resource allocation order by setting workflow job scheduling strategies including first-in-first-out, backfill scheduling, queue priority, etc.; determine job concurrent scheduling execution strategies by setting concurrent scheduling parameters including file mode, folder mode, waterfall mode, and combine with other job configuration parameters to complete flexible workflow orchestration.

[0069] Optionally, the workflow execution monitoring module includes: The log collection unit is used to collect workflow execution logs, performance indicators and execution status in real time based on full-link tracking of workflow logs. The execution logs include start time, end time and execution status. The performance monitoring unit is used to monitor the resource usage of the system's computing nodes and provide early warnings. The resource usage includes CPU, memory, disk, and network metrics. The visualization unit provides a visual monitoring interface, which displays workflow execution information and resource usage. The alarm management unit is used to issue alarms for workflows in abnormal states.

[0070] In the embodiments of this application, Figure 8 This is a logical flowchart of the workflow execution monitoring module's monitoring steps. For example... Figure 8 As shown, when executing the three-cabin data product production workflow task, the workflow execution monitoring module is used to execute the following process.

[0071] The log collection unit is based on full-link tracking of workflow logs and collects workflow execution logs in real time. A distributed log collection system can also be used to improve collection efficiency.

[0072] A further step in the end-to-end log tracking information display process involves standardizing and displaying various logs generated during workflow execution on the business support platform. Logs are associated through workflow UUID, workflow taskID, workflow processUUID, etc. The logs of workflow execution, including backend logs, engine logs, job logs, and execution program message logs, are linked together through association fields to connect log data of each link in the workflow business chain. Through the link relationship between logs, all related logs on the link can be intuitively located.

[0073] The workflow log end-to-end tracing information retrieval process involves using a message middleware to initiate programs and transmit progress messages during workflow execution. To prevent message loss and troubleshooting, the system needs to establish a monitoring message query module on the front end. This module can quickly locate all monitoring messages for a workflow based on criteria such as the workflow UUID. The workflow log end-to-end tracing information retrieval supports querying and locating logs across the entire workflow business chain. By viewing detailed information about the call chain of the target link (abnormal log) and the log file details, the system can analyze the cause of the anomaly.

[0074] The performance monitoring unit is a real-time workflow monitoring mechanism that monitors the resource usage of system computing nodes, including indicators such as CPU, memory, disk, and network, and provides early warning prompts.

[0075] The business support platform provides workflow computing node monitoring functionality during workflow execution. Within their authorized scope, users can view the server running status, cluster node running status, and resource consumption information during workflow execution in the workflow resource information interface.

[0076] Figure 9 The visualization unit displays a logical flowchart of computing node resource information, such as... Figure 9 As shown, users can open the compute node monitoring interface, enter the identity identifier of the monitored workflow (i.e., the workflow number when the workflow was created), and obtain resource metrics information such as server operating status, cluster node operating status, memory usage, CPU usage, and disk load. Alarms are provided when resource metrics do not meet preset ranges.

[0077] The visualization unit provides a visual monitoring interface to display information such as workflow execution progress, workflow running status, duration, and status statistics for each task. An alarm management unit alerts to abnormal status information, allowing users to easily understand the workflow's execution status in real time. 3D visualization technology can also be used to enhance the display effect.

[0078] The business support platform provides workflow execution progress monitoring functionality during workflow operation. Figure 10 A logical flowchart is used to display the workflow execution progress in the visualization unit.

[0079] Users can view the execution status and progress of workflows in the workflow execution progress monitoring interface. The business support platform supports displaying the progress, status, and results of data production business processes by module (TGTH, TGMT, TGWT, TGXT), workflow running status, duration, and status statistics of each job. Alarm information is displayed to users through highlighting or information prompts. It supports a one-stop display of the status and data processing of each module and TGXT. The workflow monitoring refresh interval is 0.5 milliseconds, and it can monitor no fewer than 200 workflows simultaneously, with no fewer than 5,000 jobs monitored simultaneously in each workflow.

[0080] In addition, the business support platform provides a visual display of workflow steps, enabling visual monitoring and interactive display of the execution status and output files of each step in the workflow. It can also display the data processing flow according to different product granularities (APID / DPID). After logging in, users can select the display granularity of workflow steps based on their personal permissions, and the system monitors and displays the data according to the user-configured display rules.

[0081] Figure 11 A flowchart illustrating workflow orchestration and resource scheduling that supports space science data processing and production is shown.

[0082] like Figure 11 As shown, a method for orchestrating and scheduling resources in space science data processing production processes mainly includes: S1, Obtain the space science data processing algorithm, encapsulate the space science data processing algorithm based on containerization technology, and generate a standard container image and container configuration file; S2, in response to the user's trigger action on the graphical drag-and-drop function of components in the visual workflow design interface, a workflow description file is generated, which represents the data processing process composed of multiple container images or programs; S3, acquire resource monitoring information in real time, and allocate computing resources and schedule multiple data processing tasks for the workflow based on the resource monitoring information and the workflow description file; S4. Collect execution information during the workflow execution process, and display the execution information and resource usage of the workflow based on a visual display interface. The execution information includes execution logs, performance indicators, and execution status.

[0083] The resource scheduling method corresponds one-to-one with the functional modules in the resource scheduling system of the foregoing embodiments, and its problem-solving principle is similar to that of the system.

[0084] Figure 12 This is a structural block diagram of an electronic device 300 according to an embodiment of this application.

[0085] like Figure 12 As shown, the electronic device 300 includes a processor 301 and a memory 302, and may further include one or more of an information input / output (I / O) interface 303, a communication component 304, and a communication bus 305.

[0086] The processor 301 controls the overall operation of the electronic device 300 to complete all or part of the steps in the aforementioned method for orchestrating and scheduling space science data processing production processes and resources. The memory 302 stores various types of data to support the operation of the electronic device 300. This data may include, for example, instructions for any application or method operating on the electronic device 300, as well as application-related data. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as one or more of Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0087] I / O interface 303 provides an interface between processor 301 and other interface modules, such as keyboards, mice, and buttons. These buttons can be virtual or physical. Communication component 304 is used to test wired or wireless communication between electronic device 300 and other devices. Wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof. Therefore, the corresponding communication component 304 may include a Wi-Fi component, a Bluetooth component, and an NFC component.

[0088] The communication bus 305 may include a path for transmitting information between the aforementioned components. The communication bus 305 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 305 may be divided into an address bus, a data bus, a control bus, etc.

[0089] The electronic device 300 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute a space science data processing production process orchestration and resource scheduling method given in the above embodiments.

[0090] The following describes the computer-readable storage medium provided in the embodiments of this application. The computer-readable storage medium described below can be referred to in correspondence with the space science data processing production process orchestration and resource scheduling method described above.

[0091] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for arranging and scheduling a space science data processing production process.

[0092] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0094] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing application concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions claimed in this application.

Claims

1. A system for orchestrating and scheduling production processes and resources for space science data processing, characterized in that, include: The container management module is used to acquire space science data processing algorithms, encapsulate the space science data processing algorithms based on containerization technology, and generate standard container images and container configuration files. The workflow design and orchestration module is used to generate a workflow description file in response to the user's trigger action on the graphical drag-and-drop function of components in the visual workflow design interface. The workflow description file represents the data processing process composed of multiple container images or programs. The resource scheduling module is used to acquire resource monitoring information in real time, and based on the resource monitoring information and the workflow description file, allocate computing resources to the workflow and schedule multiple data processing tasks. The workflow execution monitoring module is used to collect execution information during the workflow execution process and display the execution information and resource usage of the workflow based on a visual display interface. The execution information includes execution logs, performance indicators and execution status. The container management module includes: A container image repository building unit, used to build container image repositories; A containerization encapsulation tool unit is used to identify the language type of the space science data processing algorithm, generate a standard container image based on the language type of the space science data processing algorithm and a pre-set template library, generate a container configuration file corresponding to the standard container image based on the dependency relationship, and store the standard container image and container configuration file in the container image repository. The security scanning unit is used to obtain uploaded container images from the container image repository, perform security vulnerability assessments on the operating system layer, middleware layer, and application layer of the container images based on a layered scanning strategy and a vulnerability database, and obtain the vulnerability level of the container images; based on the vulnerability level of the container images and the corresponding handling strategy, determine the remediation recommendations for the container images. The image optimization unit is used to analyze the hierarchical structure of the container image based on the repair suggestions and hierarchical analysis algorithm of the container image, and obtain the hierarchical analysis results; based on the multi-stage build technology and the hierarchical analysis results of the container image, optimize the container image, generate the optimized container image and store it in the image repository; The container runtime environment adaptation unit is used to select the container image runtime environment based on the deployment environment characteristics of the container image. The workflow design and orchestration module includes: The workflow step library management unit is used to set up a workflow step library, obtain the functions of creating, editing, deleting, copying, importing, exporting or managing historical versions of workflows set by users, set up and manage the processing steps corresponding to common space science data business processing flows; wherein, the workflow step library includes multiple data processing workflows; The graphical designer unit is used to design the interface and user permissions based on the visual workflow, obtain the drag-and-drop component function set by the user, and generate a workflow description file of algorithm steps in real time based on JSON. The template management unit is used to store standardized workflow templates; it is also used to obtain user-defined workflow template import or export functions; and it is also used to obtain user-defined workflow template overriding or workflow creation functions. The workflow verification unit is used to perform compliance verification on the workflow description file based on static analysis technology. The compliance verification includes dependency loop detection, parameter compliance verification, and resource constraint verification. The dependency loop detection is used to identify circular dependencies in the workflow description file. The parameter compliance verification is used to verify the parameter data types and value ranges in the workflow description file. The resource constraint verification is used to evaluate the matching degree between the resource requirements of the workflow description file and the available resources. The version management unit is used to record the version history of each workflow configuration, and also to obtain the workflow editing, saving or version switching functions set by the user. The resource scheduling module includes: The scheduling strategy unit is used to obtain resource monitoring information in real time, select the optimal scheduling strategy from multiple scheduling strategies based on the resource monitoring information and workflow description file, and dynamically allocate computing resources based on the optimal scheduling. The multiple scheduling strategies include FIFO strategy, priority scheduling strategy and backfilling scheduling strategy. The resource management unit monitors the resource usage of each node, calculates resource allocation results based on the resource usage of each node, and calls the SLURM cluster management system through the Nextflow workflow engine to perform resource allocation; the resource usage includes CPU, memory, disk and network metrics. The parallel control unit is used to parse the task dependencies in the workflow description file and perform sequence management, dynamically adjust the parallelism based on the resource usage of each node, implement fault tolerance and fault recovery mechanisms, and coordinate the pause, resume, restart or terminate operations of multiple workflows in response to user control commands. The quota management unit is used to manage resource quotas based on resource monitoring information and resource requirements defined in the workflow description file, so that resource allocation conforms to preset quota limits.

2. The space science data processing production process orchestration and resource scheduling system according to claim 1, characterized in that, The container image repository building unit is also used for: In response to a user's trigger action for the container image upload function, the system obtains the container image sent by the user, as well as the name, type, provider, image description, version number, version description, and program of the container image, and stores the container image in the container image repository. In response to a user's trigger action for updating the container image version, obtain the version number, version description, and updated program of the container image to be updated, and update the container image in the workflow template based on the version number, version description, and updated program of the container image to be updated.

3. The space science data processing production process orchestration and resource scheduling system according to claim 1, characterized in that, The step library management unit is also used for: Obtain the step parameters set by the user, including step name, step description, execution image configuration, parameter configuration required for program execution, execution priority, concurrency mode configuration, execution trigger condition, program execution environment configuration, memory, CPU, and GPU.

4. The space science data processing production process orchestration and resource scheduling system according to claim 1, characterized in that, The graphical designer unit is also used for: Get the component pull, drag, connect, and associate functions set by the user, and combine the components into a complete workflow; The system provides a user-defined visualization configuration function for algorithm steps and execution environment, including step names, execution image versions, and resource parameters. After the user completes the workflow design, a workflow description file is generated and saved. The workflow description file is a description language supported by the workflow engine. The system provides a real-time preview function based on user settings, and visualizes the steps and execution logic of the business process based on workflow description files.

5. The space science data processing production process orchestration and resource scheduling system according to claim 1, characterized in that, The scheduling strategy unit is further configured to: Obtain the resource scheduling mode set by the user, which includes node sharing mode and node non-sharing mode; Obtain the scheduling parameters of the user-set concurrency strategy, and determine the execution strategy for job concurrency scheduling. The scheduling parameters include file mode, folder mode, and waterfall mode.

6. The space science data processing production process orchestration and resource scheduling system according to claim 1, characterized in that, The workflow execution monitoring module includes: The log collection unit is used to collect workflow execution logs, performance indicators and execution status in real time based on full-link tracking of workflow logs. The execution logs include start time, end time and execution status. The performance monitoring unit is used to monitor the resource usage of the system's computing nodes and provide early warnings. The resource usage includes CPU, memory, disk, and network metrics. The visualization unit provides a visual monitoring interface, which displays workflow execution information and resource usage. The alarm management unit is used to issue alarms for workflows in abnormal states.

7. A method for orchestrating and scheduling resources for space science data processing production processes, applied to the system described in any one of claims 1 to 6, characterized in that, include: Obtain space science data processing algorithms, encapsulate the space science data processing algorithms based on containerization technology, and generate standard container images and container configuration files; In response to a user's action to trigger the graphical drag-and-drop function of components in the visual workflow design interface, a workflow description file is generated, which represents the data processing process composed of multiple container images or programs. Real-time acquisition of resource monitoring information; based on the resource monitoring information and the workflow description file, allocation of computing resources to the workflow and scheduling of multiple data processing tasks. Collect execution information during the workflow execution process, and display the workflow execution information and resource usage through a visual display interface. The execution information includes execution logs, performance indicators, and execution status.

8. An electronic device, characterized in that, Includes a processor, which is coupled to a memory; The processor is configured to execute a computer program stored in the memory, so that the electronic device performs the method as described in claim 7.

9. A computer-readable storage medium, characterized in that, It includes a computer program or instructions that, when run on a computer, cause the computer to perform the method as described in claim 7.

Images