A large-scale satellite data processing and dynamic scheduling method and device

Abstract

The application provides a large-scale satellite data processing and dynamic scheduling method, comprising the following steps: writing a task table defined by an extensible markup language for different satellites or loads; parsing the task table corresponding to different satellites or loads to extract metadata; performing multi-dimensional screening based on a regular algorithm according to the metadata; generating a job sheet from the data obtained through multi-dimensional screening; submitting the job sheet to a workflow management program for process control; submitting a process node to a distributed computing task scheduling program for resource scheduling; automatically inserting / cloning a sub-process according to the metadata output by the distributed computing task scheduling program; synchronizing the task state in real time by using a Redis cache, and writing back the workflow management program for dynamic retention feedback. The application also provides a device for large-scale satellite data processing and dynamic scheduling. The method has the advantages of short deployment period, high data throughput, online evolution, self-recovery and high concurrency processing for newly added satellite loads.

Description

TECHNICAL FIELD

[0001] The present application relates to large-scale data processing and task scheduling technology, and in particular to an automatic processing and process control method for multi-satellite, multi-load remote sensing observation data. BACKGROUND

[0002] With the significant increase in the number of remote sensing satellites and the frequency of data acquisition, multi-satellite, multi-load data acquisition forms a large amount of high-frequency heterogeneous remote sensing data. The traditional "script + single-layer scheduling" system has bottlenecks in process construction, resource scaling and dynamic evolution. The specific problems that hinder its industrial application include: ① high maintenance cost, which requires re-writing scripts for each newly added satellite / load; ② scheduling coupling, which mixes dependency, resource allocation and job delivery, making it difficult to expand; ③ lack of adaptability, which makes it impossible to insert or replace sub-tasks online if the algorithm is segmented or upgraded during operation; ④ any exception requires manual intervention, limiting overall throughput and reliability.

[0003] Therefore, it is necessary to propose a new method more suitable for large-scale remote sensing data processing to solve the above industrial application problems. SUMMARY

[0004] In view of the problem that the traditional script maintenance method cannot adapt to high-concurrency, high-frequency heterogeneous data processing, the primary purpose of the present application is to provide a remote sensing data automatic processing and dynamic scheduling method with short deployment cycle, high throughput and adaptability, to realize automatic control of the whole process from data reception, product generation to archiving, thereby significantly reducing maintenance cost, improving processing efficiency and enhancing industrial applicability.

[0005] Another purpose of the present application is to provide a device and electronic equipment capable of executing the above remote sensing data automatic processing and dynamic scheduling method.

[0006] The above purposes of the present application are achieved by the following technical solutions:

[0007] In a first aspect, the present application provides a large-scale satellite data processing and dynamic scheduling method, comprising the following steps:

[0008] 1) Write an extensible markup language (XML) defined task table (Task Table) for different satellites or loads to realize unified task description;

[0009] 2) After obtaining remote sensing observation data from different satellites or loads, parse the corresponding XML format Task Table and extract metadata;

[0010] 3) Based on the metadata, adapt the multi-source files in the product list, directory or database based on regular algorithms to realize multi-dimensional filtering;

[0011] 4) Assemble the data obtained by multi-dimensional screening into command line, parameters and I / O list according to ESA JobOrder standard to generate a job order;

[0012] 5) Respectively adopt different program to process flow control and resource scheduling; including: submit the job order to a workflow management program (such as ecFlow) for flow control; submit the flow node to a distributed computing task scheduling program (such as HTCondor) for resource scheduling;

[0013] 6) Automatically insert / clone sub-process according to the metadata output by the distributed computing task scheduling program;

[0014] 7) Use Redis cache to synchronize task status in real time, and write back to the workflow management program for dynamic retention feedback, thereby updating the task status and archiving monitoring.

[0015] In a second aspect, the application further provides a device for large-scale satellite data processing and dynamic scheduling, which is applied in a distributed cluster, and the device comprises:

[0016] A task definition module for parsing XML format TaskTable, extracting input / output type, processing component, resource requirement and other metadata;

[0017] An input matching module for self-adapting multi-source files in product list, directory or database based on regular algorithm, realizing satellite number, orbit number, time interval and other dimensional screening;

[0018] A job order generation module for assembling command line, parameters and I / O list according to ESA JobOrder standard, supporting dynamic selection of multi-version components;

[0019] A flow scheduling module for flowchart instantiation, node dependency and event triggering, supporting node-level cloning, replacement and asynchronous triggering;

[0020] A cluster scheduling module for receiving jobs submitted by first-level scheduling, realizing queue management, resource allocation and parallel execution;

[0021] and,

[0022] A dynamic flow expansion module for automatically inserting / cloning sub-process according to output metadata; when multiple segments of data are generated, the corresponding segmented flow tree can be generated and activated online.

[0023] Preferably, the device further comprises a front panel for visualizing tracking of tasks and the entire processing flow in the cluster, or breakpoint recovery or one-key retry of tasks interrupted due to abnormalities in processing.

[0024] More preferably, the device further comprises a state feedback and monitoring module, which is configured to synchronize task states in real time by using Redis cache and write back to ecFlow for front-end visualization and breakpoint recovery.

[0025] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the large-scale satellite data processing and dynamic scheduling method according to the first aspect of the present application.

[0026] In addition, the present application further provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the large-scale satellite data processing and dynamic scheduling method according to the first aspect of the present application.

[0027] The large-scale satellite data processing and dynamic scheduling method and device according to the present application is a remote sensing data automatic processing system with the core innovation points and functions as follows:

[0028] 1. Unified task description TaskTable

[0029] The processor name, version, I / O type, resource upper limit, dynamic parameters, etc. are defined by XML, regular matching and time / orbit window are supported, and any satellite can be connected by adding TaskTable. Thus, the scattered scripts can be eliminated and the online threshold can be reduced.

[0030] 2. Double-layer decoupling scheduling architecture

[0031] On the one hand, the workflow management program (for example, ecFlow) is responsible for process dependency, event triggering, node cloning / replacement; on the other hand, the distributed computing task scheduling program (for example, HTCondor) is responsible for resource queuing and parallel execution. The double-layer decoupling scheduling architecture formed thereby can separate "process control" and "resource scheduling", and can be horizontally expanded without lock bottleneck.

[0032] 3. Dynamic sub-process driven by output

[0033] In the method of the present application, product.xml is parsed at runtime, new JobOrder is automatically generated for segmented / multi-station data, and ecFlow replace() or clone() is called to insert family online. Therefore, the process of running the method of the present application can be hot-plugged and hot-upgraded, avoiding downtime.

[0034] 4. Redis + Catalog real-time feedback

[0035] In the preferred method of the application, the task status and product list are written into Redis and synchronized into the database; thus, when the method of the application is run, the device front-end dashboard visual tracking, breakpoint recovery, and one-key retry can be realized. This can make the data processing process transparent and fault-tolerant.

[0036] Compared with the prior art, the scheme provided by the application can effectively break through the technical bottlenecks of multi-satellite high-frequency remote sensing production in process construction, resource scaling and dynamic adaptation, and has significant engineering and economic value. In particular, compared with traditional scripting scheduling, the application has the following beneficial effects:

[0037] Zero-script deployment: new satellite payloads only need to write TaskTable to be online, and the average deployment cycle is shortened from 2 weeks to 1 day;

[0038] Double-layer decoupling scheduling: under the same hardware (64 CPU nodes x 10) conditions, the throughput is improved by about 40 %;

[0039] Online evolution and self-healing: sub-process plugging and node cloning do not require downtime, and failed nodes can be breakpointed and retried or rolled back;

[0040] High-concurrency processing: for parallel production of 3 TB of raw data per day, the CPU utilization rate is > 90 %. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The figure is a schematic diagram of the system structure of the embodiment of the application.

[0042] Figure 2 The figure is an example of the h1e_czi data processing flow of the embodiment of the application. DETAILED DESCRIPTION

[0043] The technical scheme of the application is further described in detail through the description of the specific embodiments. The examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements, modules or steps, or elements, modules or steps having the same or similar functions throughout. The specific embodiments described below with reference to the drawings are exemplary and are intended to explain the application, and cannot be understood as limiting the application.

[0044] The application provides a process-driven remote sensing data scheduling system, which includes a device that can realize automatic processing and dynamic scheduling of large-scale satellite remote sensing data, as shown in FIG. 1. The device includes the module components shown in Table 1 and works cooperatively in the order.

[0045] Table 1

[0046] Serial number Module Key function 101 Task definition module Parse XML format TaskTable, extract metadata such as input / output type, processing component, resource requirement, etc. 102 Input matching engine Adapt multi-source files in product list, catalog or database based on regular algorithm, realize dimension filtering such as satellite number, orbit number, time interval, etc. 103 Job order generation module Assemble command line, parameters and I / O list according to ESA JobOrder standard, support dynamic selection of multi-version components. 104 Process scheduling engine (ecFlow) Responsible for process graph instantiation, node dependency and event triggering, support node-level cloning, replacement, asynchronous triggering. 105 Cluster scheduling engine (HTCondor) Receive jobs submitted by primary scheduling, realize queue management, resource allocation and parallel execution. 106 Dynamic process extension module Automatically insert / clone sub-processes according to output metadata; when multiple segments of data are generated, corresponding segmented process trees can be generated and activated online. 107 State feedback and monitoring mechanism Use Redis cache to synchronize task state in real time, and write back to ecFlow for front-end visualization and breakpoint recovery.

[0047] In conjunction with FIG. 2 (implementation flow timing diagram), the system performs the workflow in the following steps (step numbers prefixed with S):

[0048] S1 Task Parsing: When the data is ready, the core is to parse and verify the TaskTable XML file.

[0049] S1-1: Load TaskTable, parse <Processor_Name>, <version>, resource limits (Min_Disk_Space, Max_Time, Number_Of_CPUs), and <Exec_Path>.

[0050] An example of TaskTable XML definition is as follows:

[0051] <Ipf_Task_Table schemaVersion="0.30">

[0052] <Processor_Name>h1e_czi_l0a_pro< / Processor_Name> <!--①task name-->

[0053] <version> 01.00< / version> <!--②version-->

[0054] <Min_Disk_Space units="MB">1024< / Min_Disk_Space> <!--③resource requirement-->

[0055] <Max_Time units="sec">1800< / Max_Time> <!--④resource requirement-->

[0056] <List_of_Dyn_ProcParam count="…"> … < / List_of_Dyn_ProcParam>

[0057] <!--⑤dynamic parameters-->

[0058] <List_of_Pools count="1">

[0059] <pool>

[0060] <task> … < / task> <!-- 6. Processing components and parameter templates -->

[0061] < / pool>

[0062] < / List_of_Pools>< / Ipf_Task_Table>

[0063] Table 2

[0064] Field (label) Data type Necessity Technical role ① Processor_Name String Must Unique identification processing component for subsequent job <Task_Name> field reference. ② Version String Must Bind with executable file version, used for multi-version coexistence and rollback. ③ Min_Disk_Space④ Max_Time Integer Optional Mapped to RequestDisk / +MaxRuntime in HTCondor ClassAd. Input definition section See Process-1 Must Describe File_Type, regular pattern, time / orbit range, etc. Output definition section See Process-3 Must Give product catalog template (support placeholder ${SAT} / ${YYYY} / ${MM}). ⑤ List_of_Dyn_ProcParam Name-Value list Depending on business ​ ⑥ Task / File_Name Absolute path Must Point to executable component; parameter template supports ${Param} placeholder, replaced when generating job order.

[0065] S1-2: Input entry construction, generating Input objects one by one for <List_of_Inputs / Input>, fields including File_Type, regular expression, time / track window T0-T1, Mandatory, Major flag, etc.

[0066] S1-3: Output rule reading: parse <List_of_Outputs>, get File_Type and directory template (may contain placeholders ${SAT} / ${YYYY} / ${MM}).

[0067] S1-4: Dynamic parameter mapping: if <List_of_Dyn_ProcParam> exists, write default values into runtime dictionary; can be overwritten by external configuration during runtime.

[0068] S1-5: Check and cache: throw exception if task name-version or Exec_Path is missing; after check, serialize TaskTable object to Redis Key taskdef:{Name}:{Ver}, and publish taskdef_ready event.

[0069] S2 Input matching: input matching engine (102) listens to trigger signal, calls OCSPattern regular algorithm to perform file screening on directory / database and forms {File_Type: [path]} mapping. File names in data directory, product list, and database are parsed and extracted, and fields such as satellite number (sate), mission mode (mode), orbit number (orb), and time code (tma|tmb) are extracted. Main steps include:

[0070] S2-1 Data source enumeration: according to File_Type in TaskTable, three types of sources are enumerated, a) raw data directory; b) product list ProductList XML; c) catalog database Catalog DB.

[0071] S2-2 Regular matching: using regular templates provided by OCSPattern matching engine, file name fields (satellite, payload, date, orbit, segment number, etc.) are matched in sequence, and multi-file merging is performed according to DataId.

[0072] S2-3 Input check: mark inputs with Major=True, verify T0-T1 or Orb_From / Orb_To constraints; if not met, discard.

[0073] S2-4 Result set generation: output {File_Type: [absolute path]}, write to TaskTable.Task_.Dict_List_of_Inputs_Concrete_.

[0074] S2-5 Exception handling, Mandatory input missing → node state set to waiting_input with periodic retries; retry timeout records Warning but does not terminate suite.

[0075] S3 Job Order generation: Job Order generation module (103) generates JobOrderXML according to mapping and TaskTable and writes to workflow directory, meanwhile injects JOBORDER=<path> variable to corresponding ecFlow node.

[0076] S3-1 Call interface, Job Order generation module calls TaskTable.to_dict_joborder().

[0077] S3-2 XML assembly, call Ipf_conf for algorithm component configuration, write Processor_Name, Version, Order_Id (increment globally), resource limit; for Joborder data input <List_of_Inputs>, write file path obtained in step 2; for dynamic parameter <Dynamic_Processing_Parameters>, write dynamic parameters parsed or overwritten in step 1; for output directory <List_of_Outputs>, write output directory template.

[0078] S3-3 File save, generate JobOrder_<10-digit ID>.xml and save to ECF_HOME / <suite> / <family> / <task> / Directory

[0079] S3-4 variable mounting, writing JOBORDER in the corresponding ecFlow task node; if writing fails, the current node aborts immediately.

[0080] S4 process scheduling: the process engine realizes the driving and monitoring of the process in the form of a job order. The first process scheduling realizes the driving of the execution of the job order by the "job unit". The key design of the first process scheduling engine includes the trigger mechanism (Event) and the state conversion (Status). ecflow is integrated in the implementation. ecFlow (104) parses JobOrder, instantiates process nodes; if the node contains variables such as TO_BE_DUPLICATED, the family is automatically cloned; all dependencies are controlled by trigger / event. Figure 2 The processing flow is described taking H1E_CZI as an example, according to Figure 2 The process template defined in the implementation is as follows:

[0081] suite h1e_czi

[0082] edit ECF_MICRO "`"

[0083] family orbitr_family

[0084] family UNPACK_Family

[0085] task h1ef_l0a_pro

[0086] endfamily

[0087] task dynamic_process_l0a

[0088] trigger UNPACK_Family eq complete

[0089] edit FAMILY_BE_REPLACED CZI_Family

[0090] family CZI_Family

[0091] trigger dynamic_process_l0a eq complete

[0092] edit TO_BE_REPLACED_WHEN h1ef_l0a_pro

[0093] edit PROCESS_FILE_TYPE CZI_L0A_D,CZI_L0A_R

[0094] family STATION_Family

[0095] edit TO_BE_DUPLICATED 1

[0096] task h1e_czi_l0a_pro

[0097] edit TO_GEN_JOBORDER_ON_DUPLICATED h1ef_l0a_pro

[0098] family l0b_process

[0099] trigger h1e_czi_l0a_pro eq complete

[0100] edit TO_BE_REPLACED_WHEN h1e_czi_l0a_pro

[0101] family MSS_PAN_SEG_Family

[0102] edit TO_BE_DUPLICATED 1

[0103] task h1e_czi_l0b_pro_PAN

[0104] edit TO_GEN_JOBORDER_ON_DUPLICATED h1e_czi_l0a_pro

[0105] task h1e_czi_l0b_pro_MSS

[0106] edit TO_GEN_JOBORDER_ON_DUPLICATED h1e_czi_l0a_pro

[0107] task dynamic_process_std

[0108] trigger h1e_czi_l0b_pro_MSS eq complete and h1e_czi_l0b_pro_PAN eq complete

[0109] edit FAMILY_BE_REPLACED PAN_MSS_SEG_SLIC

[0110] family PAN_MSS_SEG_SLIC

[0111] trigger dynamic_process_std eq complete

[0112] edit TO_BE_REPLACED_WHEN h1e_czi_l0b_pro

[0113] family seg_slic_family

[0114] edit TO_BE_DUPLICATED 1

[0115] task h1e_czi_std_pro_PAN

[0116] edit TO_GEN_JOBORDER_ON_DUPLICATED h1e_czi_l0b_pro_PAN

[0117] edit XFER_PLIST XFER_h1e_czi_l0a_pro_PAN

[0118] task h1e_czi_std_pro_MSS

[0119] edit TO_GEN_JOBORDER_ON_DUPLICATED h1e_czi_l0b_pro_MSS

[0120] edit XFER_PLIST XFER_h1e_czi_l0a_pro_MSS

[0121] task h1e_czi_l2_pro

[0122] trigger h1e_czi_std_pro_PAN eq complete and h1e_czi_std_pro_MSS eq complete

[0123] endfamily

[0124] endfamily

[0125] endfamily

[0126] endfamily

[0127] endfamily

[0128] endfamily

[0129] endfamily

[0130] endsuite.

[0131] S4-1 Flow instantiation, ecFlow server reads node variable JOBORDER and creates task nodes corresponding to it.

[0132] S4-2 Dependency control, execution order is established between nodes through trigger / event expressions and complete / abort events; if a node contains variables such as TO_BE_DUPLICATED, FAMILY_BE_REPLACED, etc., cloning or replacement operations are performed at the family level by the server.

[0133] S4-3 Resource upper limit, all nodes are controlled by ecFlow limits (CPU, memory, water line), which can prevent queue overload.

[0134] S4-4 Error propagation, child node abort is transmitted upwards by default to the nearest family; operations and maintenance personnel can manually restart / force-complete through the ecFlow GUI.

[0135] S5 Cluster scheduling: the node script submits a ClassAd to HTCondor (105) with JobOrder as a parameter. Condor starts the processing component on RemoteHost and writes the running status back to Redis.

[0136] S5-1 Script submission, the task script reads the job order from the JOBORDER path and creates a submission description according to the ClassAd syntax:

[0137] cmd = Exec_Path;

[0138] arguments = JobOrder path;

[0139] RequestCpus / Memory corresponds to the TaskTable resource field;

[0140] requirements contain queue limits, architecture matching, etc.

[0141] S5-2 Schedule and execute, Condor schedd assigns jobs to matching compute nodes (RemoteHost); runtime status Idle / Running / Held / Completed written to Redis job: <clusterid>.

[0142] S5-3 Script polling, node script polls Redis status and calls ci.child_complete() if ExitCode < 127; otherwise child_abort.

[0143] S5-4 Retry mechanism, if ecFlow node variable ECF_TRIES > 1, Condor failure will be automatically re-casted until the number of times is exhausted.

[0144] S6 Dynamic flow extension: when node output product.xml shows the segment / multi-site result, dynamic flow extension module (106) automatically generates a sub JobOrder for each segment and calls ci.replace() to insert or clone a sub family in the running suite, realizing online evolution.

[0145] S6-1 Output detection, after the processing component ends, the script parses product.xml in the same directory; if multiple segments, sites or new version marks are detected, the extension process is entered.

[0146] S6-2 Sub JobOrder generation, for each segment / site data, step 3 is called to regenerate JobOrder; the new file is stored in the sub family path.

[0147] S6-3 Online insertion, call ecFlow API replace() to inject the new family into the running suite; if there is a TO_BE_DUPLICATED identifier, clone the same level.

[0148] S6-4 Immediate triggering, after writing JOBORDER and necessary input variables for the new task, directly resume execution, without affecting the main process to continue.

[0149] S7 Status feedback and archiving: the script writes the PLIST variable (product list path) back to ecFlow; extract_plist_and_cate() parses the list and writes it to the Catalog DB, and finally the products and logs are archived to cold storage according to the date-satellite hierarchy; the front-end Dashboard displays the process Gantt chart and resource utilization in real time through Redis subscription. Product list writing back: the script writes PLIST=<product.xml> into the node variable in the post stage, and persists it in Redis plist:<Order_Id>.

[0150] S7-1 Catalog, the background service extract_plist_and_cate() writes the product meta information (satellite, payload, orbit, time period, storage path, production version) into the Catalog database.

[0151] S7-2 File archiving, according to <archive> / SAT / YYYY / MM / directory structure to copy product data, logs, JobOrder; logs and archive files are retained for no less than 6 years.

[0152] S7-3 visualization, front-end Dashboard subscribes Redis __keyspace@0__:* event real-time refresh process Gantt and resource utilization; when the node abort or archive fails, send an alarm to the operation and maintenance system.

[0153] Embodiments of the present application also provide an electronic device, which includes the apparatus of the foregoing embodiments. The electronic device exists in a variety of forms, including but not limited to:

[0154] (1) Ultra-mobile personal computer device: This kind of device belongs to the category of personal computers, has computing and processing functions, and generally also has network connection features. This kind of terminal is generally a computer for processing computing data.

[0155] (2) Server: A device that provides computing services. The components of a server include a processor, a hard disk, a memory, a system bus, etc. The architecture of a server is similar to that of a general-purpose computer, but because it needs to provide high-reliability services, it has higher requirements in terms of processing capability, stability, reliability, security, scalability, manageability, etc.

[0156] (3) Other electronic devices with data computing and processing.

[0157] In addition, the terms "first", "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0158] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms is not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or features of different embodiments or examples described in the present application without contradiction.

[0159] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that changes, modifications, substitutions and variations can be made by those skilled in the art without departing from the scope of the present application.< / archive> < / clusterid> < / task> < / family> < / suite> < / version>

Claims

1. A method for large-scale satellite data processing and dynamic scheduling, characterized in that, The method comprises the following steps: 1) writing an extensible markup language defined task table for different satellite payloads to realize unified task description; 2) after obtaining remote sensing observation data from different satellites or payloads, parsing the corresponding extensible markup language defined task table, and extracting metadata; 3) according to the metadata, self-adapting multi-source files in a product list, a catalog or a database based on a regular algorithm to realize multi-dimensional screening; 4) assembling command lines, parameters and input / output lists according to the data obtained through multi-dimensional screening according to the ESA JobOrder standard to generate a job order; 5) respectively using different program to process flow control and resource scheduling; including: submitting the job order to a workflow management program for flow control; submitting the flow node to a distributed computing task scheduling program for resource scheduling; 6) automatically inserting / cloning a sub-process according to the metadata output by the distributed computing task scheduling program; 7) using Redis cache to realize real-time synchronization of task status, and writing back to the workflow management program for dynamic retention feedback, and then updating the task status and archiving monitoring.

2. The method of claim 1, wherein: 5) The workflow management program is an ecFlow program.

3. The method of any of claims 1-2, wherein: 5) The distributed computing task scheduling program is an HTCondor program.

4. An apparatus for large-scale satellite data processing and dynamic scheduling, applied in a distributed cluster, characterized in that, The device comprises: a task definition module for parsing an extensible markup language defined task table and extracting metadata; the metadata comprises input / output types, processing components and / or resource requirements; an input matching module for self-adapting multi-source files in a product list, a catalog or a database based on a regular algorithm to realize satellite number, orbit number, time interval dimension screening; a job order generation module for assembling command lines, parameters and input / output lists according to the ESA JobOrder standard to support dynamic selection of multiple versions of components; a flow scheduling module for flowchart instantiation, node dependency and event triggering, supporting node-level cloning, replacement and asynchronous triggering; a cluster scheduling module for receiving jobs submitted by the first-level scheduling to realize queue management, resource allocation and parallel execution; and a dynamic process extension module for automatically inserting / cloning a sub-process according to the output metadata; when multiple segments of data are generated, the corresponding segmented process tree is generated and activated online.

5. The apparatus of claim 4, wherein: The flow scheduling module is embedded with an ecFlow program as a workflow management program for flowchart instantiation, node dependency and event triggering, supporting node-level cloning, replacement and asynchronous triggering.

6. The apparatus of any of claims 4-5, wherein: The cluster scheduling module is embedded with an HTCondor program for receiving jobs submitted by the first-level scheduling to realize queue management, resource allocation and parallel execution.

7. The apparatus of claim 5, wherein: Further provided is a front-end dashboard for visualizing tracking of the entire process and cluster tasks, or breakpoint recovery or one-key retry operation processing of an abnormal task node in a process link.

8. The apparatus of claim 7, wherein: Further provided is a state feedback and monitoring module for using Redis cache to realize real-time synchronization of task status, and writing back to the ecFlow for front-end visualization and breakpoint recovery.

9. An electronic device, comprising: The method comprises the following steps: The memory, the processor and the computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the method for processing and dynamically scheduling massive satellite data according to any one of claims 1-3.

10. A computer-readable storage medium, characterized in that: The computer program is stored in the memory, and the processor executes the computer program to implement the method for processing and dynamically scheduling massive satellite data according to any one of claims 1-3.

Images