Large-scale satellite data processing and dynamic scheduling method and device

Abstract

The invention provides a large-scale satellite data processing and dynamic scheduling method. The large-scale satellite data processing and dynamic scheduling method comprises the steps of compiling a task table defined by an extensible markup language for different satellites or loads; analyzing task tables corresponding to different satellites or loads, and extracting metadata; according to the metadata, performing multi-dimensional screening based on a regular algorithm; generating a job list from the data obtained by multi-dimensional screening; submitting the job order to a workflow management program for flow control; submitting the process nodes to a distributed computing task scheduling program for resource scheduling; automatically inserting / cloning a sub-process according to metadata output by the distributed computing task scheduling program; and synchronizing the task state in real time by utilizing the Redis cache, and writing back the workflow management program to carry out dynamic retention feedback. The invention further provides a device for large-scale satellite data processing and dynamic scheduling. According to the method, the newly added satellite load deployment period is short, the data throughput is high, and online evolution, self-healing and high-concurrency processing can be achieved.

Description

Technical Field

[0001] The present invention relates to large-scale data processing and task scheduling technologies, and in particular to an automated processing and process control method for remote sensing observation data of multiple satellites and multiple payloads. Background Art

[0002] With the significant increase in the number of remote sensing satellites and the data acquisition frequency, the data acquisition of multiple satellites and multiple payloads forms large-volume, high-frequency, and heterogeneous remote sensing data. The traditional "script + single-layer scheduling" system reveals bottlenecks in process construction, resource scaling, and dynamic evolution. The specific problems hindering its industrial application include: ① high maintenance cost, manifested as the need to rewrite scripts for each newly added satellite / payload; ② scheduling coupling, with dependencies, resource allocation, and job submission mixed, making it difficult to expand; ③ lack of adaptability, where if the algorithm is segmented or upgraded during operation, it is impossible to insert or replace subtasks online; ④ any anomaly requires manual intervention, limiting the overall throughput and reliability.

[0003] Therefore, it is necessary to propose a new method more suitable for large-scale remote sensing data processing to solve the above-mentioned industrial application problems. Summary of the Invention

[0004] Aiming at the problem that the traditional script maintenance method is difficult to adapt to high-concurrency and high-frequency heterogeneous data processing, the primary object of the present invention is to provide an automated processing and dynamic scheduling method for remote sensing data with a short deployment cycle, large throughput, and self-adaptability, so as to achieve full-process automatic control from data reception, product generation to archiving, thereby significantly reducing maintenance costs, improving processing efficiency, and enhancing industrial applicability.

[0005] Another object of the present invention is to provide a device and an electronic device capable of executing the above-mentioned automated processing and dynamic scheduling method for remote sensing data.

[0006] The above objects of the present invention are achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides a large-scale satellite data processing and dynamic scheduling method, including the following steps:

[0008] 1) Write a task table (Task Table) defined in Extensible Markup Language (XML) for different satellites or payloads to achieve unified task description;

[0009] 2) After obtaining remote sensing observation data from different satellites or payloads, parse the corresponding XML format Task Table to extract metadata;

[0010] 3) Based on the metadata, self-adaptively screen multi-source files in the product list, directory, or database based on a regular algorithm to achieve multi-dimensional screening;

[0011] 4) Assemble the command line, parameters, and I / O list from the data obtained through multi-dimensional screening according to the ESA JobOrder standard to generate a job sheet.

[0012] 5) Adopt different program processing flow controls and resource scheduling respectively; including: submitting the job sheet to a workflow management program (such as ecFlow) for process control; submitting the process nodes to a distributed computing task scheduling program (such as HTCondor) for resource scheduling.

[0013] 6) Automatically insert / clone sub-processes according to the metadata output by the distributed computing task scheduling program.

[0014] 7) Use Redis cache to synchronize the task status in real time, write back to the workflow management program for dynamic retention feedback, and then update the task status and archive the monitoring.

[0015] In a second aspect, the present invention also provides a device for large-scale satellite data processing and dynamic scheduling, which is applied in a distributed cluster. The device includes:

[0016] A task definition module, which is used to parse the XML format TaskTable and extract metadata such as input / output types, processing components, and resource requirements.

[0017] An input matching module, which is used to adaptively match multi-source files in a product list, directory, or database based on a regular algorithm to achieve screening in dimensions such as satellite number, orbit number, and time interval.

[0018] A job sheet generation module, which is used to assemble the command line, parameters, and I / O list according to the ESA JobOrder standard and support dynamic selection of multi-version components.

[0019] A process scheduling module, which is used for flowchart instantiation, node dependency, and event triggering, and supports node-level cloning, replacement, and asynchronous triggering.

[0020] A cluster scheduling module, which is used to receive the jobs submitted by the first-level scheduling, and implement queue management, resource allocation, and parallel execution.

[0021] And,

[0022] A dynamic process extension module, which is used to automatically insert / clone sub-processes according to the output metadata; when the task generates multiple segments of data, it can generate and activate the corresponding segmented process tree online.

[0023] In the preferred device of the present invention, a front-end dashboard is further provided, which is used to visually track the tasks executed in the cluster and the entire processing flow, or perform operation processing such as breakpoint recovery or one-key retry on the tasks interrupted due to exceptions during processing.

[0024] In a more preferred embodiment of the device of the present invention, a status feedback and monitoring module is further provided, which is used to synchronize the task status in real time by using Redis cache and write back to ecFlow for front-end visualization and breakpoint recovery.

[0025] In a third aspect, the present invention further provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the large-scale satellite data processing and dynamic scheduling method described in the first aspect of the present invention is implemented.

[0026] In addition, the present invention further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the large-scale satellite data processing and dynamic scheduling method described in the first aspect of the present invention is implemented.

[0027] The large-scale satellite data processing and dynamic scheduling method and device of the present invention is a remote sensing data automatic processing system of "unified task description + hierarchical decoupled scheduling + online sub-process evolution", and the core innovation points and their functions are as follows:

[0028] 1. Unified task description TaskTable

[0029] Define the processor name, version, I / O type, resource limit, dynamic parameters, etc. in XML, support regular matching and time / orbit window; any satellite can be connected by simply adding a TaskTable. This can eliminate scattered scripts and lower the online threshold.

[0030] 2. Double-layer decoupled scheduling architecture

[0031] On the one hand, the workflow management program (such as ecFlow) is responsible for process dependencies, event triggering, node cloning / replacement; on the other hand, the distributed computing task scheduler (such as HTCondor) is responsible for resource queuing and parallel execution. The resulting double-layer decoupled scheduling architecture can separate "process control" and "resource scheduling", and there is no lock bottleneck for horizontal expansion.

[0032] 3. Output-driven dynamic sub-process

[0033] In the method of the present invention, product.xml is parsed at runtime, a new JobOrder is automatically generated for segmented / multi-station data, and ecFlow replace() or clone() is called to insert the family online. Therefore, during the process of running the method of the present invention, hot plugging and hot upgrading can be performed, avoiding downtime.

[0034] 4. Redis + Catalog Real-time Feedback

[0035] In the preferred method of the present invention, the task status and product list are written into Redis and synchronized into the database. Thus, when running the method of the present invention, visual tracking on the front-end dashboard of the device, breakpoint recovery, and one-key retry can be achieved. This can make the data processing process transparent and highly fault-tolerant.

[0036] Compared with the prior art, the solution provided by the present invention can effectively break through the technical bottlenecks in process construction, resource scaling, and dynamic adaptability of multi-satellite high-frequency remote sensing production, and has significant engineering and economic value. Especially compared with traditional scripted scheduling, the beneficial effects obtained by the present invention include:

[0037] Zero-script deployment: For newly added satellite payloads, only the TaskTable needs to be written to go online, and the average deployment cycle is shortened from 2 weeks to 1 day;

[0038] Double-layer decoupled scheduling: Under the same hardware conditions (64 CPU nodes × 10), the throughput is increased by approximately 40%;

[0039] Online evolution and self-healing: Plugging and unplugging of sub-processes and cloning of nodes do not require downtime, and failed nodes can be retried at the breakpoint or rolled back;

[0040] High-concurrency processing: For parallel production of 3 TB of raw data per day on average, the CPU utilization rate > 90%. Description of the Drawings

[0041] Figure 1 It is a schematic diagram of the system structure of the specific embodiment of the present invention.

[0042] Figure 2 It is an example of the h1e_czi data processing flow of the specific embodiment of the present invention. Specific Embodiments

[0043] The technical solutions of the present application will be further described in detail below through the description of specific embodiments. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements, modules, or steps, or elements, modules, or steps with the same or similar functions, from beginning to end. The specific embodiments described below with reference to the drawings are exemplary and are intended to explain the present application, and should not be construed as a limitation to the present application.

[0044] The present invention provides a process-driven remote sensing data scheduling system, including a device that can realize automatic processing and dynamic scheduling of large-scale satellite remote sensing data. As shown in Figure 1, the device includes the module components shown in Table 1 below and works in cooperation in the following order:

[0045] Table 1

[0046] Serial Number Module Key Function 101 Task Definition Module Parse the XML-formatted TaskTable, and extract metadata such as input / output types, processing components, resource requirements, etc. 102 Input Matching Engine Based on a regular algorithm, self-adapt to multi-source files in product lists, catalogs or databases, and implement filtering in dimensions such as satellite number, orbit number, time interval, etc. 103 Job Sheet Generation Module Assemble command lines, parameters, and I / O lists according to the ESA JobOrder standard, and support dynamic selection of multi-version components. 104 Process Scheduling Engine (ecFlow) Responsible for flowchart instantiation, node dependencies, and event triggering, and supports node-level cloning, replacement, and asynchronous triggering. 105 Cluster Scheduling Engine (HTCondor) Receive jobs submitted by the primary scheduling, and implement queue management, resource allocation, and parallel execution. 106 Dynamic Process Extension Module Automatically insert / clone sub-processes according to the output metadata; when the task generates multi-segment data, online generate and activate the corresponding segmented process tree. 107 Status Feedback and Monitoring Mechanism Use Redis cache to synchronize task status in real time, and write back to ecFlow for front-end visualization and breakpoint recovery.

[0047] Combined with Figure 2 (Implementation Process Timing Diagram), the system executes the workflow in the following steps (step number prefix S):

[0048] S1 Task Parsing: When the data is ready, its core is the parsing and verification of the TaskTable XML file.

[0049] S1-1: Load TaskTable, parse <Processor_Name>, <version>, resource limits (Min_Disk_Space, Max_Time, Number_Of_CPUs), and <Exec_Path>.

[0050] An example of the TaskTable XML definition is as follows:

[0051] <Ipf_Task_TableschemaVersion="0.30">

[0052] <Processor_Name>h1e_czi_l0a_pro< / Processor_Name> <!--① Task name -->

[0053] <version> 01.00< / version> <!--② Version -->

[0054] <Min_Disk_Spaceunits="MB">1024< / Min_Disk_Space> <!--③ Resource requirements -->

[0055] <Max_Timeunits="sec">1800< / Max_Time> <!--④ Resource requirements -->

[0056] <List_of_Dyn_ProcParamcount="…"> … < / List_of_Dyn_ProcParam>

[0057] <!--⑤ Dynamic parameters -->

[0058] <List_of_Poolscount="1">

[0059] <pool>

[0060] <task> … < / task> <!--⑥Processing component and parameter template-->

[0061] < / pool>

[0062] < / List_of_Pools>< / Ipf_Task_Table>

[0063] Table 2

[0064] Field (Label) Data Type Necessity Technical Role ① Processor_Name String Must Uniquely identify the processing component for subsequent reference in the <Task_Name> field of the work order. ② Version String Must Bind to the version of the executable file, and is used for coexistence and rollback of multi-versions. ③ Min_Disk_Space④ Max_Time Integer Optional Mapped to RequestDisk / +MaxRuntime in HTCondor ClassAd. Input Definition Section See Process-1 Must Describe File_Type, regular pattern, time / orbit range, etc. Output Definition Section See Process-3 Must Give the product catalog template (supporting placeholders ${SAT} / ${YYYY} / ${MM}). ⑤ List_of_Dyn_ProcParam Name-Value List Depending on the business Transcribe to JobOrder <Dynamic_Processing_Parameters> during runtime. ⑥ Task / File_Name Absolute Path Must Point to the executable component; the parameter template supports the ${Param} placeholder, which is replaced when generating the job sheet.

[0065] S1-2: Input entry construction, generating an Input object for each <List_of_Inputs / Input> item, with fields including File_Type, regular expression, time / orbit window T0–T1, Mandatory, Major flag, etc.

[0066] S1-3: Output rule reading: Parse <List_of_Outputs> to obtain the output File_Type and directory template (which may contain placeholders ${SAT} / ${YYYY} / ${MM}).

[0067] S1-4: Dynamic parameter mapping: If <List_of_Dyn_ProcParam> exists, write the default values into the runtime dictionary; they can be overridden by external configurations during runtime.

[0068] S1-5: Verification and caching: Throw an exception if the task name - version or Exec_Path is missing; after passing the verification, serialize the TaskTable object to the Redis Key taskdef:{Name}:{Ver}, and publish the taskdef_ready event.

[0069] S2 Input matching: The input matching engine (102) listens for trigger signals, calls the OCSPattern regular algorithm to filter files in the directory / database and form a {File_Type: [path]} mapping. Parse and extract the file names in the three data sources of the data directory, product list, and database, and extract fields such as satellite number (sate), task mode (mode), orbit number (orb), time code (tma|tmb), etc. The main steps include:

[0070] S2-1 Data source enumeration, enumerate the three types of sources according to the File_Type in TaskTable: a) Raw data directory; b) Product list ProductList XML; c) Catalog database Catalog DB.

[0071] S2-2 Regular matching, use the regular templates provided by the OCSPattern matching engine to match the file name fields (satellite, payload, date, orbit, segment number, etc.) in sequence, and merge multiple files by DataId.

[0072] S2-3 Input check, mark the inputs with Major=True, and verify the T0–T1 or Orb_From / Orb_To constraints; discard those that do not meet the requirements.

[0073] S2-4 Result set generation, output {File_Type: [absolute path]}, and write it into TaskTable.Task_.Dict_List_of_Inputs_Concrete_.

[0074] S2-5 Exception handling, Mandatory input missing → Set the node status to waiting_input and retry periodically; record Warning for retry timeout, but do not terminate the suite.

[0075] S3 Job order generation: The job order generation module (103) generates JobOrderXML according to the mapping and TaskTable and writes it to the workflow directory. At the same time, inject the JOBORDER=<path> variable into the corresponding ecFlow node.

[0076] S3-1 Call interface, the job order generation module calls TaskTable.to_dict_joborder().

[0077] S3-2 XML assembly, for the algorithm component configuration (Ipf_conf), write Processor_Name, Version, Order_Id (globally incremented), and resource limits; for the Joborder data input <List_of_Inputs>, write the file path obtained in step 2; for the dynamic parameters <Dynamic_Processing_Parameters>, write the dynamically parsed or overwritten parameters in step 1; for the output directory <List_of_Outputs>, write the output directory template.

[0078] S3-3 File saving, generate JobOrder_<10-digit ID>.xml and save it to ECF_HOME / <suite> / <family> / <task> / Table of Contents

[0079] S3-4 Variable Mounting, write JOBORDER to the corresponding ecFlow task node; if the write fails, the current node will abort immediately.

[0080] S4 Process Scheduling: The process engine realizes the driving and monitoring of the process in the form of a job order. The first-level process scheduling realizes the execution of the job order made by the "job unit". The key to the design of the first-level process scheduling engine includes the trigger mechanism (Event) and status transition (Status). ecflow is integrated in the implementation. ecFlow (104) parses the JobOrder and instantiates the process nodes; if the node contains variables such as TO_BE_DUPLICATED, it automatically clones the family; all dependencies are controlled by trigger / event. Figure 2 Taking H1E_CZI as an example, its processing process is described. According to Figure 2 The implemented process template is defined as follows:

[0081] suite h1e_czi

[0082] edit ECF_MICRO "`"

[0083] family orbitr_family

[0084] family UNPACK_Family

[0085] task h1ef_l0a_pro

[0086] endfamily

[0087] task dynamic_process_l0a

[0088] trigger UNPACK_Family eq complete

[0089] edit FAMILY_BE_REPLACED CZI_Family

[0090] family CZI_Family

[0091] trigger dynamic_process_l0a eq complete

[0092] edit TO_BE_REPLACED_WHEN h1ef_l0a_pro

[0093] edit PROCESS_FILE_TYPE CZI_L0A_D,CZI_L0A_R

[0094] family STATION_Family

[0095] edit TO_BE_DUPLICATED 1

[0096] task h1e_czi_l0a_pro

[0097] edit TO_GEN_JOBORDER_ON_DUPLICATED h1ef_l0a_pro

[0098] family l0b_process

[0099] trigger h1e_czi_l0a_pro eq complete

[0100] edit TO_BE_REPLACED_WHEN h1e_czi_l0a_pro

[0101] family MSS_PAN_SEG_Family

[0102] edit TO_BE_DUPLICATED 1

[0103] task h1e_czi_l0b_pro_PAN

[0104] edit TO_GEN_JOBORDER_ON_DUPLICATED h1e_czi_l0a_pro

[0105] task h1e_czi_l0b_pro_MSS

[0106] edit TO_GEN_JOBORDER_ON_DUPLICATED h1e_czi_l0a_pro

[0107] task dynamic_process_std

[0108] trigger h1e_czi_l0b_pro_MSS eq complete and h1e_czi_l0b_pro_PAN eqcomplete

[0109] edit FAMILY_BE_REPLACED PAN_MSS_SEG_SLIC

[0110] family PAN_MSS_SEG_SLIC

[0111] trigger dynamic_process_std eq complete

[0112] edit TO_BE_REPLACED_WHEN h1e_czi_l0b_pro

[0113] family seg_slic_family

[0114] edit TO_BE_DUPLICATED 1

[0115] task h1e_czi_std_pro_PAN

[0116] edit TO_GEN_JOBORDER_ON_DUPLICATED h1e_czi_l0b_pro_PAN

[0117] edit XFER_PLIST XFER_h1e_czi_l0a_pro_PAN

[0118] task h1e_czi_std_pro_MSS

[0119] edit TO_GEN_JOBORDER_ON_DUPLICATED h1e_czi_l0b_pro_MSS

[0120] edit XFER_PLIST XFER_h1e_czi_l0a_pro_MSS

[0121] task h1e_czi_l2_pro

[0122] trigger h1e_czi_std_pro_PAN eq complete and h1e_czi_std_pro_MSS eqcomplete

[0123] endfamily

[0124] endfamily

[0125] endfamily

[0126] endfamily

[0127] endfamily

[0128] endfamily

[0129] endfamily

[0130] endsuite。

[0131] The S4-1 process is instantiated, and the ecFlow server reads the node variable JOBORDER and creates a corresponding task node.

[0132] S4-2 Dependency control: The execution order is established between nodes through trigger / event expressions and complete / abort events; if a node contains variables such as TO_BE_DUPLICATED and FAMILY_BE_REPLACED, the server performs cloning or replacement operations at the family level.

[0133] S4-3 Resource limit: All nodes are controlled by ecFlow limit (CPU, memory, water level) to prevent queue overload.

[0134] S4-4 Error propagation: The abort of a child node is default passed up to the nearest family; the operation and maintenance can perform manual restart / force-complete through the ecFlow GUI.

[0135] S5 Cluster scheduling: The node script submits a ClassAd to HTCondor (105) with JobOrder as a parameter. Condor starts a processing component on RemoteHost and writes the running status back to Redis.

[0136] S5-1 Script submission: The task script reads the job order from the JOBORDER path and creates a submission description according to the ClassAd syntax:

[0137] cmd = Exec_Path;

[0138] arguments = JobOrder path;

[0139] RequestCpus / Memory corresponds to the resource fields in TaskTable;

[0140] requirements include queue limits, architecture matching, etc.

[0141] S5-2 Scheduling and Execution, Condor schedd assigns jobs to matching compute nodes (RemoteHost); the runtime states Idle / Running / Held / Completed are written to Redis job: <clusterid>。

[0142] S5-3 Script polling, the node script polls the Redis status and calls ci.child_complete() when ExitCode < 127; otherwise child_abort.

[0143] S5-4 Retry mechanism, if the ecFlow node variable ECF_TRIES > 1, the Condor failure will be automatically re-invested until the number of times is exhausted.

[0144] S6 Dynamic process extension: When the node outputs product.xml showing the segmenting / multi-station results, the dynamic process extension module (106) automatically generates sub-JobOrders for each segment, and calls ci.replace() to insert or clone sub-families within the running suite, realizing online evolution.

[0145] S6-1 Output detection, after the processing component ends, the script parses the product.xml in the same directory; if multi-segment, site, or new version markers are detected, enter the extension process.

[0146] S6-2 Sub-JobOrder generation, calls step 3 to regenerate the JobOrder for each segment / station data; the new file is stored in the sub-family path.

[0147] S6-3 Online insertion, calls the ecFlow API replace() to inject the new family into the running suite; if there is a TO_BE_DUPLICATED flag, clone at the same level.

[0148] S6-4 Immediate trigger, writes the JOBORDER and necessary input variables for the new task and then directly resumes execution, without affecting the continuation of the main process.

[0149] S7 Status feedback and archiving: The script writes back the PLIST variable (product list path) to ecFlow; extract_plist_and_cate() parses the list and writes it to the Catalog DB, and the final products and logs are archived to cold storage according to the date-satellite hierarchy; the front-end Dashboard subscribes through Redis to display the process Gantt chart and resource utilization in real time. Product list write-back: The script writes PLIST=<product.xml> to the node variable in the post stage and persists it in Redis plist:<Order_Id>.

[0150] S7-1 Cataloging and warehousing, the background service extract_plist_and_cate() writes the product meta-information (satellite, payload, orbit, time period, storage path, production version) into the Catalog database.

[0151] S7-2 File archiving, according to <archive> / SAT / YYYY / MM / The directory structure copies product data, logs, and JobOrders; logs and archived files are retained for at least 6 years.

[0152] S7-3 visualization, the front-end Dashboard subscribes to the Redis __keyspace@0__:* event to refresh the process Gantt chart and resource utilization rate in real time; when a node aborts or archiving fails, an alarm is sent to the operation and maintenance system.

[0153] The embodiments of the present application also provide an electronic device, which includes a device for implementing the foregoing implementation manners. This electronic device exists in various forms, including but not limited to:

[0154] (1) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing capabilities, and generally also have networking characteristics. Such terminals are generally computers for processing computing data.

[0155] (2) Servers: Devices that provide computing services. The composition of a server includes a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but due to the need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0156] (3) Other electronic devices with data computing and processing capabilities.

[0157] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0158] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0159] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.< / archive> < / clusterid> < / task> < / family> < / suite> < / version>

Claims

1. A method for large-scale satellite data processing and dynamic scheduling, characterized in that, It includes the following steps: 1) Write a task table defined in Extensible Markup Language for different satellite payloads to achieve unified task description; 2) After obtaining remote sensing observation data from different satellites or payloads, parse the corresponding task table defined in Extensible Markup Language and extract metadata; 3) Based on the metadata, self-adapt to multi-source files in the product list, catalog or database based on the regular algorithm to achieve multi-dimensional screening; 4) Assemble the command line, parameters and input / output list according to the ESA JobOrder standard for the data obtained from multi-dimensional screening to generate a job sheet; 5) Adopt different program processing flow control and resource scheduling respectively; including: submitting the job sheet to the workflow management program for process control; submitting the process node to the distributed computing task scheduler for resource scheduling; 6) Automatically insert / clone sub-processes according to the metadata output by the distributed computing task scheduler; 7) Use Redis cache to synchronize the task status in real time, write back to the workflow management program for dynamic retention feedback, and then update the task status and archive the monitoring.

2. The method according to claim 1, wherein: 5) The workflow management program described is the ecFlow program.

3. The method according to any one of claims 1-2, characterized in that: 5) The distributed computing task scheduler described is the HTCondor program.

4. A device for large-scale satellite data processing and dynamic scheduling, which is applied in a distributed cluster, is characterized in that The device includes: A task definition module, which is used to parse the task table defined in Extensible Markup Language and extract metadata; the metadata includes input / output type, processing component and / or resource requirements; An input matching module, which is used to self-adapt to multi-source files in the product list, catalog or database based on the regular algorithm to achieve screening in dimensions of satellite number, orbit number and time interval; A job sheet generation module, which is used to assemble the command line, parameters and input / output list according to the ESA JobOrder standard and support dynamic selection of multi-version components; A process scheduling module, which is used for flowchart instantiation, node dependency and event triggering, and supports node-level cloning, replacement and asynchronous triggering; A cluster scheduling module, which is used to receive the jobs submitted by the first-level scheduling, and achieve queue management, resource allocation and parallel execution; And, A dynamic process extension module, which is used to automatically insert / clone sub-processes according to the output metadata; when the task generates multi-segment data, online generate and activate the corresponding segmented process tree.

5. The device according to claim 4, characterized in that: The process scheduling module described embeds the ecFlow program as the workflow management program, which is used for flowchart instantiation, node dependency and event triggering, and supports node-level cloning, replacement and asynchronous triggering.

6. The device according to any one of claims 4-5, characterized in that: The cluster scheduling module described embeds the HTCondor program, which is used to receive the jobs submitted by the first-level scheduling, and achieve queue management, resource allocation and parallel execution.

7. The device according to claim 5, characterized in that: A front-end dashboard is further provided, which is used for visual tracking of the entire process and cluster tasks, or for breakpoint recovery or one-key retry operation processing of abnormal task nodes in a certain link of the process.

8. The device according to claim 7, characterized in that: A status feedback and monitoring module is also provided, which is used to use Redis cache to synchronize the task status in real time and write back to ecFlow for front-end visualization and breakpoint recovery.

9. An electronic device, characterized in that, It includes: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the large-scale satellite data processing and dynamic scheduling method according to any one of claims 1-3 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the large-scale satellite data processing and dynamic scheduling method according to any one of claims 1-3 is implemented.

Images