Multi-star multi-task parallel scheduling processing method, device, equipment, medium and product

Abstract

This application discloses a multi-satellite, multi-task parallel scheduling and processing method, apparatus, device, medium, and product, relating to the field of data processing technology. The method includes: acquiring satellite data according to regional priority; acquiring real-time tasks sequentially according to a real-time service scheduling plan; determining whether a real-time task meets execution requirements; when multiple real-time tasks meet execution requirements, determining whether execution resources can simultaneously execute all real-time tasks that meet the requirements; when execution resources cannot simultaneously execute all real-time tasks that meet the requirements, executing the real-time tasks that meet the requirements sequentially according to payload priority, and obtaining task execution results respectively. This application acquires satellite data sequentially according to regional priority, and processes real-time tasks according to payload priority when execution resources cannot simultaneously execute all tasks, thereby effectively improving data processing speed.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a multi-star, multi-task parallel scheduling processing method, apparatus, equipment, medium, and product. Background Technology

[0002] DBNet (The DirectBroadcast Network) is a global, multi-national, operational dataset of real-time low-Earth orbit (LEO) satellite direct-received data (similar to the MPT or HRPT data from the Fengyun-3 satellite) obtained through a global network. It is received by member direct broadcast receiving stations worldwide and preprocessed. Currently, the European Centre for Numerical Weather Prediction has established a timeliness monitoring system to monitor the acquisition of data from payloads such as AMSU-A, MHS and HIRS (ATOVS), IASI, ATMS, and CrIS. However, it cannot yet acquire high-timeliness data globally. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, equipment, medium, and product for multi-satellite multi-task parallel scheduling and processing, which can realize the preprocessing of low-orbit satellite data and global sharing in a short time.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] Firstly, this application provides a multi-star, multi-task parallel scheduling processing method, including:

[0006] Satellite data is acquired according to regional priority; the regional priority is determined based on the region to which the satellite belongs.

[0007] Real-time tasks are obtained sequentially according to the real-time service scheduling plan; the real-time service scheduling plan is a pre-defined plan.

[0008] Determine whether the real-time task meets the execution requirements; if the satellite data required by the real-time task has been received completely, determine that the real-time task meets the execution requirements; if the satellite data required by the real-time task has not been received completely, determine that the real-time task does not meet the execution requirements.

[0009] When the real-time task meets the execution requirements, the real-time task is executed, and the task execution result is obtained;

[0010] When multiple real-time tasks meet the execution requirements, determine whether the execution resources can execute all the real-time tasks that meet the execution requirements in parallel at the same time;

[0011] When the execution resources are capable of executing all the real-time tasks that meet the execution requirements in parallel, all the real-time tasks that meet the execution requirements are executed in parallel, and the task execution results are obtained respectively.

[0012] When the execution resources cannot execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed sequentially according to the load priority order, and the task execution results are obtained respectively; the load priority order is divided according to the satellite's load.

[0013] The results of the task execution will be shared with users worldwide.

[0014] Optionally, the regional priority order is as follows: domestic satellites are the first priority, and foreign satellites are the second priority.

[0015] Optionally, satellite data may be acquired according to regional priority, specifically including:

[0016] Obtain direct satellite reception data and configuration data according to regional priority;

[0017] Save the directly received data and the configuration data to the resource pool;

[0018] The satellite data is obtained by parsing the directly received data based on the configuration data.

[0019] Optionally, the real-time task includes: data preprocessing and format conversion tasks;

[0020] Executing the real-time task includes:

[0021] The satellite data is preprocessed to obtain preprocessed data; the preprocessing includes one or more of the following: data quality inspection, data unpacking, scientific data positioning, and scientific data calibration.

[0022] The preprocessed data is then converted to a new format to obtain the task execution result.

[0023] Optionally, the payload priority order is: FY3G satellite, FY3E satellite, FY3D satellite, NOAA20 satellite, and Metop-C satellite.

[0024] Optionally, after sharing the task execution results with global users, the method further includes:

[0025] The work of each satellite will be summarized in the order of day / week / ten-day period / month.

[0026] Secondly, this application provides a multi-star, multi-task parallel scheduling and processing apparatus, comprising:

[0027] The data acquisition module is used to acquire satellite data according to regional priority order; the regional priority order is divided according to the region to which the satellite belongs;

[0028] The real-time data processing and scheduling module is used for:

[0029] Real-time tasks are obtained sequentially according to the real-time service scheduling plan; the real-time service scheduling plan is a pre-defined plan.

[0030] Determine whether the real-time task meets the execution requirements; if the satellite data required by the real-time task has been received completely, determine that the real-time task meets the execution requirements; if the satellite data required by the real-time task has not been received completely, determine that the real-time task does not meet the execution requirements.

[0031] The data processing module is used for:

[0032] When the real-time task meets the execution requirements, the real-time task is executed, and the task execution result is obtained;

[0033] When multiple real-time tasks meet the execution requirements, determine whether the execution resources can execute all the real-time tasks that meet the execution requirements in parallel at the same time;

[0034] When the execution resources are capable of executing all the real-time tasks that meet the execution requirements in parallel, all the real-time tasks that meet the execution requirements are executed in parallel, and the task execution results are obtained respectively.

[0035] When the execution resources cannot execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed sequentially according to the load priority order, and the task execution results are obtained respectively; the load priority order is divided according to the satellite's load.

[0036] The results of the task execution will be shared with users worldwide.

[0037] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-star multi-task parallel scheduling processing method described above.

[0038] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-star, multi-task parallel scheduling processing method described above.

[0039] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the multi-star, multi-task parallel scheduling processing method described above.

[0040] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0041] This application provides a multi-satellite, multi-task parallel scheduling processing method, apparatus, device, medium, and product. The method acquires satellite data according to regional priority; the regional priority is determined based on the region to which the satellite belongs; real-time tasks are acquired sequentially according to a pre-defined real-time service scheduling plan; it determines whether each real-time task meets execution requirements; when all the satellite data required for a real-time task has been received, the real-time task is deemed to meet execution requirements; when the required satellite data has not been received, the real-time task is deemed not to meet execution requirements; when the real-time task meets execution requirements, the real-time task is executed. The process involves obtaining task execution results. When multiple real-time tasks meet the execution requirements, it is determined whether the execution resources can simultaneously execute all the real-time tasks that meet the requirements. If the execution resources can simultaneously execute all the real-time tasks that meet the requirements, they are executed in parallel, and task execution results are obtained respectively. If the execution resources cannot simultaneously execute all the real-time tasks that meet the requirements, they are executed sequentially according to payload priority, and task execution results are obtained respectively. The payload priority order is determined based on the satellite's payload. The task execution results are shared with global users. Because this application acquires satellite data sequentially according to regional priority, and processes real-time tasks according to payload priority when execution resources cannot simultaneously execute all tasks, each task runs automatically and efficiently at the correct time and location. Furthermore, the priority order is determined according to the payload, avoiding contention between different payloads for tasks at the same time, effectively improving data processing speed. This application can achieve low-orbit satellite data preprocessing and global sharing in a short time. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1This is an application environment diagram of a multi-star, multi-task parallel scheduling and processing method according to an embodiment of this application;

[0044] Figure 2 A flowchart illustrating a multi-star, multi-task parallel scheduling method provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the data acquisition process of a data acquisition module provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the data scheduling process of a real-time data processing scheduling module provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the data processing flow of a data processing module provided in an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the integrated scheduling model and driver module provided in one embodiment of this application;

[0049] Figure 7 An automated service scheduling flowchart provided in one embodiment of this application;

[0050] Figure 8 This is a schematic diagram of the service scheduling process of a parallelized scheduling subsystem provided in an embodiment of this application;

[0051] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] Although the data received by a single station is limited to a regional scope by the real-time reception visible arc and reception time, Dbnet can standardize and integrate regional data from multiple stations around the world to obtain near-global, high-timeliness data, which can then be distributed to global users through appropriate network communication methods.

[0054] The purpose of this application is to provide a method, apparatus, equipment, medium, and product for multi-satellite multi-task parallel scheduling and processing, which can complete low-orbit satellite data preprocessing and global sharing within 30 minutes.

[0055] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] This application provides a multi-star, multi-task parallel scheduling processing method, which can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send the direct-received satellite data to be processed to server 104. After receiving the direct-received satellite data, server 104 sequentially judges whether the tasks meet the execution requirements according to the priority order, obtaining the judgment result; the task is the task of processing the direct-received data; the tasks whose judgment results meet the execution requirements are executed in parallel to obtain the task execution result; the task execution result is shared with global users.

[0057] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0058] In one exemplary embodiment, such as Figure 2 As shown, a multi-star, multi-task parallel scheduling processing method is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S1 to S4. Wherein:

[0059] S1. Obtain satellite data according to regional priority order; the regional priority order is divided according to the region to which the satellite belongs.

[0060] The regional priority order is as follows: domestic satellites have the first priority, and foreign satellites have the second priority. In this embodiment, data is acquired from three domestic satellites and two foreign satellites. Since there is no time conflict in acquiring data between domestic satellites and data between foreign satellites, this embodiment only needs to distinguish the regional priority of domestic and foreign satellites.

[0061] In this embodiment, the satellites currently participating in parallel scheduling and processing on the cloud include: domestic satellites: Fengyun series polar-orbiting meteorological satellites (FY3D, FY3E, FY3G) and foreign satellites (NOAA20, Metop-C). This embodiment adopts a data-driven hierarchical approach to achieve service integration through immediate preprocessing after data reception.

[0062] Real-time acquisition of direct reception data, reception status information, and external data required for processing from Fengyun series satellites (FY3D, FY3E, FY3G) and foreign satellites (NOAA20, Metop-C). External data includes numerical forecast data, buoy data, atmospheric model deterministic forecast products, ERA5 hourly stratified data in the analysis field, etc.

[0063] In this embodiment, when acquiring satellite data, the direct reception data and configuration data of the satellite are first acquired according to the regional priority order; the direct reception data and the configuration data are then saved to the resource pool; and the direct reception data is then parsed according to the configuration data to obtain the satellite data.

[0064] S2. Obtain real-time tasks sequentially according to the real-time service scheduling plan; the real-time service scheduling plan is a pre-defined plan.

[0065] S3. Determine whether the real-time task meets the execution requirements; if the satellite data required by the real-time task has been completely received, determine that the real-time task meets the execution requirements; if the satellite data required by the real-time task has not been completely received, determine that the real-time task does not meet the execution requirements. Simultaneously, track and record the execution status of each real-time task.

[0066] S4. When the real-time task meets the execution requirements, the real-time task is executed to obtain the task execution result.

[0067] S5. When multiple real-time tasks meet the execution requirements, determine whether the execution resources can execute all the real-time tasks that meet the execution requirements in parallel at the same time.

[0068] S6. When the execution resources are able to execute all the real-time tasks that meet the execution requirements in parallel, execute all the real-time tasks that meet the execution requirements in parallel and obtain the task execution results respectively.

[0069] S7. When the execution resources cannot execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed in sequence according to the load priority order, and the task execution results are obtained respectively; the load priority order is divided according to the satellite load; the load priority order is: FY3G satellite, FY3E satellite, FY3D satellite, NOAA20 satellite and Metop-C satellite.

[0070] S8. Share the task execution results with users worldwide.

[0071] The real-time tasks in this embodiment are mainly: data preprocessing and format conversion tasks;

[0072] The satellite data is preprocessed to obtain preprocessed data; the preprocessing includes one or more of the following: data quality inspection, data unpacking, scientific data positioning, and scientific data calibration; then the preprocessed data is converted into a format to obtain the task execution result.

[0073] Specifically:

[0074] Fengyun-3 satellite data processing: The L0 level data generated from the data of each payload of FY3D, FY3E, and FY3G are preprocessed. The preprocessing includes data quality inspection, scientific data positioning, and scientific data calibration, etc., to generate Level 1 data.

[0075] Metop-C data processing: Preprocessing of Level 0 data generated from AMAUA data, MHS data, AVHRR data, and HIRS data. Preprocessing includes data unpacking, scientific data localization, and scientific data calibration to generate Level 1 data.

[0076] NOAA20 data processing: The L0 level data generated from VIIRS, ATMS, and CRIS data are preprocessed, including data unpacking, scientific data localization, and scientific data calibration, to generate Level 1 data.

[0077] Product format conversion: Level 1 products output from data processing modules such as Metop-C, NOAA20, and Fengyun-3 are converted into specified formats that meet DBNet requirements, based on their specifications. This example can generate formats such as HDF and BUFR.

[0078] In this embodiment, after the task execution results are shared with global users, the following steps are also included: summarizing the completion of each satellite in the order of day / week / ten-day period / month.

[0079] Based on the tracking and recording results of the execution status of each task mentioned above, the daily / weekly / ten-day / monthly completion processing of the same instrument and the same product is executed sequentially, ensuring the input constraint relationship of product processing.

[0080] Based on the same inventive concept, this application also provides a multi-satellite multi-task parallel scheduling and processing apparatus. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the multi-satellite multi-task parallel scheduling and processing apparatus provided below can be found in the limitations of the multi-satellite multi-task parallel scheduling and processing method described above, and will not be repeated here.

[0081] The multi-satellite multi-task parallel scheduling and processing device includes: a data processing module, a scheduling module, and a real-time data processing and scheduling module.

[0082] The data acquisition module is used to acquire satellite data according to regional priority; the regional priority is determined based on the region to which the satellite belongs. The regional priority order is: domestic satellites are the first priority, and foreign satellites are the second priority.

[0083] The data acquisition module acquires in real time direct data, reception status information, and external data required for processing from Fengyun satellites (FY3D, FY3E, FY3G) and foreign satellites (NOAA20, Metop-C). External data includes numerical weather prediction data, buoy data, atmospheric model deterministic forecast products, ERA5 hourly stratified data in the analysis field, etc. The data acquisition process is as follows: Figure 3 As shown.

[0084] The real-time data processing and scheduling module is used for:

[0085] Real-time tasks are obtained sequentially according to the real-time service scheduling plan; the real-time service scheduling plan is a pre-defined plan.

[0086] Determine whether the real-time task meets the execution requirements; if the satellite data required by the real-time task has been received completely, determine that the real-time task meets the execution requirements; if the satellite data required by the real-time task has not been received completely, determine that the real-time task does not meet the execution requirements.

[0087] The real-time data processing and scheduling module completes the concurrent scheduling and operation of automatic control and monitoring satellite data processing tasks, and adjusts the scheduling tasks according to predetermined strategies in response to changes in the timetable. The functional module can acquire and determine unprocessed records in the cache plan, judge whether the processing conditions are met (i.e., the current orbit has received complete data and is ready for processing), and promptly process the real-time tasks that meet the conditions (i.e., data unpacking, preprocessing, etc.). The data scheduling process is as follows: Figure 4 As shown.

[0088] The data processing module is used for:

[0089] When the real-time task meets the execution requirements, the real-time task is executed, and the task execution result is obtained.

[0090] When multiple real-time tasks meet the execution requirements, it is determined whether the execution resources can execute all the real-time tasks that meet the execution requirements in parallel at the same time.

[0091] When the execution resources are capable of executing all the real-time tasks that meet the execution requirements in parallel, all the real-time tasks that meet the execution requirements are executed in parallel, and the task execution results are obtained respectively.

[0092] When the execution resources cannot execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed in sequence according to the load priority order, and the task execution results are obtained respectively; the load priority order is divided according to the satellite's load.

[0093] The results of the task execution will be shared with users worldwide.

[0094] The data processing module can call various software packages to complete data processing and quality inspection operations according to business scheduling. The processed data products must meet the requirements of the DBNet technical manual, and format conversion is required when necessary. Products meeting the format requirements (HDF, BUFR, etc.) must be produced within a specified time. The functional modules include: Fengyun-3 satellite data processing, Metop-C data processing, NOAA20 data processing, and product format conversion processing. Detailed functions are shown below:

[0095] Fengyun-3 satellite data processing: The L0 level data generated from the data of each payload of FY3D, FY3E, and FY3G are preprocessed. The preprocessing includes data quality inspection, scientific data positioning, and scientific data calibration, etc., to generate Level 1 data.

[0096] Metop-C data processing: Preprocessing of Level 0 data generated from AMAUA data, MHS data, AVHRR data, and HIRS data. Preprocessing includes data unpacking, scientific data localization, and scientific data calibration to generate Level 1 data.

[0097] NOAA20 data processing: The L0 level data generated from VIIRS, ATMS, and CRIS data are preprocessed, including data unpacking, scientific data localization, and scientific data calibration, to generate Level 1 data.

[0098] Product Format Conversion: This module converts Level 1 products output from data processing modules such as Metop-C, NOAA20, and Fengyun-3 into specified formats that meet DBNet requirements. This module can generate formats such as HDF and BUFR. The specific data processing flow is as follows: Figure 5 As shown.

[0099] The multi-satellite, multi-task parallel scheduling and processing device provided in this embodiment also includes a timed scheduling module. This module polls the set of scheduled tasks to be processed, scheduling each product's processing job to run automatically and efficiently at the correct time and location, and tracking and recording the running status of each job at each time interval. The list of scheduled tasks to be processed is an ordered list. The timed scheduling process polls the list according to the scheduled task times, and when the most recently scheduled task is polled, a timeout is set for its start. The timed scheduling module executes the daily / weekly / ten-day / monthly completion processing of the same instrument and the same product sequentially, ensuring the input constraints of the product processing.

[0100] Furthermore, based on the combined design, the multi-satellite, multi-task parallel scheduling processing device provided in this embodiment may also include an integrated scheduling model and a driver module. The integrated scheduling model and driver module parse the real-time / timed business scheduling plan (i.e., parse the scheduling plan configuration, obtain the job template, and generate job execution instances based on track information, data information, etc.), rationally utilize resources, and schedule each product processing job to run automatically and efficiently at the correct time and location, while tracking and recording the running status of each job at each time interval. When generating plan parameters, the functional module parses the real-time business scheduling plan for the day following the parameter date, obtaining the product processing flow plan start time, task type, etc., for all tasks; when updating plan parameters, the functional module can parse the real-time business scheduling plan after the parameter date update time. The functional module can perform real-time business scheduling control for different loads separately. Real-time business scheduling manages business scheduling based on the process plan, task priority configuration, task type configuration, etc., avoiding competition for tasks between different loads at the same time; it can also automatically schedule and start the corresponding product processing flow execution process according to the product processing scheduling plan start time of each task. Please refer to [link / reference] for details. Figure 6 .

[0101] As an optional implementation, the device described above can also implement a multi-satellite, multi-task parallel scheduling and processing method based on cloud data. It adopts an automated scheduling operation mode, providing flexible and convenient data processing capabilities. Through human-computer interaction, it customizes processing and scheduling plans for single jobs, multiple jobs, and multi-day data products for users, generating processing plan task sheets. Based on the task sheets, it locates and confirms the acquisition path and storage location of the required data, prepares the data and environment for processing, performs parallel processing scheduling, checks the rationality of the processed products and data, records processing results, production plans, operating status, and other information, and returns these results to the system and user. The automated scheduling business process diagram is shown below. Figure 7 As shown.

[0102] The specific implementation method of automated scheduling in this embodiment is as follows:

[0103] The data acquisition module acquires in real time direct data, reception status information, and external data required for processing from Fengyun satellites (FY3D, FY3E, FY3G) and foreign satellites (NOAA20, Metop-C).

[0104] The timed scheduling module obtains scheduling information based on execution parameters (execution parameter information includes program commands, program execution parameters, resources required for program operation, program input and output, etc.), and generates each processing flow plan XML file by combining the data preprocessing and product processing flow configuration XML template files of each instrument.

[0105] The real-time data processing and scheduling module performs job scheduling control. Its main task is to automatically control and monitor the scheduling and operation of satellite data processing jobs. It also receives manual intervention commands from operators to adjust scheduling tasks according to predetermined strategies in response to changes in the timetable. Another important task is receiving operator commands and, based on the command parameters and corresponding data information, driving the delayed scheduling processing module or initiating the single-job redo processing module; loading the plan (the plan file here is the actual XML plan file generated based on the XML template and actual task time information), parsing it, and obtaining the job flow file path, job directory, and acknowledgment flag. It parses the job information to be submitted from the job flow file and creates the corresponding job script, submits the job script, and monitors the job running status and the running host status (the parsing includes the job flow file path, job directory, acknowledgment flag, and other information required for operation, all of which are placed in the job running script); directories need to be created selectively; if a directory for the current date exists, no judgment is needed; otherwise, the directory needs to be created. The naming rules for job script information and the temporary files used need to be handled uniformly; and corresponding acknowledgments are made based on different scheduling flags given in the scheduling command, indicating whether to read / write shared memory segments or send messages.

[0106] The integrated scheduling model and driver module are responsible for the execution management of the entire job workflow. Its core function is to parse the entire job flow file, analyze job node types, and handle aspects such as job naming rules and output file paths. Monitoring the job running status and the running host status requires corresponding controls, such as real-time detection, probing tests, exception handling, and timeout handling.

[0107] Job scheduling control manages the job flow in the following ways:

[0108] Job workflow execution management. This module parses scheduling commands to obtain the job flow file path, job directory, and receipt flag. It extracts the job information to be submitted from the job flow file, creates the corresponding job script, submits the job script, and monitors the job execution status and the status of the running host. The core of the job workflow execution management module is parsing the entire job flow file and analyzing the job node type. Monitoring the job execution status and the running host status requires appropriate controls, such as real-time detection, probing tests, exception handling, and timeout handling.

[0109] Job execution breakpoint recovery. Satellite data processing is a highly complex real-time system with stringent time requirements and a strong emphasis on timeliness. Therefore, it is essential that if a brief system outage or job failure occurs during workflow processing, the workflow can be restarted from the failed job once the system recovers. Specifically, the job execution breakpoint recovery module must support the re-running of failed jobs and restore the workflow; already executed parts of the workflow do not need to be re-run. The module utilizes the Flow_checkpoint file, which monitors the workflow's execution status, to set checkpoints according to a configurable checkpoint strategy. It periodically extracts the working status of each running workflow in the system, including the current job command line, job submission time, job execution time, submission queue, and running nodes. This information is recorded in the Flow_Checkpoint file, which contains information about failed jobs during workflow execution. When the module detects a file that needs to be redone, it submits it to the workflow executor for re-execution. During job execution, the job breakpoint recovery module periodically extracts checkpoints based on the configurable Checkpoint strategy. If the job execution is interrupted due to operating system issues or business scheduling downtime, the job will restart from the last checkpoint once normal operation is restored. Specifically, the job breakpoint recovery module needs to receive commands from the job executor to restart the faulty job and resume job flow execution. Then, it performs breakpoint recovery processing, parses command parameters, locates the job flow, and parses the job flow execution information in the Flow_CheckPoint file to identify jobs that need to be re-executed. Finally, it resubmits the job execution based on the characteristic information and notifies the job executor to reactivate and resume job flow execution, thus avoiding wasting already executed jobs and ensuring that jobs can be processed in real time.

[0110] The business scheduling system employs file-triggered and time-triggered scheduling mechanisms to manage scheduling plans, dispatch jobs, and initiate processes. It receives and responds to scheduling plan management commands, providing operators with a means to manually intervene in scheduling plans. It receives and processes manual scheduling commands, including resubmitting a product job or resending an internal product; the real-time business scheduling module then initiates the single-job redo processing module for processing. Upon receiving redo segment or whole-track data job processes, the parallel scheduling subsystem's business scheduling process is as follows: Figure 8 As shown.

[0111] This embodiment includes: a data acquisition module, a timed scheduling module, a real-time data processing scheduling module, an integrated scheduling model and driver module, and a data processing module. The data acquisition process acquires direct-received data and reception status information from Fengyun satellites (FY3D, FY3E, FY3G) and foreign satellites (NOAA20, Metop-C), as well as external data required for processing, in real time. The timed scheduling module polls the set of pending tasks in the timed schedule, scheduling each product processing job to run automatically and efficiently at the correct time and location, and tracks and records the running status of each job at each time interval. The real-time data processing scheduling module performs concurrent scheduling tasks for automatically controlling and monitoring satellite data processing operations, adjusting scheduling tasks according to predetermined strategies in response to changes in the timetable. The integrated scheduling model and driver module analyzes the real-time / timed service scheduling plan, rationally utilizes resources, schedules each product processing job to run automatically and efficiently at the correct time and location, and tracks and records the running status of each job at each time interval. The data processing module can call various software packages to complete data processing and quality inspection operations according to business scheduling. The products obtained from data processing must meet the requirements of the DBNet technical manual, and format conversion is required when necessary. Products meeting the format requirements (HDF, BUFR, etc.) must be produced within a specified time. This disclosed technical solution adopts data-driven hierarchical scheduling, establishes a unified scheduling and control technology system for multiple satellites and multiple tasks, and achieves low-orbit satellite data preprocessing and global sharing within 30 minutes.

[0112] This embodiment addresses the limitations of single-station data reception due to the constraints of real-time reception of visible arc segments and reception time. It achieves data preprocessing and sharing with global users within 30 minutes. Satellites not only receive visible arc segment data but also global data; this method aims to rapidly process visible arc segment data and share it with global users. By using Dbnet to standardize and process regional data from multiple global stations, near-global, high-timeliness data can be obtained.

[0113] The integrated scheduling model and driver module parses real-time / timed business scheduling plans, rationally utilizes resources, and schedules each product processing job to run automatically and efficiently at the correct time and location, while tracking and recording the running status of each job at each time interval. When generating plan parameters, the functional module parses the real-time business scheduling plan of the parameters to obtain the product processing flow plan start time, task type, etc. for all tasks; when updating plan parameters, the functional module can parse the real-time business scheduling plan after the parameter date update time. The functional module can perform real-time business scheduling control for different loads separately, avoiding competition between tasks at the same time for different loads; it can also automatically schedule and start the corresponding product processing flow execution process according to the product processing scheduling plan start time of each task.

[0114] The automated operation scheduling provides flexible and convenient data processing capabilities. Through human-computer interaction, it customizes processing and scheduling plans for single jobs, multiple jobs, and multi-day data products, generating processing plan task sheets. Based on the task sheets, it locates and confirms the acquisition path and storage location of the required data, prepares the data and environment for processing, performs parallel processing scheduling, checks the rationality of the processed products and data, records processing results, production plans, operating status, and other information, and returns these results to the system and the user.

[0115] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 9 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-task parallel scheduling processing method.

[0116] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0117] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0118] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0119] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0122] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-star, multi-task parallel scheduling and processing method, characterized in that, The multi-star, multi-task parallel scheduling processing method includes: Satellite data is acquired according to regional priority; the regional priority is determined based on the region to which the satellite belongs. Real-time tasks are obtained sequentially according to the real-time service scheduling plan; the real-time service scheduling plan is a pre-defined plan. Determine whether the real-time task meets the execution requirements; if the satellite data required by the real-time task has been received completely, determine that the real-time task meets the execution requirements; if the satellite data required by the real-time task has not been received completely, determine that the real-time task does not meet the execution requirements. When the real-time task meets the execution requirements, the real-time task is executed, and the task execution result is obtained; When multiple real-time tasks meet the execution requirements, determine whether the execution resources can execute all the real-time tasks that meet the execution requirements in parallel at the same time; When the execution resources are capable of executing all the real-time tasks that meet the execution requirements in parallel, all the real-time tasks that meet the execution requirements are executed in parallel, and the task execution results are obtained respectively. When the execution resources cannot execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed sequentially according to the load priority order, and the task execution results are obtained respectively; the load priority order is divided according to the satellite's load. The results of the task execution will be shared with users worldwide.

2. The multi-star, multi-task parallel scheduling and processing method according to claim 1, characterized in that, The priority order for the regions is as follows: domestic satellites are the first priority, and foreign satellites are the second priority.

3. The multi-star, multi-task parallel scheduling and processing method according to claim 1, characterized in that, Satellite data is acquired according to regional priority, specifically including: Obtain direct satellite reception data and configuration data according to regional priority; Save the directly received data and the configuration data to the resource pool; The satellite data is obtained by parsing the directly received data based on the configuration data.

4. The multi-star, multi-task parallel scheduling and processing method according to claim 1, characterized in that, The real-time tasks include: data preprocessing and format conversion tasks; Executing the real-time task includes: The satellite data is preprocessed to obtain preprocessed data; the preprocessing includes one or more of the following: data quality inspection, data unpacking, scientific data positioning, and scientific data calibration. The preprocessed data is then converted to a new format to obtain the task execution result.

5. The multi-star, multi-task parallel scheduling and processing method according to claim 1, characterized in that, The payload priority order is: FY3G satellite, FY3E satellite, FY3D satellite, NOAA20 satellite, and Metop-C satellite.

6. The multi-star, multi-task parallel scheduling and processing method according to claim 1, characterized in that, After sharing the task execution results with users worldwide, the process also includes: The work of each satellite will be summarized in the order of day / week / ten-day period / month.

7. A multi-star, multi-task parallel scheduling and processing device, characterized in that, The multi-star, multi-task parallel scheduling and processing device includes: The data acquisition module is used to acquire satellite data according to regional priority order; the regional priority order is divided according to the region to which the satellite belongs; The real-time data processing and scheduling module is used for: Real-time tasks are obtained sequentially according to the real-time service scheduling plan; the real-time service scheduling plan is a pre-defined plan. Determine whether the real-time task meets the execution requirements; if the satellite data required by the real-time task has been received completely, determine that the real-time task meets the execution requirements; if the satellite data required by the real-time task has not been received completely, determine that the real-time task does not meet the execution requirements. The data processing module is used for: When the real-time task meets the execution requirements, the real-time task is executed, and the task execution result is obtained; When multiple real-time tasks meet the execution requirements, determine whether the execution resources can execute all the real-time tasks that meet the execution requirements in parallel at the same time; When the execution resources are capable of executing all the real-time tasks that meet the execution requirements in parallel, all the real-time tasks that meet the execution requirements are executed in parallel, and the task execution results are obtained respectively. When the execution resources cannot execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed sequentially according to the load priority order, and the task execution results are obtained respectively; the load priority order is divided according to the satellite's load. The results of the task execution will be shared with users worldwide.

8. A computer device, comprising: The memory and processor contain a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the multi-star multi-task parallel scheduling processing method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-star multi-task parallel scheduling processing method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-star multi-task parallel scheduling processing method as described in any one of claims 1-6.

Images