Multi-satellite multi-task parallel scheduling processing method, device, equipment, medium and product

Abstract

The invention discloses a multi-satellite multi-task parallel scheduling processing method and device, equipment, a medium and a product, and relates to the technical field of data processing, and the method comprises the steps: obtaining satellite data according to a region priority order; obtaining real-time tasks in sequence according to the real-time service scheduling plan; judging whether the real-time task meets an execution requirement or not; when a plurality of real-time tasks meet the execution requirements, judging whether the execution resources can execute all the real-time tasks meeting the execution requirements in parallel at the same time or not; when the execution resource cannot execute all the real-time tasks meeting the execution requirements in parallel at the same time, the real-time tasks meeting the execution requirements are executed in sequence according to the load priority order, and task execution results are obtained respectively. According to the method, the satellite data are sequentially acquired according to the sequence of the regional priorities, and when the execution resources cannot simultaneously execute all tasks in parallel, the real-time tasks are processed according to the sequence of the load priorities, so that the data processing speed is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multi-satellite multi-task parallel scheduling processing method, device, equipment, medium and product. Background Art

[0002] DBNet (The Direct Broadcast Network) is a real-time low-orbit satellite direct data set (similar to the MPT or HRPT data of Fengyun-3 satellite) that is received and pre-processed by the global participating direct broadcast receiving stations and is available through the global network under the framework of CGMS (Coordination Group for Meteorological Satellites) with the participation of many countries around the world. At present, the European Center for Numerical Weather Forecasts has established a timeliness monitoring system to monitor the data acquisition of AMSU-A, MHS and HIRS (ATOVS), IASI, ATMS, CrIS and other payload data at the direct receiving station. However, it is not possible to obtain high-timeliness data from the world. Summary of the invention

[0003] The purpose of this application is to provide a multi-satellite multi-task parallel scheduling processing method, device, equipment, medium and product, which can complete low-orbit satellite data preprocessing and global sharing in a short time.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a multi-satellite multi-task parallel scheduling processing method, comprising:

[0006] Acquiring satellite data according to a regional priority order; the regional priority order is divided according to the region to which the satellite belongs;

[0007] Acquire real-time tasks in sequence according to a real-time business scheduling plan; the real-time business scheduling plan is a plan that has been formulated;

[0008] Determine whether the real-time task meets the execution requirements; when the satellite data required by the real-time task has been completely received, determine that the real-time task meets the execution requirements; when the satellite data required by the real-time task has not been completely received, determine that the real-time task does not meet the execution requirements;

[0009] When the real-time task meets the execution requirement, the real-time task is executed to obtain the task execution result;

[0010] When there are multiple real-time tasks that meet the execution requirements, determining whether the execution resources can simultaneously and concurrently execute all the real-time tasks that meet the execution requirements;

[0011] When the execution resource is capable of executing all the real-time tasks that meet the execution requirements in parallel at the same time, all the real-time tasks that meet the execution requirements are executed in parallel to obtain task execution results respectively;

[0012] When the execution resources cannot execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed in sequence according to the load priority order to obtain task execution results respectively; the load priority order is divided according to the load of the satellite;

[0013] The task execution results are shared with global users.

[0014] Optionally, the regional priority order is: domestic satellites are the first priority and foreign satellites are the second priority.

[0015] Optionally, satellite data is obtained in order of regional priority, including:

[0016] Obtain satellite direct reception data and configuration data in order of regional priority;

[0017] Saving the directly received data and the configuration data in a resource pool;

[0018] The directly received data is parsed according to the configuration data to obtain satellite data.

[0019] Optionally, the real-time tasks include: data preprocessing and format conversion tasks;

[0020] Executing the real-time task includes:

[0021] Preprocessing the satellite data to obtain preprocessed data; the preprocessing includes: one or more of data quality inspection, data unpacking, scientific data positioning and scientific data calibration;

[0022] The pre-processed data is formatted to obtain a task execution result.

[0023] Optionally, the payload priority order is: FY3G satellite, FY3E satellite, FY3D satellite, NOAA20 satellite and Metop-C satellite.

[0024] Optionally, after sharing the task execution result with global users, the method further includes:

[0025] A summary of each satellite's work is conducted in the order of day / hou / ten-day / month.

[0026] In a second aspect, the present application provides a multi-satellite multi-task parallel scheduling processing device, comprising:

[0027] A data acquisition module, used for acquiring satellite data according to a regional priority order; the regional priority order is divided according to the region to which the satellite belongs;

[0028] Real-time data processing scheduling module, used for:

[0029] Acquire real-time tasks in sequence according to a real-time business scheduling plan; the real-time business scheduling plan is a plan that has been formulated;

[0030] Determine whether the real-time task meets the execution requirements; when the satellite data required by the real-time task has been completely received, determine that the real-time task meets the execution requirements; when the satellite data required by the real-time task has not been completely received, determine that the real-time task does not meet the execution requirements;

[0031] Data processing module for:

[0032] When the real-time task meets the execution requirement, the real-time task is executed to obtain the task execution result;

[0033] When there are multiple real-time tasks that meet the execution requirements, determining whether the execution resources can simultaneously and concurrently execute all the real-time tasks that meet the execution requirements;

[0034] When the execution resource is capable of executing all the real-time tasks that meet the execution requirements in parallel at the same time, all the real-time tasks that meet the execution requirements are executed in parallel to obtain task execution results respectively;

[0035] When the execution resources cannot execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed in sequence according to the load priority order to obtain task execution results respectively; the load priority order is divided according to the load of the satellite;

[0036] The task execution results are shared with global users.

[0037] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the multi-satellite multi-task parallel scheduling processing methods described above.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multi-satellite multi-task parallel scheduling processing methods described above.

[0039] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the multi-satellite multi-task parallel scheduling processing methods described above.

[0040] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0041] The present application provides a multi-satellite multi-task parallel scheduling processing method, device, equipment, medium and product, wherein the method obtains satellite data in order of regional priority; the regional priority order is divided according to the region to which the satellite belongs; real-time tasks are obtained in sequence according to a real-time business scheduling plan; the real-time business scheduling plan is a plan that has been formulated; it is determined whether the real-time task meets the execution requirements; when the satellite data required by the real-time task has been completely received, it is determined that the real-time task meets the execution requirements; when the satellite data required by the real-time task has not been completely received, it is determined that the real-time task does not meet the execution requirements; when the real-time task meets the execution requirements, the real-time task is executed , get the task execution result; when there are multiple real-time tasks that meet the execution requirements, determine whether the execution resources can execute all the real-time tasks that meet the execution requirements at the same time in parallel; when the execution resources can execute all the real-time tasks that meet the execution requirements at the same time in parallel, execute all the real-time tasks that meet the execution requirements in parallel, and get the task execution results respectively; when the execution resources cannot execute all the real-time tasks that meet the execution requirements at the same time in parallel, execute the real-time tasks that meet the execution requirements in sequence according to the load priority order, and get the task execution results respectively; the load priority order is divided according to the satellite's load; share the task execution results with global users. Since the present application acquires satellite data in sequence according to the order of regional priority, and when the execution resources cannot execute all tasks at the same time in parallel, the real-time tasks are processed in the order of load priority, so that each task runs automatically and efficiently at the right time and the right location, and the priority order is divided according to the load, the competition for tasks at the same time between different loads is avoided, and the speed of data processing is effectively improved. The present application can complete the preprocessing and global sharing of low-orbit satellite data in a short time. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1This is an application environment diagram of a multi-satellite multi-task parallel scheduling processing method in an embodiment of the present application;

[0044] Figure 2 A flowchart of a multi-satellite multi-task parallel scheduling processing method provided in one embodiment of the present application;

[0045] Figure 3 A schematic diagram of a data collection process of a data collection module provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of a data scheduling process of a real-time data processing scheduling module provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of a data processing flow of a data processing module provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of an integrated scheduling model and a driving module flow chart provided in an embodiment of the present application;

[0049] Figure 7 An automated service scheduling flow chart provided for an embodiment of the present application;

[0050] Figure 8 A schematic diagram of a business scheduling process of a parallel scheduling subsystem provided in an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] Although the data received by a single station is only regional due to the limitations of the real-time reception visible arc and reception time, by integrating and processing the regional data of multiple stations around the world in a standardized format through Dbnet, nearly global high-time data can be obtained and distributed to global users through appropriate network communication means.

[0054] The purpose of this application is to provide a multi-satellite multi-task parallel scheduling processing method, device, equipment, medium and product, which can complete low-orbit satellite data preprocessing and global sharing within 30 minutes.

[0055] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0056] The multi-satellite multi-task parallel scheduling processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, or it can be integrated on the server 104, or it can be placed on the cloud or other servers. The terminal 102 can send the direct-received data of the satellite to be processed to the server 104. After the server 104 receives the direct-received data of the satellite to be processed, the server 104 judges whether the task meets the execution requirements in the order of priority, and obtains a judgment result; the task is a task for processing the direct-received data; the tasks whose judgment results meet the execution requirements are executed in parallel to obtain task execution results; and the task execution results are shared with global users.

[0057] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0058] In an exemplary embodiment, Figure 2 As shown, a multi-satellite multi-task parallel scheduling processing method is provided. The method is executed by a computer device, and can be specifically executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps S1 to S4. Among them:

[0059] S1. Acquire satellite data according to a regional priority order; the regional priority order is divided according to the region to which the satellite belongs.

[0060] The regional priority order is: domestic satellites are the first priority, and foreign satellites are the second priority. In this embodiment, data from three domestic satellites and two foreign satellites are acquired. Since there is no conflict in time between data acquisition between domestic satellites and there is no conflict in time between data acquisition between foreign satellites, in this embodiment, only the regional priorities of domestic and foreign satellites need to be distinguished.

[0061] In this embodiment, the satellites currently participating in the parallel scheduling processing on the cloud include: domestic satellites: Fengyun series polar-orbiting meteorological satellites (FY3D, FY3E, FY3G) and foreign satellites (NOAA20, Metop-C). This embodiment adopts data-driven classification to achieve business integration of pre-processing immediately after data is received.

[0062] Real-time acquisition of direct data from Fengyun series satellites (FY3D, FY3E, FY3G) and foreign satellites (NOAA20, Metop-C), receiving status information, and external data required for processing (external data includes numerical forecast data, buoy data, atmospheric model deterministic forecast products, ERA5 hourly stratified data in the analysis field, etc.).

[0063] In this embodiment, when acquiring satellite data, firstly, the direct reception data and configuration data of the satellite are acquired in the order of regional priority; and the direct reception data and the configuration data are saved in a resource pool; and then the direct reception data is parsed according to the configuration data to obtain the satellite data.

[0064] S2. Acquire real-time tasks in sequence according to a real-time business scheduling plan; the real-time business scheduling plan is a plan that has been formulated.

[0065] S3, judging whether the real-time task meets the execution requirements; when the satellite data required by the real-time task has been completely received, determining that the real-time task meets the execution requirements; when the satellite data required by the real-time task has not been completely received, determining that the real-time task does not meet the execution requirements. At the same time, tracking and recording the execution status of each real-time task.

[0066] S4. When the real-time task meets the execution requirement, the real-time task is executed to obtain a task execution result.

[0067] S5. When there are multiple real-time tasks that meet the execution requirements, determine whether the execution resources can execute all the real-time tasks that meet the execution requirements in parallel at the same time.

[0068] S6. When the execution resource is capable of executing all the real-time tasks that meet the execution requirements in parallel at the same time, all the real-time tasks that meet the execution requirements are executed in parallel to obtain task execution results respectively.

[0069] S7. When the execution resources cannot execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed in sequence according to the load priority order to obtain the task execution results respectively; the load priority order is divided according to the payload of the satellite; the load priority order is: FY3G satellite, FY3E satellite, FY3D satellite, NOAA20 satellite and Metop-C satellite.

[0070] S8. Share the task execution result with global users.

[0071] The real-time tasks in this embodiment are mainly: data preprocessing and format conversion tasks;

[0072] The satellite data is preprocessed to obtain preprocessed data; the preprocessing includes: one or more of data quality inspection, data unpacking, scientific data positioning and scientific data calibration; and then the preprocessed data is format converted to obtain a task execution result.

[0073] Specifically:

[0074] FY-3 satellite data processing: The L0 level data generated by the data of each payload of FY3D, FY3E, and FY3G are pre-processed. The pre-processing includes data quality inspection, scientific data positioning, scientific data calibration and other processes to generate level 1 data.

[0075] Metop-C data processing: Preprocess the L0 level data generated by AMAUA data, MHS data, AVHRR data, and HIRS data. The preprocessing includes data unpacking, scientific data positioning, scientific data calibration, etc. to generate level 1 data.

[0076] NOAA20 data processing: Preprocess the L0 level data generated by VIIRS data, ATMS data, and CRIS data. The preprocessing includes data unpacking, scientific data positioning, scientific data calibration, etc. to generate level 1 data.

[0077] Product format conversion: convert the level 1 products output by data processing modules such as Metop-C, NOAA20, and Fengyun-3 into a specified format that meets the requirements of DBNet according to the requirements. This embodiment can generate formats such as HDF and BUFR.

[0078] In this embodiment, after the task execution results are shared with global users, the process further includes: performing a work summary for each satellite in the order of day / month / ten-day period / month.

[0079] According to the tracking and recording results of the execution status of each task mentioned above, the daily / weekly / ten-day / monthly closing processing of the same instrument and the same product is executed in sequence, ensuring the input constraint relationship of the product processing.

[0080] Based on the same inventive concept, the embodiment of the present application also provides a multi-satellite multi-task parallel scheduling processing device. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multi-satellite multi-task parallel scheduling processing device embodiments provided below can refer to the limitations of the multi-satellite multi-task parallel scheduling processing method above, and will not be repeated here.

[0081] The multi-satellite multi-task parallel scheduling processing device comprises: a data processing module, a scheduling module and a real-time data processing scheduling module.

[0082] The data acquisition module is used to obtain satellite data according to the regional priority order; the regional priority order is divided according to the region to which the satellite belongs. The regional priority order is: domestic satellites are the first priority, and foreign satellites are the second priority.

[0083] The data acquisition module acquires direct data from Fengyun satellites (FY3D, FY3E, FY3G) and foreign satellites (NOAA20, Metop-C) in real time, receives status information, and processes external data required (external data includes numerical forecast data, buoy data, atmospheric model deterministic forecast products, ERA5 hourly analysis field stratification data, etc.). The data acquisition process is as follows: Figure 3 shown.

[0084] Real-time data processing scheduling module, used for:

[0085] Acquire real-time tasks in sequence according to a real-time business scheduling plan; the real-time business scheduling plan is a plan that has been formulated;

[0086] Determine whether the real-time task meets the execution requirements; when the satellite data required by the real-time task has been completely received, determine that the real-time task meets the execution requirements; when the satellite data required by the real-time task has not been completely received, determine that the real-time task does not meet the execution requirements.

[0087] The real-time data processing scheduling module completes the concurrent scheduling operation tasks of automatically controlling and monitoring the satellite data processing operation process, and adjusts the scheduling tasks according to the changes in the schedule according to the predetermined strategy. The functional module can obtain and determine the unprocessed records in the cache plan, determine whether the processing conditions are met (that is, the current orbit reception is complete and data processing can be performed), and process the real-time tasks that meet the conditions in a timely manner (that is, data unpacking, preprocessing and other operation processes). The data scheduling process is as follows Figure 4 shown.

[0088] Data processing module for:

[0089] When the real-time task meets the execution requirement, the real-time task is executed to obtain a task execution result.

[0090] When there are multiple real-time tasks that meet the execution requirements, it is determined whether the execution resources can simultaneously and concurrently execute all the real-time tasks that meet the execution requirements.

[0091] When the execution resource is capable of executing all the real-time tasks that meet the execution requirements in parallel at the same time, all the real-time tasks that meet the execution requirements are executed in parallel to obtain task execution results respectively.

[0092] When the execution resources are unable to execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed in sequence according to the load priority order to obtain the task execution results respectively; the load priority order is divided according to the satellite payload.

[0093] The task execution results are shared with global users.

[0094] The data processing module can call various software packages to complete data processing and quality inspection operations according to business scheduling. The products obtained by data processing must meet the requirements of the DBNet technical manual. If necessary, format conversion must be performed to process products that meet the format requirements (HDF, BUFR, etc.) within the specified time. The functional modules include: Fengyun-3 satellite data processing, Metop-C data processing, NOAA20 data processing, product format conversion processing, and the detailed functions are as follows:

[0095] FY-3 satellite data processing: The L0 level data generated by the data of each payload of FY3D, FY3E, and FY3G are pre-processed. The pre-processing includes data quality inspection, scientific data positioning, scientific data calibration and other processes to generate level 1 data.

[0096] Metop-C data processing: Preprocess the L0 level data generated by AMAUA data, MHS data, AVHRR data, and HIRS data. The preprocessing includes data unpacking, scientific data positioning, scientific data calibration, etc. to generate level 1 data.

[0097] NOAA20 data processing: Preprocess the L0 level data generated by VIIRS data, ATMS data, and CRIS data. The preprocessing includes data unpacking, scientific data positioning, scientific data calibration, etc. to generate level 1 data.

[0098] Product format conversion: Convert the level 1 products output by Metop-C, NOAA20, Fengyun-3 and other data processing modules into a specified format that meets the requirements of DBNet. This module can generate HDF, BUFR and other formats. The specific data processing process is as follows: Figure 5 shown.

[0099] The multi-satellite multi-task parallel scheduling processing device provided in this embodiment also includes a timing scheduling module, which polls the timing plan to-be-processed set, schedules each product processing job to automatically and efficiently run at the correct time and correct location, and tracks and records the running status of each job at each time. Among them, the scheduled task to-be-processed list is an ordered list, and the timing scheduling process polls the to-be-processed list according to the scheduled task plan time, and sets a timeout start when polling the scheduled task that is about to be started. The timing scheduling module executes the daily / hou / ten-day / monthly closing processing of the same instrument and the same product in sequence, ensuring the input constraint relationship of product processing.

[0100] In addition, based on the combined design, the multi-satellite multi-task parallel scheduling processing device provided in this embodiment may also include an integrated scheduling model and a driving module, which parses the real-time / scheduled business scheduling plan (i.e., the process of parsing the scheduling plan configuration, obtaining the job template and generating a job running instance according to orbit information, data information, etc.), reasonably utilizes resources, and schedules each product processing job to automatically and efficiently run at the correct time and location, and tracks and records the running status of each job at each time. When generating the plan parameters, the functional module parses the real-time business scheduling plan on the second day of the parameter date, and obtains the product processing flow plan start time, task type, etc. of all tasks; when updating the plan parameters, the functional module can parse the real-time business scheduling plan after the parameter date update time point. The functional module can perform real-time business scheduling control for different loads respectively, and the real-time business scheduling performs business scheduling management according to the process plan, task priority configuration, task type configuration, etc., to avoid the competition of tasks at the same time between different loads; it can also automatically schedule and start the corresponding product processing flow execution process according to the product processing scheduling plan start time of each task. For details, please refer to Figure 6 .

[0101] As an optional implementation, the device described above can also implement a multi-star multi-task parallel scheduling and processing method based on cloud data, adopt an automated scheduling operation mode, provide flexible and convenient data processing capabilities, and customize single jobs, multiple jobs, and multi-day data products for users through human-computer interaction to perform processing scheduling plans and generate processing plan task sheets. According to the planned task sheet, find and confirm the acquisition path and storage location of the required data, prepare the data and environment for processing, perform parallel processing scheduling, conduct rationality checks on processing products and data, record processing results, production plans, operating status and other information, and return these results to the system and users. The automated scheduling business process diagram is shown below. Figure 7 shown.

[0102] The specific implementation method of the automatic scheduling in this embodiment is:

[0103] The data acquisition module obtains direct data from Fengyun satellites (FY3D, FY3E, FY3G) and foreign satellites (NOAA20, Metop-C) in real time, receiving status information, and external data required for processing.

[0104] The timing scheduling module obtains scheduling information based on execution parameters (execution parameter information includes program commands, program execution parameters, resources required for program operation, program input and output, etc.), combines the data preprocessing of each instrument and the product processing flow configuration xml template file to generate each processing flow plan xml file.

[0105] The real-time data processing scheduling module performs job scheduling control. Its main task is to complete the scheduling and operation tasks of automatically controlling and monitoring the satellite data processing job flow; at the same time, it receives the operator's manual intervention command to adjust the scheduling task according to the predetermined strategy according to the change of the schedule. Another important task is to receive the operator's command, drive the delay scheduling processing module or start the single job redo processing module according to the corresponding data information of the command parameter; load the plan (the plan file here is the actual xml plan file generated according to the xml template and the actual time of the task and other information), parse it, and obtain the job flow file path, job directory and receipt mark. Parse the job information to be submitted from the job flow file and create the corresponding job script, submit the job script, monitor the job running status, and monitor the running host status (the parsing includes the job flow file path, job directory and receipt mark, and also includes other information required for operation, which are put into the job running script together); the directory needs to be created selectively. If there is a directory for the current date, there is no need to judge. If there is no directory, it is necessary to create the directory. The naming rules of the job script information and the temporary files used need to be processed uniformly; make corresponding receipts according to the different scheduling marks given in the scheduling command, whether to read and write shared memory segments or send messages.

[0106] The integrated scheduling model and driver module are responsible for the execution management of the entire job process. The core is to parse the entire job flow file and analyze the job node type, involving the naming rules of the job name, the path of the output file, etc. The job running status and the running host status monitoring require corresponding control, such as real-time detection, trial testing, exception handling, and timeout handling.

[0107] Job scheduling control controls the job flow in the following ways:

[0108] Job flow execution management. Job flow execution management parses the scheduling command, obtains the job flow file path, job directory and receipt mark. Parses the job information that needs to be submitted from the job flow file and creates the corresponding job script, submits the job script, and monitors the job running status and the running host status; the core of the job flow execution management module is to parse the entire job flow file and analyze the job node type. Job running status and running host status monitoring require corresponding control, such as real-time detection, trial testing, exception handling and timeout handling.

[0109] Job operation breakpoint recovery execution. Satellite data processing is a very complex real-time system for job flow processing, and its time requirements are very strict and have strong timeliness. Therefore, it is required that during the operation and processing of the job flow, if the business system is temporarily shut down or the job flow fails, the operation of the job flow can be re-triggered from the failed job after the business system is restored. Specifically, the job operation breakpoint recovery execution module is required to support the re-running of the failed job and restore the operation of the job flow; the part of the job flow that has been running does not need to be re-run. The job operation breakpoint recovery execution module uses the Flow_checkpoint file record that monitors the operation status of the job flow, sets checkpoints according to the configurable Checkpoint strategy, and periodically extracts the working status of each running job flow in the business system, including the current execution job command line, job submission time, job execution time, submission queue, running node and other characteristic information; and records it in the Flow_Checkpoint file. This file records the operation information of failed jobs during the operation of the job flow. The job operation breakpoint recovery execution module detects the files that need to be redone and submits them to the job flow executor for re-execution. During the execution of the job process, the job operation breakpoint recovery execution module will periodically extract checkpoints according to the configurable Checkpoint policy settings. If the job process execution is interrupted due to operating system problems or business scheduling downtime, when it returns to normal, the job process will restart the execution of the interrupted job from the last checkpoint. Specifically, the job operation breakpoint recovery execution module needs to receive the command sent by the job process executor to restart the failed job and restore the job flow operation, and then perform breakpoint recovery processing, parse the command parameters, locate the job flow, and parse the job flow operation information of the Flow_CheckPoint file to locate the job that needs to be re-executed. Finally, resubmit the job execution according to the feature information, and notify the job process executor to reactivate and restore the execution of the job flow without wasting the executed job, thereby ensuring that the job can be processed in real time.

[0110] Business scheduling uses scheduling mechanisms such as file triggering and time triggering to achieve scheduling plan management, job distribution and process startup; business scheduling receives and responds to scheduling plan management commands, providing operators with a means of manual intervention in scheduling plans; receives and processes manual scheduling commands, including resubmission of a product job or resend of internal products, etc. The real-time business scheduling module starts the single job redo processing module for processing; receives redo segment or whole track data job processes, etc., and the parallel scheduling subsystem business scheduling process such as Figure 8 shown.

[0111] This embodiment includes: a data acquisition module, a timing scheduling module, a real-time data processing scheduling module, an integrated scheduling model and a driving module, and a data processing module. The data acquisition process acquires the direct data and receiving status information of Fengyun satellites (FY3D, FY3E, FY3G) and foreign satellites (NOAA20, Metop-C) in real time, as well as the external data required for processing. The timing scheduling module polls the timing plan to be processed set, schedules each product processing operation to run automatically and efficiently at the right time and the right location, and tracks and records the running status of each operation at each time. The real-time data processing scheduling module completes the concurrent scheduling operation tasks of automatically controlling and monitoring the satellite data processing operation process, and adjusts the scheduling tasks according to the changes in the schedule according to the predetermined strategy. The integrated scheduling model and the driving module parse the real-time / timing business scheduling plan, reasonably utilizes resources, schedules each product processing operation to run automatically and efficiently at the right time and the right location, and tracks and records the running status of each operation at each time. The data processing module can call various software packages to complete data processing and quality inspection operations according to business scheduling. The products obtained by data processing must meet the requirements of the DBNet technical manual. If necessary, format conversion is required to process products that meet the format requirements (HDF, BUFR, etc.) within the specified time. The disclosed technical solution uses data-driven scheduling and classification, establishes a multi-satellite, multi-task unified scheduling and control technology system, and realizes the preprocessing and global sharing of low-orbit satellite data within 30 minutes.

[0112] This embodiment solves the problem that the data received by a single station is limited by the real-time visible arc segment and the receiving time. By adopting, data preprocessing is completed within 30 minutes and shared with global users. Satellites can not only receive visible arc segment data, but also global data. This method is to quickly process visible arc segment data and share it with global users. Through Dbnet, regional data of multiple stations around the world are integrated and processed in a standardized format, and nearly global high-efficiency data can be obtained.

[0113] Among them, the integrated scheduling model and the driving module parse the real-time / scheduled business scheduling plan, make rational use of resources, schedule each product processing job to run automatically and efficiently at the right time and the right location, and track and record the running status of each job at each time. When generating plan parameters, the functional module parses the real-time business scheduling plan of the parameters to obtain the planned start time, task type, etc. of the product processing flow of all tasks; when updating the plan parameters, the functional module can parse the real-time business scheduling plan after the parameter date update time point. The functional module can perform real-time business scheduling control for different loads separately to avoid competition for tasks at the same time between different loads; it can also automatically schedule and start the corresponding product processing flow execution process according to the start time of the product processing scheduling plan of each task.

[0114] The automated operation scheduling provides flexible and convenient data processing capabilities. Through human-computer interaction, it customizes single jobs, multiple jobs, and multiple days of data products for users to perform processing scheduling plans and generate processing plan task sheets. According to the plan task sheet, it finds and confirms the acquisition path and storage location of the required data, prepares data and environment for processing, performs parallel processing scheduling, performs rationality checks on processing products and data, records processing results, production plans, operating status and other information, and returns these results to the system and users.

[0115] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-star multi-task parallel scheduling processing method is implemented.

[0116] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0117] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0118] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0119] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0121] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0122] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0123] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multi-satellite multi-task parallel scheduling processing method, characterized in that: The multi-satellite multi-task parallel scheduling processing method comprises: Acquiring satellite data according to a regional priority order; the regional priority order is divided according to the region to which the satellite belongs; Acquire real-time tasks in sequence according to a real-time business scheduling plan; the real-time business scheduling plan is a plan that has been formulated; Determine whether the real-time task meets the execution requirements; when the satellite data required by the real-time task has been completely received, determine that the real-time task meets the execution requirements; when the satellite data required by the real-time task has not been completely received, determine that the real-time task does not meet the execution requirements; When the real-time task meets the execution requirement, the real-time task is executed to obtain the task execution result; When there are multiple real-time tasks that meet the execution requirements, determining whether the execution resources can simultaneously and concurrently execute all the real-time tasks that meet the execution requirements; When the execution resource is capable of executing all the real-time tasks that meet the execution requirements in parallel at the same time, all the real-time tasks that meet the execution requirements are executed in parallel to obtain task execution results respectively; When the execution resources cannot execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed in sequence according to the load priority order to obtain task execution results respectively; the load priority order is divided according to the load of the satellite; The task execution results are shared with global users.

2. The multi-satellite multi-task parallel scheduling processing method according to claim 1, characterized in that: The regional priority order is: domestic satellites are the first priority and foreign satellites are the second priority.

3. The multi-satellite multi-task parallel scheduling processing method according to claim 1, characterized in that: Satellite data is obtained in order of regional priority, including: Obtain satellite direct reception data and configuration data in order of regional priority; Saving the directly received data and the configuration data in a resource pool; The directly received data is parsed according to the configuration data to obtain satellite data.

4. The multi-satellite multi-task parallel scheduling processing method according to claim 1, characterized in that: The real-time tasks include: data preprocessing and format conversion tasks; Executing the real-time task includes: Preprocessing the satellite data to obtain preprocessed data; the preprocessing includes: one or more of data quality inspection, data unpacking, scientific data positioning and scientific data calibration; The pre-processed data is formatted to obtain a task execution result.

5. The multi-satellite multi-task parallel scheduling processing method according to claim 1, characterized in that: The payload priority order is: FY3G satellite, FY3E satellite, FY3D satellite, NOAA20 satellite and Metop-C satellite.

6. The multi-satellite multi-task parallel scheduling processing method according to claim 1, characterized in that: After sharing the task execution results with global users, it also includes: A summary of each satellite's work is conducted in the order of day / hou / ten-day / month.

7. A multi-satellite multi-task parallel scheduling processing device, characterized in that: The multi-satellite multi-task parallel scheduling processing device comprises: A data acquisition module, used for acquiring satellite data according to a regional priority order; the regional priority order is divided according to the region to which the satellite belongs; Real-time data processing scheduling module, used for: Acquire real-time tasks in sequence according to a real-time business scheduling plan; the real-time business scheduling plan is a plan that has been formulated; Determine whether the real-time task meets the execution requirements; when the satellite data required by the real-time task has been completely received, determine that the real-time task meets the execution requirements; when the satellite data required by the real-time task has not been completely received, determine that the real-time task does not meet the execution requirements; Data processing module for: When the real-time task meets the execution requirement, the real-time task is executed to obtain the task execution result; When there are multiple real-time tasks that meet the execution requirements, determining whether the execution resources can simultaneously and concurrently execute all the real-time tasks that meet the execution requirements; When the execution resource is capable of executing all the real-time tasks that meet the execution requirements in parallel at the same time, all the real-time tasks that meet the execution requirements are executed in parallel to obtain task execution results respectively; When the execution resources cannot execute all the real-time tasks that meet the execution requirements in parallel at the same time, the real-time tasks that meet the execution requirements are executed in sequence according to the load priority order to obtain task execution results respectively; the load priority order is divided according to the satellite's load; The task execution results are shared with global users.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-satellite multi-task parallel scheduling processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-satellite multi-task parallel scheduling processing method described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-satellite multi-task parallel scheduling processing method described in any one of claims 1 to 6 is implemented.

Images