A method and system for optimizing the scheduling of meteorological satellite data throughout the entire process

Abstract

The present invention provides a method and system for optimizing the scheduling of the entire process of meteorological satellite data, including: first, connecting a meteorological data processing module to obtain N entire process task processing nodes, then collecting the historical processing data of these nodes, identifying key nodes, and formulating collaborative scheduling constraints. Next, according to the task requirements, the tasks to be executed are evaluated and matched with priorities, and a scheduling strategy optimization space is generated. Finally, an optimization scheduling model is loaded, multiple task scheduling strategies are generated, and the optimal strategy is found in the optimization space for task scheduling. The present invention can solve the problem that the bottleneck nodes in the data processing flow in the prior art are not effectively identified and optimized, resulting in low overall processing efficiency and low resource utilization, which further affects the accuracy and timeliness of the generation and processing of meteorological satellite observation data to meteorological forecast support information elements such as "maps, fields, states, and labels", improves the overall processing efficiency, optimizes resource allocation, and avoids resource waste.

Description

Technical Field

[0001] The present invention relates to the technical field of meteorological satellite data processing operation scheduling, and in particular to a method and system for optimizing and scheduling the entire process of meteorological satellite data. Background Art

[0002] With the advancement of science and technology and social development, meteorological satellites are playing an increasingly important role in weather forecasting, climate research, natural disaster warning, etc. Meteorological satellites are a key means of obtaining data on the earth's atmosphere, clouds, water vapor, etc. These data are crucial to improving the accuracy and timeliness of weather forecasts.

[0003] At present, the existing meteorological data processing and scheduling technology still has certain limitations in practical applications. The existing technology has not effectively identified and optimized the bottleneck nodes in the data processing process, resulting in low overall processing efficiency and low resource utilization, which further affects the accuracy and timeliness of meteorological satellite data product processing. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for optimizing the scheduling of the entire process of meteorological satellite data, so as to solve the problem in the prior art that the bottleneck nodes in the meteorological data processing scheduling process are not effectively identified and optimized, resulting in low overall processing efficiency and low resource utilization, which further affects the accuracy and timeliness of meteorological satellite data product processing.

[0005] In view of the above problems, the present invention provides a method and system for optimizing and scheduling the entire process of meteorological satellite data.

[0006] In a first aspect, the present invention provides a method for optimizing and scheduling the entire process of meteorological satellite data, which is implemented by a system for optimizing and scheduling the entire process of meteorological satellite data, wherein the method for optimizing and scheduling the entire process of meteorological satellite data includes: connecting a meteorological data processing module to obtain N full-process task processing nodes, wherein N is a positive integer; collecting N historical processing data of the N full-process task processing nodes, and identifying key nodes based on the N historical processing data, wherein the key nodes include a first node set whose processing time exceeds a preset time, a second node set whose resource consumption is greater than a preset consumption, and a third node set whose failure rate is greater than a preset failure rate; based on the The first node set, the second node set and the third node set formulate collaborative scheduling constraints; obtain task requirements, obtain multiple tasks to be executed, evaluate the timeliness, priority and processing complexity of the multiple tasks to be executed, and match the task execution priority for each task to be executed according to the evaluation results, wherein the task priorities may be the same; generate a scheduling strategy optimization space according to the collaborative scheduling constraints and the task execution priorities; load an optimization scheduling model, and generate multiple task scheduling strategies according to the task requirements; evaluate the multiple task scheduling strategies, and optimize in the scheduling strategy optimization space according to the evaluation results to obtain the optimal task scheduling strategy, and perform task scheduling according to the optimal task scheduling strategy.

[0007] In a second aspect, the present invention further provides a system for optimizing and scheduling the entire process of meteorological satellite data, which is used to execute a method for optimizing and scheduling the entire process of meteorological satellite data as described in the first aspect, wherein the system for optimizing and scheduling the entire process of meteorological satellite data includes: a node acquisition module, which is used to connect to the meteorological data processing module to obtain N full-process task processing nodes, where N is a positive integer; a key node identification module, which is used to collect N historical processing data of the N full-process task processing nodes, and identify key nodes based on the N historical processing data, wherein the key nodes include a first node set whose processing time exceeds a preset time, a second node set whose resource consumption is greater than a preset consumption, and a third node set whose failure rate is greater than a preset failure rate; a collaborative constraint formulation module, which is used to identify key nodes based on the first node set, the The second node set and the third node set formulate collaborative scheduling constraints; a priority acquisition module is used to obtain task requirements, obtain multiple tasks to be executed, evaluate the timeliness, priority and processing complexity of the multiple tasks to be executed, and match the task execution priority for each task to be executed according to the evaluation results, wherein the task priorities may be the same; an optimization space generation module is used to generate a scheduling strategy optimization space according to the collaborative scheduling constraints and the task execution priorities; a scheduling strategy generation module is used to load an optimization scheduling model and generate multiple task scheduling strategies according to the task requirements; an optimization module is used to evaluate the multiple task scheduling strategies, perform optimization in the scheduling strategy optimization space according to the evaluation results, obtain the optimal task scheduling strategy, and perform task scheduling according to the optimal task scheduling strategy.

[0008] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0009] By connecting the meteorological data processing module, N full-process task processing nodes are obtained, where N is a positive integer; N historical processing data of the N full-process task processing nodes are collected, and key nodes are identified based on the N historical processing data, wherein the key nodes include a first node set whose processing time exceeds a preset time, a second node set whose resource consumption is greater than a preset consumption, and a third node set whose failure rate is greater than a preset failure rate; collaborative scheduling constraints are formulated based on the first node set, the second node set, and the third node set; task requirements are obtained to obtain multiple tasks to be executed, the timeliness, priority, and processing complexity of the multiple tasks to be executed are evaluated, and the task execution priority is matched for each task to be executed based on the evaluation results , wherein the task priorities may be the same; generating a scheduling strategy optimization space according to the collaborative scheduling constraints and the task execution priorities; loading an optimization scheduling model, generating multiple task scheduling strategies according to the task requirements; evaluating the multiple task scheduling strategies, optimizing in the scheduling strategy optimization space according to the evaluation results, obtaining an optimal task scheduling strategy, and scheduling tasks according to the optimal task scheduling strategy, effectively solving the problem in the prior art that the bottleneck nodes in the meteorological data processing scheduling process are not effectively identified and optimized, resulting in low overall processing efficiency and low resource utilization, further affecting the accuracy and timeliness of meteorological satellite data product processing, improving the overall processing efficiency, optimizing resource allocation, and avoiding resource waste.

[0010] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically cited below. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0012] Figure 1 A schematic diagram of a flow chart of a method for optimizing and scheduling the entire process of meteorological satellite data according to the present invention;

[0013] Figure 2 The present invention is a schematic diagram of the structure of a system for optimizing and scheduling the entire process of meteorological satellite data.

[0014] Description of reference numerals:

[0015] Node acquisition module 11, key node identification module 12, collaborative constraint formulation module 13, priority acquisition module 14, optimization space generation module 15, scheduling strategy generation module 16, optimization module 17. DETAILED DESCRIPTION

[0016] The present invention provides a method and system for optimizing and scheduling the entire process of meteorological satellite data, thereby solving the problem in the prior art that the bottleneck nodes in the meteorological data processing and scheduling process are not effectively identified and optimized, resulting in low overall processing efficiency and low resource utilization, which further affects the accuracy and timeliness of meteorological satellite data product processing, improves the overall processing efficiency, optimizes resource allocation, and avoids resource waste.

[0017] Specifically, the method and system for optimizing the scheduling of the entire process of meteorological satellite data of the present invention have the following beneficial effects:

[0018] 1. Improve overall processing efficiency: By identifying and optimizing bottleneck nodes in the data processing process, reducing task waiting time and execution time, the overall processing efficiency can be significantly improved.

[0019] 2. Optimize resource allocation: Dynamically adjust the allocation of computing resources and storage resources according to the actual needs of the task to ensure that high-load tasks have sufficient resource support, while avoiding resource waste and improving resource utilization.

[0020] 3. Enhance system stability: By identifying nodes with high failure rates and formulating corresponding fault-tolerant strategies, such as setting up backup nodes and restart mechanisms, the system's fault tolerance and stability can be improved to ensure the continuity and integrity of data processing tasks.

[0021] 4. Realize intelligent scheduling: Comprehensively consider multiple factors such as task dependencies, timeliness, priority, processing complexity, and resource consumption, and use the optimization scheduling model to generate the optimal task scheduling strategy to achieve intelligent scheduling of the entire process of meteorological satellite data.

[0022] 5. Support dynamic adjustment: Real-time monitoring of node performance data, and dynamic adjustment of scheduling strategies using big data analysis methods to adapt to changing data processing requirements and node performance status, ensuring continuous optimization and adaptability of scheduling strategies.

[0023] Below, the technical solutions in the present invention will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments described herein. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all of them.

[0024] For example, please refer to the attached Figure 1 The present invention provides a method for optimizing and scheduling the entire process of meteorological satellite data, wherein the method for optimizing and scheduling the entire process of meteorological satellite data is applied to a system for optimizing and scheduling the entire process of meteorological satellite data, and the method for optimizing and scheduling the entire process of meteorological satellite data specifically comprises the following steps:

[0025] S1: Connect the meteorological data processing module to obtain N full-process task processing nodes, where N is a positive integer.

[0026] Specifically, in the process of meteorological data processing, each functional module is executed in a certain order and dependency relationship to form a complete processing flow. Each functional module can be regarded as a task processing node, which receives input data, performs specific processing operations, and generates output data. Each functional module in the module is abstracted as a task processing node, and its input, output, and processing logic are clarified. A unique identifier is assigned to each task processing node. Relevant information of the node is extracted, such as processing time, resource consumption, failure rate, etc.

[0027] S2: Collect N historical processing data of the N full-process task processing nodes, and identify key nodes based on the N historical processing data. The key nodes include a first set of nodes whose processing time exceeds a preset time, a second set of nodes whose resource consumption is greater than a preset consumption, and a third set of nodes whose failure rate is greater than a preset failure rate.

[0028] Specifically, determine the source of historical processing data, including log files, database records or monitoring systems. Extract historical processing data of each task processing node from the determined data source, including processing time, resource consumption, such as CPU usage, memory usage, etc., and fault records. Remove incomplete or abnormal data records to ensure the accuracy and reliability of the data. Convert the extracted data into a format suitable for analysis, such as converting a data table into time series data or a feature vector. Calculate the average processing time and maximum processing time of each node, compare with the preset time threshold, and identify the first set of nodes whose processing time exceeds the preset time. Analyze the resource consumption of each node, including CPU usage, memory usage, etc., compare with the preset resource consumption threshold, and identify the second set of nodes whose resource consumption is greater than the preset consumption. Count the number of failures and failure rate of each node, compare with the preset failure rate threshold, and identify the third set of nodes whose failure rate is greater than the preset failure rate.

[0029] S3: Formulate collaborative scheduling constraints according to the first node set, the second node set, and the third node set.

[0030] Specifically, for the nodes in the first node set, the reasons for their long processing time are identified, which may be high algorithm complexity, large data volume or uneven allocation of system resources. For the nodes in the second node set, the reasons for their high resource consumption are analyzed, which may be memory leaks, computationally intensive operations or improper concurrent processing. For the nodes in the third node set, the reasons for their high failure rate are identified, which may be hardware aging, software defects or external environmental interference. Time constraints: For the nodes in the first node set, the maximum allowed processing time is set, and priority is given to reducing the waiting time and execution time of these nodes during scheduling. Resource constraints: For the nodes in the second node set, the upper limit of resource usage is set, and during scheduling, it is ensured that these nodes do not exceed the resource budget, while considering the dynamic allocation and recovery of resources. Reliability constraints: For the nodes in the third node set, the fault tolerance is set, and monitoring and fault-tolerant processing of these nodes are added during scheduling, such as setting backup nodes or restart strategies. When formulating scheduling strategies, it is necessary to consider the constraints of time, resources and reliability in a coordinated manner to ensure the feasibility and optimization of the overall scheduling plan.

[0031] S4: Obtain task requirements, obtain multiple tasks to be executed, evaluate the timeliness, priority and processing complexity of the multiple tasks to be executed, and match the task execution priority for each task to be executed according to the evaluation results, wherein the task priorities can be the same.

[0032] Specifically, the meteorological data processing tasks that need to be executed may come from user query requests, real-time data from meteorological observation stations, or data processing requirements of other business systems. For each task to be executed, its timeliness, priority, and processing complexity need to be evaluated. Timeliness refers to how long a task needs to be completed. For example, a real-time data processing task may require a response time of seconds or minutes. Priority indicates the importance and urgency of the task, and high-priority tasks should be executed first. Processing complexity reflects the computing resources and time required for task execution. Tasks with high complexity may consume more resources. Based on the evaluation results, an execution priority is assigned to each task to be executed. The priorities can be the same, indicating that these tasks can be processed in the same order or strategy when scheduled. Task requirements are collected through user interfaces, API calls, or internal business systems. Predefined evaluation indicators and algorithms are used to quantify the timeliness, priority, and processing complexity of tasks. For example, a time window can be set for timeliness, a weight value can be defined for priority, and a computing resource consumption indicator can be assigned for processing complexity. Based on the evaluation results, a sorting algorithm or priority queue is used to assign execution priority to each task. Tasks with the same priority can be classified into the same priority group for subsequent scheduling and processing.

[0033] S5: Generate a scheduling strategy optimization space according to the collaborative scheduling constraints and the task execution priority.

[0034] Specifically, determine the parameters that need to be optimized in the scheduling strategy, such as the execution order of tasks, resource allocation method, number of backup nodes, etc. According to the constraints and parameter definitions, a multi-dimensional optimization space is constructed. Each dimension represents a scheduling strategy parameter, and each point in the optimization space corresponds to a specific scheduling solution.

[0035] S6: Loading the optimization scheduling model, and generating multiple task scheduling strategies according to the task requirements.

[0036] Specifically, load the trained optimization scheduling model. This model should be able to consider the co-scheduling constraints and task execution priorities, and output a scheduling strategy that meets these conditions. Use the loaded optimization scheduling model to generate multiple feasible scheduling strategies based on task requirements. These strategies will consider different task execution orders, resource allocation methods, fault tolerance mechanisms, etc. Evaluate each generated scheduling strategy to check whether they meet all co-scheduling constraints and task execution priorities. At the same time, the expected performance of each strategy can also be evaluated, such as total processing time, resource utilization, etc. Based on the evaluation results, select the optimal scheduling strategy as the final execution plan. This strategy should be the strategy with the best expected performance while meeting all constraints and priorities.

[0037] S7: Evaluate the multiple task scheduling strategies, perform optimization in the scheduling strategy optimization space according to the evaluation results, obtain an optimal task scheduling strategy, and perform task scheduling according to the optimal task scheduling strategy.

[0038] Specifically, a detailed evaluation is performed on the generated multiple task scheduling strategies. The evaluation indicators may include total processing time, resource utilization, fault tolerance, fairness of task execution, etc. Use simulation or prediction models to estimate the performance of each strategy during actual execution. Based on the evaluation results, search in the scheduling strategy optimization space to find the optimal task scheduling strategy. Optimization algorithms such as genetic algorithms, simulated annealing, particle swarm optimization, etc. can be used to iteratively search the optimization space until a strategy that satisfies all constraints and has the best optimization goal is found. After the optimization process, the optimal task scheduling strategy is obtained. This strategy should be a strategy with the best expected performance under the premise of satisfying all collaborative scheduling constraints and task execution priorities. According to the obtained optimal task scheduling strategy, actual task scheduling is performed. Tasks are assigned to corresponding processing nodes and executed according to the execution order and resource allocation method defined in the strategy.

[0039] Further, step S2 of the present invention also includes:

[0040] Determine a historical data collection time window, and collect N historical processing data within the historical data collection time window, including processing time data, resource consumption data, and failure number data; calculate the average processing time, average resource consumption, and average failure rate of each node; perform cluster analysis based on the average processing time, average resource consumption, and average failure rate to obtain the first node set, the second node set, and the third node set, respectively.

[0041] Specifically, determine a suitable time period as the time window for historical data collection. This time window should be long enough to contain enough processing data, and close enough to reflect the current system status. Collect N historical processing data within the determined time window. These data should include processing time data, resource consumption data, and failure count data for each processing node. For each node, calculate its average processing time within the time window. This is obtained by dividing the total processing time of the node by the number of tasks it processes. Similarly, calculate the average resource consumption of each node within the time window. This can include CPU usage, memory usage, disk I / O, etc. Calculate the average failure rate of each node within the time window. This is obtained by dividing the number of failures of the node by the number of tasks it processes. The calculated average processing time, average resource consumption, and average failure rate are used as input features for clustering. Perform cluster analysis based on average processing time, average resource consumption, and average failure rate, respectively. Clustering algorithms such as K-means, DBSCAN, etc. can be used. For each feature, cluster analysis divides nodes into different groups or clusters, and the nodes in these groups have similar performance on the corresponding features. Based on the clustering result of the average processing time, a first node set is obtained, in which the nodes have similar performance in processing time. Based on the clustering result of the average resource consumption, a second node set is obtained, in which the nodes have similar performance in resource consumption. Based on the clustering result of the average failure rate, a third node set is obtained, in which the nodes have similar performance in failure rate.

[0042] Further, step S3 of the present invention also includes:

[0043] Determine collaborative scheduling goals, including data processing rate, resource utilization, and security; limit the execution time of the first node set, set a maximum execution time threshold, and exceed the maximum execution time threshold as a first constraint; limit the resource consumption of the second node set, set a maximum storage consumption threshold, and exceed the maximum storage consumption threshold as a second constraint; limit the failure rate of the third node set, set a maximum failure rate threshold, and exceed the maximum failure rate threshold as a third constraint; formulate collaborative scheduling constraints based on the first constraint, the second constraint, and the third constraint.

[0044] Specifically, the data processing rate goal is to improve the processing speed of the meteorological data processing task and ensure that the data can be processed in time. The resource utilization goal is to optimize the use of resources, avoid resource waste, and ensure that the task can get enough resources to execute. The security goal is to ensure that the meteorological data processing task will not be threatened during execution and protect the integrity and confidentiality of the data. Set the maximum execution time threshold to ensure that the nodes in the first node set will not exceed this time limit when executing the task. Exceeding the maximum execution time threshold will be regarded as a violation of the first constraint. Set the maximum storage consumption threshold to limit the resource consumption of the nodes in the second node set when executing the task. Exceeding the maximum storage consumption threshold will be regarded as a violation of the second constraint. Set the maximum failure rate threshold to require the nodes in the third node set to keep below this failure rate when executing the task. Exceeding the maximum failure rate threshold will be regarded as a violation of the third constraint. According to the goals of data processing rate, resource utilization and security, as well as the first constraint, the second constraint and the third constraint, formulate a collaborative scheduling strategy.

[0045] Further, step S5 of the present invention also includes:

[0046] A first allocation rule for computing resource and storage resource task allocation is defined based on resource demand, resource type, and resource status; a second allocation rule for task scheduling order is defined based on task execution priority, task type, and task execution time; a third allocation rule for task allocation is defined based on node load and node characteristics; and a scheduling strategy optimization space is constructed based on the first allocation rule, the second allocation rule, and the third allocation rule.

[0047] Specifically, the first allocation rule is the task allocation of computing resources and storage resources. Resource requirements: consider the specific requirements of tasks for resources such as CPU, memory, and storage. Resource type: distinguish different types of resources, such as computing resources and storage resources. Resource status: consider the current status of resources, such as whether they are available or occupied by other tasks. Prioritize the allocation of resources that meet their resource requirements and are available. The second allocation rule is the order of task scheduling; task execution priority: set the priority according to the importance and urgency of the task. Task type: distinguish different types of tasks, such as real-time processing tasks and batch processing tasks. Task execution time: consider the expected execution time of the task. Sort the tasks from high to low according to the execution priority. When the priorities are the same, consider the task type and execution time to ensure that important and urgent tasks are executed first. The third allocation rule is the allocation of tasks to nodes; node load: consider the current load of the node, including CPU usage, memory usage, etc. Node characteristics: distinguish nodes with different characteristics, such as high-performance computing nodes and large storage nodes. Prioritize the allocation of tasks to nodes with low loads and characteristics that meet the task requirements. The scheduling strategy optimization space is a set of all possible scheduling strategies. In this space, each point represents a specific scheduling strategy. According to the first allocation rule, the allocation method of computing resources and storage resources is determined. According to the second allocation rule, the execution order of tasks is determined. According to the third allocation rule, it is determined which node the task is assigned to. By combining these rules, a large number of possible scheduling strategies can be generated to form an optimization space.

[0048] Further, step S7 of the present invention also includes:

[0049] Determine the optimization goal and generate an evaluation index; calculate the performance indicators of the multiple task scheduling strategies respectively, and add the performance indicators that are greater than the evaluation index to the candidate task scheduling strategy set; randomly generate a task scheduling strategy from the optimization space as the first initial task scheduling strategy according to the optimization goal; traverse and extract the strategies in the task scheduling strategy set as the first candidate scheduling strategy, evaluate the first candidate scheduling strategy and the first initial task scheduling strategy, select the first optimal strategy according to the evaluation result, repeat the steps until the preset conditions are met, and select the current first optimal strategy as the optimal task scheduling strategy.

[0050] Specifically, it is necessary to clarify the optimization goal, such as minimizing the total processing time, maximizing resource utilization, minimizing the failure rate, ensuring the fairness of task execution, and generating evaluation indicators; according to the optimization goal, generate corresponding evaluation indicators, such as: total processing time indicator, the total time required for all tasks to complete processing. Resource utilization indicator, the average utilization of system resources. Failure rate indicator, the ratio of failures during task execution. Fairness indicator, the degree of difference in execution time between tasks. For the multiple task scheduling strategies generated, calculate their performance indicators respectively, and add the strategies greater than the evaluation indicators to the candidate task scheduling strategy set. According to the optimization goal, randomly generate a task scheduling strategy from the optimization space as the first initial task scheduling strategy. Next, traverse and extract the strategies in the task scheduling strategy set as the first candidate scheduling strategy, and evaluate the first candidate scheduling strategy and the first initial task scheduling strategy. According to the evaluation results, select the strategy with better performance as the current first optimal strategy. Repeat the above traversal and evaluation steps until the preset conditions are met, such as reaching the maximum number of iterations, the performance improvement is less than the threshold, etc. Finally, select the current first optimal strategy as the optimal task scheduling strategy and use it for the actual meteorological data processing task scheduling.

[0051] Furthermore, the present invention also includes:

[0052] If the task scheduling strategy set is traversed but does not meet the preset conditions, the first optimal strategy is randomly mutated multiple times to generate a second optimal strategy set; the strategy in the second optimal strategy set is traversed and extracted as the third optimal strategy, the third optimal strategy and the first optimal strategy are evaluated, and a candidate optimal strategy is selected based on the evaluation results. The steps are repeated until the preset conditions are met, and the current candidate optimal strategy is selected as the optimal task scheduling strategy.

[0053] Specifically, the current first optimal strategy is randomly mutated multiple times to generate a second optimal strategy set. The mutation may include changing the execution order of tasks, adjusting the nodes to which tasks are assigned, modifying the resource allocation method, etc. The strategies in the second optimal strategy set are traversed and extracted as the third optimal strategy, and the third optimal strategy and the first optimal strategy are evaluated. According to the evaluation results, the strategy with better performance is selected as the current candidate optimal strategy. Repeat the above traversal and evaluation steps until the preset conditions are met, for example, the maximum number of iterations is reached, the performance improvement is less than a threshold, etc. When the preset conditions are met, the current candidate optimal strategy is selected as the optimal task scheduling strategy and used for the actual meteorological data processing task scheduling.

[0054] Furthermore, the present invention also includes:

[0055] Acquire the performance data of each node in real time; analyze the performance data using big data analysis methods; and automatically adjust the scheduling strategy based on the analysis results.

[0056] Specifically, monitor the performance indicators of each node in real time, such as CPU usage, memory usage, disk I / O, network bandwidth, etc. Use data collection tools or systems, such as Prometheus, Grafana, etc., to collect performance data from monitoring nodes regularly or in real time. Store the collected performance data in a time series database or a big data storage system. Perform preprocessing operations such as cleaning, deduplication, and conversion on the collected performance data to ensure the accuracy and consistency of the data. Extract key features from the preprocessed data, such as the average load, maximum load, and load fluctuation of the node. Use machine learning or statistical models to analyze the extracted features to identify the performance patterns, bottlenecks, or anomalies of the node. Use stream processing technology or real-time analysis frameworks, such as Apache Kafka and Spark Streaming, to analyze real-time performance data in real time. According to the analysis results, automatically adjust the task scheduling strategy, such as reallocating tasks, adjusting resource quotas, optimizing the order of task execution, etc. Implement the adjusted scheduling strategy and monitor its effect in real time. Through the feedback mechanism, collect new performance data for further optimization. The above process is repeated continuously to continuously optimize the scheduling strategy to adapt to the ever-changing meteorological data processing needs and node performance status.

[0057] In summary, the method for optimizing the scheduling of the entire process of meteorological satellite data provided by the present invention has the following technical effects:

[0058] By connecting the meteorological data processing module, N full-process task processing nodes are obtained, where N is a positive integer; N historical processing data of the N full-process task processing nodes are collected, and key nodes are identified based on the N historical processing data, wherein the key nodes include a first node set whose processing time exceeds a preset time, a second node set whose resource consumption is greater than a preset consumption, and a third node set whose failure rate is greater than a preset failure rate; collaborative scheduling constraints are formulated based on the first node set, the second node set, and the third node set; task requirements are obtained to obtain multiple tasks to be executed, the timeliness, priority, and processing complexity of the multiple tasks to be executed are evaluated, and the task execution priority is matched for each task to be executed based on the evaluation results , wherein the task priorities may be the same; generating a scheduling strategy optimization space according to the collaborative scheduling constraints and the task execution priorities; loading an optimization scheduling model, generating multiple task scheduling strategies according to the task requirements; evaluating the multiple task scheduling strategies, optimizing in the scheduling strategy optimization space according to the evaluation results, obtaining an optimal task scheduling strategy, and scheduling tasks according to the optimal task scheduling strategy, effectively solving the problem in the prior art that the bottleneck nodes in the meteorological data processing scheduling process are not effectively identified and optimized, resulting in low overall processing efficiency and low resource utilization, further affecting the accuracy and timeliness of meteorological satellite data product processing, improving the overall processing efficiency, optimizing resource allocation, and avoiding resource waste.

[0059] Embodiment 2: Based on the method for optimizing and scheduling the entire process of meteorological satellite data in the aforementioned embodiment, and with the same inventive concept, the present invention also provides a system for optimizing and scheduling the entire process of meteorological satellite data, as shown in the attached Figure 2 The system for optimizing and scheduling the entire process of meteorological satellite data comprises:

[0060] The node acquisition module 11 is used to connect to the meteorological data processing module to obtain N full-process task processing nodes, where N is a positive integer.

[0061] The key node identification module 12 is used to collect N historical processing data of the N full-process task processing nodes, and identify key nodes based on the N historical processing data. The key nodes include a first node set whose processing time exceeds a preset time, a second node set whose resource consumption is greater than a preset consumption, and a third node set whose failure rate is greater than a preset failure rate.

[0062] The collaborative constraint formulation module 13 is used to formulate collaborative scheduling constraints according to the first node set, the second node set and the third node set.

[0063] The priority acquisition module 14 is used to obtain task requirements, obtain multiple tasks to be executed, evaluate the timeliness, priority and processing complexity of the multiple tasks to be executed, and match the task execution priority for each task to be executed according to the evaluation results, wherein the task priorities can be the same.

[0064] The optimization space generation module 15 is used to generate a scheduling strategy optimization space according to the collaborative scheduling constraints and the task execution priority.

[0065] The scheduling strategy generation module 16 is used to load the optimization scheduling model and generate multiple task scheduling strategies according to the task requirements.

[0066] The optimization module 17 is used to evaluate the multiple task scheduling strategies, perform optimization in the scheduling strategy optimization space according to the evaluation results, obtain the optimal task scheduling strategy, and perform task scheduling according to the optimal task scheduling strategy.

[0067] Furthermore, the key node identification module 12 in the system for optimizing and scheduling the entire process of meteorological satellite data is also used for:

[0068] Determine a historical data collection time window, and collect N historical processing data within the historical data collection time window, including processing time data, resource consumption data, and failure number data; calculate the average processing time, average resource consumption, and average failure rate of each node; perform cluster analysis based on the average processing time, average resource consumption, and average failure rate to obtain the first node set, the second node set, and the third node set, respectively.

[0069] Furthermore, the collaborative constraint formulation module 13 in the system for optimizing and scheduling the entire process of meteorological satellite data is also used for:

[0070] Determine collaborative scheduling goals, including data processing rate, resource utilization, and security; limit the execution time of the first node set, set a maximum execution time threshold, and exceed the maximum execution time threshold as a first constraint; limit the resource consumption of the second node set, set a maximum storage consumption threshold, and exceed the maximum storage consumption threshold as a second constraint; limit the failure rate of the third node set, set a maximum failure rate threshold, and exceed the maximum failure rate threshold as a third constraint; formulate collaborative scheduling constraints based on the first constraint, the second constraint, and the third constraint.

[0071] Furthermore, the optimal space generation module 15 in the system for optimizing and scheduling the entire process of meteorological satellite data is also used for:

[0072] A first allocation rule for computing resource and storage resource task allocation is defined based on resource demand, resource type, and resource status; a second allocation rule for task scheduling order is defined based on task execution priority, task type, and task execution time; a third allocation rule for task allocation is defined based on node load and node characteristics; and a scheduling strategy optimization space is constructed based on the first allocation rule, the second allocation rule, and the third allocation rule.

[0073] Furthermore, the optimization module 17 in the system for optimizing and scheduling the entire process of meteorological satellite data is also used for:

[0074] Determine the optimization goal and generate an evaluation index; calculate the performance indicators of the multiple task scheduling strategies respectively, and add the performance indicators that are greater than the evaluation index to the candidate task scheduling strategy set; randomly generate a task scheduling strategy from the optimization space as the first initial task scheduling strategy according to the optimization goal; traverse and extract the strategies in the task scheduling strategy set as the first candidate scheduling strategy, evaluate the first candidate scheduling strategy and the first initial task scheduling strategy, select the first optimal strategy according to the evaluation result, repeat the steps until the preset conditions are met, and select the current first optimal strategy as the optimal task scheduling strategy.

[0075] Furthermore, the optimization module 17 in the system for optimizing and scheduling the entire process of meteorological satellite data is also used for:

[0076] If the task scheduling strategy set is traversed but does not meet the preset conditions, the first optimal strategy is randomly mutated multiple times to generate a second optimal strategy set; the strategy in the second optimal strategy set is traversed and extracted as the third optimal strategy, the third optimal strategy and the first optimal strategy are evaluated, and a candidate optimal strategy is selected based on the evaluation results. The steps are repeated until the preset conditions are met, and the current candidate optimal strategy is selected as the optimal task scheduling strategy.

[0077] Furthermore, the system for optimizing and scheduling the entire process of meteorological satellite data further includes a strategy adjustment module for:

[0078] Acquire the performance data of each node in real time; analyze the performance data using big data analysis methods; and automatically adjust the scheduling strategy based on the analysis results.

[0079] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. Figure 1The method and specific examples of the whole process optimization and scheduling of meteorological satellite data in the first embodiment are also applicable to the whole process optimization and scheduling of meteorological satellite data in the present embodiment. Through the detailed description of the method of the whole process optimization and scheduling of meteorological satellite data, those skilled in the art can clearly know the whole process optimization and scheduling of meteorological satellite data in the present embodiment. Therefore, for the sake of brevity of the specification, it will not be described in detail here. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0080] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0081] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention belong to the scope of the present invention and its equivalent technologies, the present invention is also intended to include these changes and variations.

Claims

1. A method for optimizing the scheduling of the entire process of meteorological satellite data, characterized in that: include: Connect the meteorological data processing module to obtain N full-process task processing nodes, where N is a positive integer; Collecting N historical processing data of the N full-process task processing nodes, and identifying key nodes according to the N historical processing data, wherein the key nodes include a first node set whose processing time exceeds a preset time, a second node set whose resource consumption is greater than a preset consumption, and a third node set whose failure rate is greater than a preset failure rate; The step of identifying key nodes further includes: Determine a historical data collection time window, and collect N historical processing data within the historical data collection time window, including processing time data, resource consumption data, and failure number data; Calculate the average processing time, average resource consumption, and average failure rate of each node; Performing cluster analysis according to the average processing time, average resource consumption, and average failure rate to obtain the first node set, the second node set, and the third node set respectively; Formulate collaborative scheduling constraints according to the first node set, the second node set, and the third node set; The formulating of collaborative scheduling constraints further includes: Determine collaborative scheduling goals, including data processing rate, resource utilization, and security; Limiting the execution time of the first node set, setting a maximum execution time threshold, and exceeding the maximum execution time threshold is a first constraint; Limiting resource consumption of the second node set, setting a maximum storage consumption threshold, and exceeding the maximum storage consumption threshold is a second constraint; Limiting the failure rate of the third node set, setting a maximum failure rate threshold, and exceeding the maximum failure rate threshold is a third constraint; Formulate a collaborative scheduling constraint according to the first constraint, the second constraint, and the third constraint; Obtaining task requirements, obtaining multiple tasks to be executed, evaluating the timeliness, priority, and processing complexity of the multiple tasks to be executed, and matching a task execution priority for each task to be executed according to the evaluation results, wherein the task priorities may be the same; Generate a scheduling strategy optimization space according to the collaborative scheduling constraints and the task execution priority; The step of generating a scheduling strategy optimization space further includes: Define a first allocation rule for computing resource and storage resource task allocation based on resource requirements, resource types, and resource states; A second allocation rule that defines the order of task scheduling based on task execution priority, task type, and task execution time; A third allocation rule for defining task allocation based on node load and node characteristics; Constructing a scheduling strategy optimization space according to the first allocation rule, the second allocation rule and the third allocation rule; Loading the optimization scheduling model, and generating multiple task scheduling strategies according to the task requirements; The multiple task scheduling strategies are evaluated, and according to the evaluation results, an optimal task scheduling strategy is obtained in the scheduling strategy optimization space, and task scheduling is performed according to the optimal task scheduling strategy.

2. A method for optimizing the whole process of meteorological satellite data scheduling as claimed in claim 1, characterized in that: The plurality of task scheduling strategies are evaluated, and according to the evaluation results, an optimization is performed in the scheduling strategy optimization space to obtain an optimal task scheduling strategy, and task scheduling is performed according to the optimal task scheduling strategy, further comprising: Determine the optimization goal and generate evaluation indicators; Calculate the performance indicators of the multiple task scheduling strategies respectively, and add the ones with greater than the evaluation indicators into the candidate task scheduling strategy set; Randomly generating a task scheduling strategy from the optimization space according to the optimization goal as a first initial task scheduling strategy; Traverse and extract the strategies in the task scheduling strategy set as the first candidate scheduling strategy, evaluate the first candidate scheduling strategy and the first initial task scheduling strategy, select the first optimal strategy according to the evaluation result, repeat the steps until the preset conditions are met, and select the current first optimal strategy as the optimal task scheduling strategy.

3. A method for optimizing the whole process of meteorological satellite data scheduling as claimed in claim 2, characterized in that: Also includes: If the task scheduling strategy set is traversed but does not meet the preset conditions, the first optimal strategy is randomly mutated multiple times to generate a second optimal strategy set; Traverse and extract the strategies in the second optimal strategy set as the third optimal strategy, evaluate the third optimal strategy and the first optimal strategy, select a candidate optimal strategy according to the evaluation result, repeat the steps until the preset conditions are met, and select the current candidate optimal strategy as the optimal task scheduling strategy.

4. A method for optimizing the whole process of meteorological satellite data scheduling as claimed in claim 1, characterized in that: Also includes: Get performance data of each node in real time; Analyzing the performance data using a big data analysis method; Automatically adjust the scheduling strategy based on the analysis results.

5. A system for optimizing and scheduling the entire process of meteorological satellite data, characterized in that: Steps for implementing the method for optimizing and scheduling the entire process of meteorological satellite data as described in any one of claims 1 to 4, wherein the system for optimizing and scheduling the entire process of meteorological satellite data comprises: The node acquisition module is used to connect to the meteorological data processing module to obtain N full-process task processing nodes, where N is a positive integer; A key node identification module, used to collect N historical processing data of the N full-process task processing nodes, and identify key nodes according to the N historical processing data, wherein the key nodes include a first node set whose processing time exceeds a preset time, a second node set whose resource consumption is greater than a preset consumption, and a third node set whose failure rate is greater than a preset failure rate; A collaborative constraint formulation module, configured to formulate collaborative scheduling constraints according to the first node set, the second node set, and the third node set; A priority acquisition module is used to acquire task requirements, obtain multiple tasks to be executed, evaluate the timeliness, priority and processing complexity of the multiple tasks to be executed, and match the task execution priority for each task to be executed according to the evaluation results, wherein the task priorities can be the same; An optimization space generation module, used to generate a scheduling strategy optimization space according to the collaborative scheduling constraints and the task execution priority; A scheduling strategy generation module is used to load the optimization scheduling model and generate multiple task scheduling strategies according to the task requirements; The optimization module is used to evaluate the multiple task scheduling strategies, perform optimization in the scheduling strategy optimization space according to the evaluation results, obtain the optimal task scheduling strategy, and perform task scheduling according to the optimal task scheduling strategy.

Images