Remote sensing big data production cloud computing task efficient scheduling method

Abstract

According to the efficient scheduling method for the remote sensing big data production cloud computing task, a MapReduce framework is improved to establish a new scheduling framework suitable for remote sensing data production, so that Worker has a decision-making right and does not completely depend on Master to dispatch tasks any more, the problem that the load of a control node of the MapReduce framework is too heavy is solved, a Worker end collects resources and task information on a local machine, and the task scheduling efficiency is improved. The task scheduling method comprises the following steps of: determining whether a next task is executed or not, sending the information to a Master through a Heartbeat mechanism, sending some tasks needing more resources to the Master when the resources of a Worker end are insufficient, and assigning the tasks to nodes with more idle resources in a cluster by the Master to complete the tasks, constructing a task game assignment algorithm, and performing task game scheduling on the tasks. The framework comprises two types of task queues: a MapReduce framework resource model is improved based on a public task game queue and a private task assignment queue, the production requirement of mass remote sensing data is met, and the production efficiency of the remote sensing data is improved.

Description

Technical Field

[0001] The present application relates to a remote sensing data production task scheduling method, and in particular to an efficient scheduling method for remote sensing big data production cloud computing tasks, belonging to the technical field of remote sensing data scheduling. Background Art

[0002] Remote sensing satellite ground stations process data acquired and transmitted by remote sensing satellite platforms for use in monitoring agriculture, forestry, oceans, land, environmental protection, and meteorology, thereby fulfilling the monitoring function of remote sensing satellites. The volume of satellite data required is enormous, and the processing process is complex. For example, the video remote sensing data processing process includes data cataloging, image cloud identification, precise attitude determination, adjustment, registration, resampling, video generation, geometric quality inspection, radiometric quality inspection, super-resolution reconstruction, and super-resolution quality inspection. Coupled with frequent I / O transactions, the production of remote sensing data is a very arduous task. To meet the challenges of remote sensing satellite data production and meet the real-time requirements of applications such as forest fire monitoring and meteorological monitoring, continuous algorithm improvements are required to improve data processing efficiency. However, this improvement is limited, and well-designed scheduling software is required to integrate computing resources, improve resource utilization efficiency, and accelerate production.

[0003] The fundamental problem lies in the limited resources and computing performance of a single node, making it difficult to achieve real-time production of massive remote sensing data. To address this issue, multiple computing nodes can be used for load balancing. However, to leverage multiple computing nodes for production, a distributed scheduling system is needed to integrate multi-node resources, fully leveraging the hardware and software to truly improve production efficiency. The effectiveness of a well-designed distributed task scheduling system depends on the effective utilization of resources within the distributed system. An excellent task scheduling system can fully utilize the distributed system's computing and storage resources, maximizing the advantages of hardware resources. A poor task scheduling system, on the other hand, wastes resources, failing to improve or even reducing overall efficiency. Therefore, to truly use distributed task scheduling to address the challenges facing remote sensing data production, in-depth research and experimental verification are required on the scheduling system's architectural design, scheduling algorithms and their implementation, task management, resource management, and exception handling. Therefore, the development of a task scheduling system for remote sensing data production is of paramount practical significance.

[0004] Some task scheduling systems and frameworks in the existing technology have their own characteristics, but they are all aimed at ordinary distributed tasks. In solving the problem of remote sensing data production, due to the particularity of production tasks, such as massive data, frequent IO interactions and complex processes, the dependency relationship between tasks also has its own particularity. Directly using a general framework cannot fully meet the task scheduling needs of remote sensing data production. It is necessary to design and develop a professional production scheduling system based on the particularity of remote sensing data production.

[0005] The problems that need to be solved in the existing remote sensing data production task scheduling and the key technical difficulties of this application include:

[0006] (1) The resources and computing performance of a single node are limited, making it difficult to achieve real-time production of massive remote sensing data. Multiple computing nodes are needed to achieve load balancing. However, the existing remote sensing data distributed task scheduling system is not well designed. The resources in the distributed system cannot be effectively utilized, and computing resources, storage resources, etc. cannot be fully utilized. The advantages of hardware resources cannot be brought into play. The task scheduling system wastes resources and cannot improve or even reduce overall efficiency. Distributed task scheduling cannot be truly used to solve the problems faced by remote sensing data production. There is a lack of design and experimental verification of the scheduling system's architecture design, scheduling algorithm and its implementation, task management, resource management, exception handling, etc. The existing technology is aimed at ordinary distributed tasks. In solving the problem of remote sensing data production, due to the particularity of production tasks, such as massive data, frequent IO interactions and complex processes, the dependency relationship between tasks also has its own particularity. Directly using a general framework cannot fully meet the task scheduling needs of remote sensing data production. It is necessary to design and develop a professional production scheduling system based on the particularity of remote sensing data production.

[0007] (2) The remote sensing data production process is complex, with dependencies between steps. Some steps must be executed after others, otherwise the execution will fail. It is necessary to use a scheduling system to manage production to ensure production quality and stable and reliable operation of the production system. The flaw of MapReduce's scheduling algorithm is that when larger jobs are submitted before smaller ones, resources will be wasted because the smaller jobs will not receive a response for a long time. At the same time, when there are a large number of interactive jobs, user requests cannot be responded to in a timely manner, affecting the user experience. The main shortcomings of applying the existing MapReduce technology to remote sensing data scheduling include: using task slots to abstract resource information, which is not flexible enough in resource measurement; scheduling is entirely done by the JobTracker, which will put too much pressure on the node where the JobTracker is located, and the possibility of single point failure increases with the increase of cluster size and the number of tasks; there is a certain delay in the information sent by the Heartbeat to the TaskTracker by the JobTracker, which affects efficiency when there are many short-term tasks; the TaskTracker has no decision-making power at all. In the event of a task failure, the TaskTracker still needs to send task information to the JobTracker and then wait for the JobTracker to decide on the failed task (such as redoing or abandoning it), which affects efficiency; and the source code is complex and difficult to maintain.

[0008] (3) Remote sensing data production is very arduous due to the large amount of data required and the high real-time requirements. Currently, there is a lack of effective methods to improve the efficiency of remote sensing data production. MapReduce, as a distributed task scheduling framework, has been widely used in various software systems. However, it has also exposed many problems in use. It has not been improved for the needs of remote sensing data production. There is a lack of a new scheduling framework for remote sensing data. There is a lack of task game-dispatching algorithms, which cannot adapt to the characteristics of remote sensing data production tasks. There is no establishment of public task game queues and private task dispatch queues. There is a lack of resource models and node pool models. For some application scenarios with high requirements for data real-time, the production efficiency is too low, which makes it take too long for users to obtain data products. When the data products are obtained, the information obtained from the remote sensing images may be outdated and cannot play its due value. While increasing the amount of remote sensing data, the difficulty of remote sensing data processing increases, and the efficiency of remote sensing data production also decreases. Summary of the Invention

[0009] This application designs good scheduling software to integrate computing resources, improve resource utilization efficiency, and thus speed up production. The proposed framework has made many improvements to MapReduce, including the proposed public task game queue and private task dispatch queue, the proposed resource model and node pool model, and the task game-dispatching algorithm. According to the proposed improved framework, the production system is specifically implemented and data production experiments are carried out. The experimental results are analyzed, and the results show that the framework proposed in this application can meet the needs of data production, improve data production efficiency to a certain extent, and truly use distributed task scheduling to solve the problems faced by remote sensing data production. The architectural design of the scheduling system, scheduling algorithms and their implementation, task management, resource management, exception handling, etc. have been deeply studied and experimentally verified, which has very important practical significance and role in the research and development of remote sensing data production task scheduling systems.

[0010] To achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0011] This paper proposes an efficient scheduling method for cloud computing tasks in remote sensing big data production. The MapReduce framework is improved to establish a new scheduling framework suitable for remote sensing data production. The improved scheduling framework still adopts a master / slave architecture, with the control master node and the computing worker nodes completing the work. The master receives job requests submitted by the client, parses and verifies the requests, and then breaks the job into tasks. The tasks are then passed to the workers through the task queue, while collecting task status information sent by the workers. The improvement to the MapReduce framework is to give the workers decision-making power, rather than relying entirely on the master to dispatch tasks, thus overcoming the problem of overloaded control nodes in the MapReduce framework. The workers collect resource and task information on the local machine to decide whether to execute the next task and also send this information to the master through the heartbeat mechanism. When the workers are short of resources, some tasks requiring more resources are sent to the master, which dispatches them to nodes with more idle resources in the cluster. The task game dispatch algorithm is constructed. The algorithm framework is based on two types of task queues: a public task game queue and a private task dispatch queue, which improve the MapReduce framework resource model.

[0012] 1) Based on the production characteristics of remote sensing data, MapReduce is used to process distributed tasks. Each data processing step of the remote sensing data processing plug-in is abstractly encapsulated into a task. Remote sensing data processing does not involve Reduce work and does not execute ReduceTasks. A Master node decomposes the job into tasks and then dispatches the tasks to multiple Worker nodes for execution.

[0013] 2) In terms of scheduling strategy, a task game dispatching algorithm is established based on the characteristics of satellite data processing tasks. That is, the two scheduling modes of task game and task dispatch are coordinated to perform overall task scheduling. First, the control node places the task into the public task game queue. When the resources of each computing node are sufficient, the task game mode is adopted. Each computing node bids for a task from the public task game queue and immediately executes the task obtained through the game. As the number of tasks executed simultaneously by a computing node increases, the remaining resources of the computing node gradually decreases until it is insufficient to complete the execution of the next task. At this time, the computing node chooses to wait until the resources are sufficient or return the task to the control node based on the required resources and the total available resources of the node. If it chooses to return the task to the control node, the task is scheduled according to the task dispatching mode. The control node dispatches the task to the appropriate computing node through the computing node's private queue based on the resource information collected from each computing node. At this time, the computing node that has obtained the task has priority to execute the assigned task, that is, it stops bidding for new tasks from the public queue until all tasks in the private queue are taken out.

[0014] Preferably, a public task game queue is used in the task game mode. There is only one in the entire system. There is only one message producer for the public queue, which is the Master, and there are multiple consumers, namely the Workers. When a message is consumed by a consumer, other consumers will no longer receive this message. The public task game queue is set as a priority queue. When the Master receives a job manually submitted by the client operator, it puts the job into the job queue with a higher priority. After taking out the job, it decomposes the job to obtain a task, and passes the job priority to the task, and puts the task into the task queue with a higher priority so that the task can be executed first, so as to solve the problem of untimely response to job submission by the client when the task volume is large. All unexecuted tasks will enter the public task game queue and be executed after the game of each Worker node. Only when the Worker end fails to execute the task for some reason will the Worker end return it to the Master end, and the Master may assign the task to the private task dispatch queue.

[0015] Preferably, private task dispatch queue: used in task dispatch mode, in the scheduling system, the number of private task dispatch queues is consistent with the number of Worker nodes, each Worker corresponds to a private task dispatch queue, responsible for completing all tasks in this queue, the tasks in this queue can only be taken out and executed by its corresponding Worker, and other Worker nodes have no access. Each private task dispatch queue has only one message producer, which is the Master, and only one consumer, which is the Worker that owns this queue; the priority of the private queue is higher than the public queue. When there are tasks in both the private queue and the public queue, the tasks in the private queue are executed first, and only when there are no tasks in the private queue will the tasks be taken out from the public queue for execution.

[0016] Preferably, the resource model does not use the concept of slots established for resources in MapReduce, but remodels resources based on their multidimensionality. The established resource model reflects the CPU, memory, disk I / O, and task quantity information of the current computing node. The constructed different resource entities are comparable and additive. The established resource type Resource class is an abstract class with several attributes such as the number of CPU cores, memory size, and I / O bandwidth, as well as resource addition, subtraction, and comparison methods. Subclasses inherit and override these methods.

[0017] The comparability of resource entities is between the resources required for the task and the remaining resources of the computing nodes, as well as between different computing nodes. When the Master node assigns a task, it compares the resource status of each computing node and determines whether the node resources are sufficient to perform the current task. By comparing, the most suitable node that meets the requirements is selected to assign the task. When the Worker node receives the task, it performs task evaluation to check whether the resources required by the task can be met by the current node. Assume that the vector R = (r1, r2, ..., r n ) represents the dimensions of resources. Considering the number of CPU cores and memory size (Mb), let R1 = (10, 2000) and R2 = (12, 1500). Then R1 and R2 are better in CPU and memory, respectively.

[0018] Preferably, a resource node pool model is used: the node pool is a container for storing the IP and resource information of all computing nodes, which is managed and maintained by the Master. The node pool manages the computing nodes and is responsible for updating the information when computing node resource information is sent. When the Master is dispatching tasks, the node pool selects appropriate computing nodes for task dispatching based on the task's resource requirements. This application establishes two node pool models: a convenient set model and a multi-hop space model.

[0019] (1) Convenient Collection Model

[0020] Put all nodes into the hash table, calculate the primary key through the hash function, get the Hashcode, and then determine the position of the key-value pair in the table based on the Hashcode. Use the node IP address as the primary key and store it in the hash table. When new resource information arrives, directly find the node object of the corresponding IP and update its resource information. When a task arrives, traverse the entire hash table and find all nodes that meet the resources required for the task through comparison and calculation. Select a node to perform the task. For different needs, two methods are constructed: one is to only select all nodes. Among any of the nodes that meet the requirements, the traversal stops when a node that meets the requirements is found during the traversal; the other method requires that the best one be selected according to a certain rule among all the nodes that meet the requirements. All nodes must be traversed. During the traversal, the current best node that meets the requirements is recorded while traversing. Finally, the node to be found is obtained after the traversal is completed. After obtaining the node, the node is deleted from the table or the node resource information is subtracted from the task information as the latest resource information of the node to prevent too many tasks from being assigned to the same node in a short period of time, thereby causing the problem of overload of a certain node.

[0021] (2) Multi-hop spatial model

[0022] Within the node pool, multiple skip tables are used to store computing nodes. The number of skip tables is equal to the number of resource dimensions. Each skip table stores and manages nodes according to a certain dimension of the resource. The node skip table corresponding to the number of CPU cores stores all nodes in ascending order of CPU core count. This rule remains unchanged when adding, updating, or deleting elements.

[0023] When new node resource information arrives, it is determined whether the node exists in the CPU skip list. If the node does not exist in the skip list, the node is inserted into the CPU skip list. Otherwise, the original node information is updated and moved to the appropriate position.

[0024] When a task arrives, the node pool dispatches nodes to execute according to the resources required for the task. For a task, the dispatched nodes must meet the requirements in all dimensions of resources. The nodes must ensure that the CPU, memory, and IO bandwidth resources are sufficient to complete the task. In order to find all nodes that meet the requirements, first construct a set to store all the nodes, and traverse the jump tables in the node pool one by one. The traversal should start from the node with the least number of resources and end at the first node whose resources meet the requirements. When a node in the jump table whose resources do not meet the task requirements is found, it is removed from the set. After processing the last jump table, the remaining elements in the set are all the nodes that meet the task requirements.

[0025] Preferably, job submission and decomposition: jobs are submitted by the client, and the submitted jobs are sent to the server through WebService, providing a WebService interface for job submission, and external programs submit tasks regularly through this interface;

[0026] When a client submits a job, it sets the job priority. The job priority attribute is passed to its decomposed tasks. The task priority not only determines the order in which tasks are executed, but also determines whether a task has the right to preempt other tasks.

[0027] Job information via the Web The service passes the job to the Master. After the job is submitted, the Master receives and verifies whether the submitted request complies with the interface specifications, including whether the request content is complete, the data path for request processing exists, and whether the data format is correct. If the verification passes, the job is placed in the job queue according to priority and submission time. The Master also checks whether the user exists in the queuing user table queuingUsers maintained by the Master. If not, the user and their jobCapacity and runningJobs information are added to the table. At this time, the total number of jobs that can be executed in parallel in the system (jobCapacity) is equal to the sum of jobCapacity and the jobCapacities of all users in queuingUsers. In another thread, the Master monitors the entry and exit of jobs in the queue at any time. When a job enters the queue, it checks the runningJobs and jobCapacity of the user corresponding to the job. If the former is less than the latter, it checks the jobs submitted by the user in the queue from the front to the back and removes them for execution. When a job is removed from the queue, it checks whether there are other jobs submitted by the user in the queue. If not, the user is removed from the queuingUsers table. jobCapacity corresponds to the number of jobs that can be executed in parallel by external programs submitted through the WebService interface.

[0028] When the Master takes a job from the job queue, it first preprocesses the job and then decomposes the job. Job preprocessing requires parsing the process and data information to be executed for processing the current data. The process records the steps of processing the data, and the data information records the data name, specific path, and number of data frames of the data to be processed. After the preprocessing is completed, the job is decomposed. Job decomposition first obtains multiple steps according to the process of processing data. The order between each step determines the dependency relationship of the decomposed tasks. The tasks corresponding to the subsequent steps need to be executed after all the tasks corresponding to the previous steps are completed. The job is decomposed into tasks, and the input and output data paths and execution parameter information are configured for the tasks.

[0029] Preferably, task queuing: tasks are decomposed by the Master and placed in task queues. Task queues include two types: public queues and private queues. Jobs submitted by users are decomposed into tasks and placed in public queues, while tasks returned to the Master by Workers that are not executed due to insufficient resources or other abnormal situations and need to be re-queued are placed in private queues. The number of private queues is equal to the number of Worker nodes. Each private queue has a corresponding Worker node. The Master node decides which private queue to place the task based on the resource status of the collected Worker nodes.

[0030] The resource information collected by the Master is sent by the Worker node through the Heartbeat. The Master node maintains a node pool for all computing nodes based on the resource and task conditions. Initially, the node pool is empty. When the Master node starts to collect the first resource information sent by the Worker node, it puts this node and resource information into the node pool. Later, when it receives new resource information, it compares the resources and inserts them into the appropriate position in the node pool or updates the elements in the node pool. It selects the node from the node pool and puts the task into the private queue corresponding to the selected node. It immediately updates the resource information corresponding to the node and updates the position of the node in the node pool at the same time. It determines that the tasks put into the private queue have used the required resources and the number of remaining resources is reduced. This prevents some nodes in the node pool from being assigned too many tasks in a short period of time, so that before the new resource information arrives, unreasonable task distribution causes load imbalance problems.

[0031] Preferably, task execution is completed by Worker nodes. All Worker nodes monitor the public task game queue and their own private task dispatch queue. When there is a new task in the public queue, all Worker nodes take the task from the public queue. Each node will wait a short time after taking out a task before trying to take a new task. This increases the probability of other nodes taking new tasks, makes the number of tasks executed by each node more balanced, and prevents one or several nodes from taking out too many tasks in a short period of time.

[0032] After the Worker takes out the task, it first evaluates the task. If the number of resources required for the task is too large and the current node resources are completely unable to meet the needs of completing the task, the Worker node returns the task to the Master node. The Master node then dispatches the task to the node's private queue based on the resource status of each node. If the number of resources required for the task is temporarily not met by the current node due to executing other tasks, but it is expected to be met after waiting for a period of time, the Worker node will wait until the resources are sufficient before executing this task. During the waiting period, the Worker will not take out new tasks from the public task game queue for execution. If due to some abnormal reasons, the Worker cannot wait for the required resources, the task will be returned to the Master node after the set timeout period.

[0033] If the number of resources required by the task can be met by the current node, the task will be executed, and the task pool will be updated to add the task to the task pool. The task pool maintains the status of all tasks being executed by the current node. During the task execution, if an exception occurs that causes the task to fail or time out, the task will be re-executed or reported directly to the Master node based on user needs or task characteristics. After the task is completed, a task completion notification will be sent to the Master node, and the task pool will be updated at the same time to remove the task from the task pool.

[0034] For tasks in the private queue, the Worker node monitors, retrieves and executes them. The private queue has a higher priority than the public queue. When tasks appear in the private queue, the Worker node stops retrieving tasks from the public queue and only retrieves tasks from the private queue. The processing flow after retrieving tasks is the same as that of the public queue. When all tasks in the private queue have been retrieved, the Worker node resumes retrieving tasks from the public queue.

[0035] Preferably, result collection: the Master node collects task result information and decides the process execution based on the result information. After executing the task, the Worker node writes the task result to the database and sends the task ID to the Master through different queues according to different task results. The processed tasks will be placed in one queue, and the unprocessed tasks will be placed in another queue. For the unprocessed task queue, the Master will take the unprocessed tasks out of the queue and then dispatch the tasks to the private queue, which will be taken out and processed again by the Worker node. For the processed task queue, after receiving the task ID, the Master determines which step of the process the task belongs to, and then determines whether all tasks of the step have been completed. If the execution is completed, the step is marked as finished and the next step is continued. If the step is the last step of the process, the process execution is marked as completed, that is, the current job execution is completed.

[0036] Preferably, the scheduling algorithm: constructs a task game dispatching algorithm, uses the task game strategy when the resources of each computing node are sufficient to improve the efficiency of task distribution, and uses the task dispatching strategy when the resources are insufficient to improve the load balancing rate and resource utilization;

[0037] 1-Task Game

[0038] When no jobs are submitted to the production system, each computing node is idle. If a job is submitted to the system, each computing node has the ability to execute the task, allowing all workers to bid for tasks from the public task bidding queue.

[0039] The task negotiation mode is applied when computing node resources are idle. When the Worker node detects that there are no tasks running in the private task dispatch queue, it negotiates for tasks from the public task queue. After obtaining the task through negotiation, it evaluates whether the current node resources are sufficient to execute the task. If not, it waits until the resources are sufficient or returns the task to the Master. If sufficient, it executes the task asynchronously and then evaluates whether the current node resources are sufficient to execute the next task. If not, it waits until the resources are sufficient or times out, and then continues to negotiate for tasks from the public queue. If sufficient, it directly negotiates for tasks from the public queue.

[0040] 2-Task Assignment

[0041] Task dispatch is completed by the Master node. The dispatch strategy is related to the computing node resources. When there are insufficient resources or other abnormal situations, the Worker node returns unprocessed tasks to the Master node. The Master node dispatches tasks when it receives unprocessed tasks.

[0042] In the task dispatching mode, the Master maintains a node pool that manages global computing node information, including IP addresses and resource information. When resource information is sent to the Master by the Worker node through the Heartbeat message, the Master will update the node information in the pool. When a task needs to be dispatched, the Master takes a node that meets the task resource requirements from the node pool, dispatches the task to the node for execution, and updates the node resource information.

[0043] It is stipulated that tasks assigned by the Master have a higher priority than ordinary tasks, that is, the priority of each Worker's private queue is higher than the public queue as a whole. When a task appears in a Worker's private queue, the Worker will stop taking tasks from the public queue and only take tasks from the private queue until all tasks in the private queue are taken and there are idle resources, and then resume taking tasks from the public queue.

[0044] Compared with the existing technology, the innovation and advantages of this application are:

[0045] (1) The task game dispatching algorithm of this application achieves the purpose of reducing the load pressure of the control node and improving the scheduling efficiency by assigning part of the task scheduling rights to the computing node. The task game mode enables the system to execute tasks efficiently when resources are sufficient, and the control node does not need to do any calculations about task dispatching, which greatly reduces the workload of the control node and reduces the load, thereby reducing the possibility of control node errors. At the same time, since the task is directly taken out by the computing node and the execution is determined based on the current node available resources and the resources required for the task, without the need for the Master to coordinate and dispatch resources, the problem of unreasonable task dispatch caused by network delays in the process of task and resource information transmission is avoided. The task dispatching mode is used when the number of tasks and resource utilization of the computing node are close to saturation. In order to prevent the computing node resources from being exhausted, resulting in a decrease in efficiency or even anomalies, the computing node first makes a decision at this time whether to continue waiting for resources or return to the control node. If it continues to wait for resources, the computing node will no longer take tasks from the task queue to execute until the current task obtains the required resources and starts execution; if it returns to the control node, the control node will dispatch the tasks. Since the control node continuously collects resource information of all computing nodes, it can reasonably dispatch tasks to appropriate computing nodes according to a certain dispatching strategy, thus maximizing the utilization of system resources and greatly improving the scheduling quality and efficiency of producing massive remote sensing data.

[0046] (2) In order to solve the problems faced by remote sensing data production, this application improves the MapReduce framework to establish a new scheduling framework suitable for remote sensing data production. The improvement of the MapReduce framework is to give the Worker the decision-making power, and no longer rely entirely on the Master to dispatch tasks, to overcome the problem of heavy load on the control node of the MapReduce framework. The Worker collects resource and task information on the local machine to decide whether to execute the next task, and at the same time sends this information to the Master through the Heartbeat mechanism. When the resources on the Worker are insufficient, some tasks requiring more resources are sent to the Master, which dispatches them to nodes with more idle resources in the cluster to complete, and constructs a task game dispatching algorithm. The framework of the algorithm is based on two types of task queues: the MapReduce framework resource model is improved based on the public task game queue and the private task dispatch queue. Experiments show that the improved framework can meet the production needs of massive remote sensing data and improve the efficiency of remote sensing data production to a certain extent.

[0047] (3) This application designs good scheduling software to integrate computing resources, improve resource utilization efficiency, and thus speed up production. The proposed framework has made many improvements to MapReduce, including the proposed public task game queue and private task dispatch queue, the proposed resource model and node pool model, and the task game-dispatching algorithm. Based on the proposed improved framework, the production system is specifically implemented and data production experiments are conducted. The experimental results are analyzed, and the results show that the framework proposed in this application can meet the needs of data production, improve data production efficiency to a certain extent, and truly use distributed task scheduling to solve the problems faced by remote sensing data production. The architecture design of the scheduling system, scheduling algorithm and its implementation, task management, resource management, exception handling, etc. have been deeply studied and experimentally verified, which has very important practical significance and role in the research and development of remote sensing data production task scheduling system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a schematic diagram of the task dispatching process of the convenient collection model.

[0049] Figure 2 It is a schematic diagram of the resource information update process of the multi-hop space model.

[0050] Figure 3 This is a schematic diagram of the laser scanning camera calibration plate.

[0051] Figure 4 This is an example diagram of job submission and decomposition.

[0052] Figure 5 It is the flow chart of mobile measurement calibration.

[0053] Figure 6 This is a schematic diagram of the task queuing process.

[0054] Figure 7 It is a schematic diagram of the correspondence between data production operations, steps and tasks.

[0055] Figure 8 It is a schematic diagram of the error conditions of each task and the accuracy of the result data.

[0056] Figure 9 It is a schematic diagram of the data processing program time test. DETAILED DESCRIPTION

[0057] Below, in conjunction with the accompanying drawings, the technical solution of the efficient scheduling method for remote sensing big data production cloud computing tasks provided by this application is further described so that those skilled in the art can better understand this application and implement it.

[0058] Advances in remote sensing and mapping technology have led to an increasingly widespread range of remote sensing applications, from target tracking and terrain exploration to land planning, weather monitoring, forest fire detection, crop monitoring, electronic map navigation, and factory pollution monitoring. Remote sensing data plays a central role in these diverse applications. While different applications require remote sensing data with varying characteristics, remote sensing data generally requires large amounts of data. This characteristic, however, is a source of challenges in remote sensing data production. Initial remote sensing data typically requires processing before it can be used by users. The process of processing this initial data into various data products is considered remote sensing data production. The efficiency of remote sensing data production largely determines its value. For applications requiring high real-time data availability, such as forest fire monitoring and weather monitoring, low production efficiency can delay the delivery of data products to users. By the time these products are available, the information derived from the remote sensing imagery may be outdated and unable to fully realize its intended value. On the other hand, the need to extract as much information as possible from remote sensing imagery increases the volume and processing complexity of remote sensing data, further challenging the efficiency of remote sensing data production.

[0059] To solve the problems faced by remote sensing data production, the MapReduce framework was improved to establish a new scheduling framework suitable for remote sensing data production. The improved scheduling framework still adopts a master / slave architecture, with the work completed by the control Master node and the calculation Worker node. The Master receives the job request submitted by the client, parses and verifies the request, and then decomposes the job into tasks. The tasks are then passed to the Worker through the task queue and the task status information sent by the Worker is collected at the same time.

[0060] The improvement to the MapReduce framework is to give the Worker decision-making power, instead of relying entirely on the Master to assign tasks, to overcome the problem of heavy load on the control node of the MapReduce framework. The Worker collects resource and task information on the local machine to decide whether to execute the next task, and at the same time sends this information to the Master through the Heartbeat mechanism. When the resources on the Worker side are insufficient, some tasks requiring more resources are sent to the Master, which assigns them to nodes with more idle resources in the cluster to complete, and construct a task game dispatching algorithm. The framework of the algorithm is based on two types of task queues: the MapReduce framework resource model is improved based on the public task game queue and the private task dispatch queue. Experiments show that the improved framework can meet the production needs of remote sensing data and improve the production efficiency of remote sensing data to a certain extent.

[0061] 1. Problem Description and Analysis

[0062] Remote sensing data production requires distributed task scheduling to solve production efficiency issues. However, due to its own problems, the general task scheduling framework handles multiple types of tasks. It is not targeted in design and cannot adapt to the needs. It needs to be improved to be suitable for the task scheduling framework of massive remote sensing data production.

[0063] General task scheduling frameworks have inherent flaws. The MapReduce scheduling framework itself has some flaws, and scheduling algorithms each have their own strengths and weaknesses. If the MapReduce scheduling framework is used, its inherent flaws must first be addressed to improve actual production efficiency. Furthermore, remote sensing data production tasks are diverse, with significant differences in resource usage and execution time between tasks. The dependencies between remote sensing data production tasks also have inherent characteristics. Improper consideration of these factors during scheduling can not only affect scheduling efficiency but can even lead to task failure.

[0064] The MapReduce framework requires accurate quantification of resources during task execution, but the resource consumption of data production programs during the production process will change dynamically over time. In particular, for some tasks with complex processing and long processing times, their resource consumption may continue to change. In addition, the resource usage of some programs is related to the size of the input data. Although using a constant to represent resource consumption brings convenience to scheduling and processing, it lacks accuracy.

[0065] Remote sensing data production users have unique needs. Beyond scheduling, remote sensing data production also requires additional tasks, including data management, task parameter configuration, and automated production. The management of remote sensing data requires particular attention. For users, inputting the initial data path into the production system effectively submits a task to the system. The production system then verifies the validity of the input data, parses the data category, and schedules different production processes based on the data category. Parsed data information, including input and output paths, intermediate result paths, and operational parameters, is dynamically parsed based on the initial data and passed to the task execution program via an interface. Tasks are then distributed to compute nodes for execution based on specific dependencies and parallelism. Furthermore, due to the massive volume of remote sensing data, automated production capabilities are required. When new data arrives, production orders can be parsed from external interfaces to automatically execute data production tasks.

[0066] To maximize efficiency, it's necessary to develop a dedicated task scheduling framework tailored to the specific characteristics of remote sensing data processing. Remote sensing data processing, as the subject of task scheduling, is specialized and unique, presenting both challenges and opportunities for scheduling systems. Only by fully understanding the operational processes and task characteristics of remote sensing data production and developing appropriate scheduling solutions based on these characteristics can we fully leverage the advantages of distributed scheduling and maximize production efficiency.

[0067] 2. Improved Scheduling Algorithm Based on MapReduce Model

[0068] The MapReduce framework is improved to adapt to the production needs of remote sensing data. The improvement method is to transfer the scheduling decision-making power to TaskTracker (Worker node) to solve the problem that the MapReduce framework JobTracker (Master node) is overloaded and affects efficiency. At the same time, a new resource model is established to replace the Slot of MapReduce to solve the problem that Slot cannot accurately reflect the resource status in resource measurement. A resource node pool management model is established to cooperate with the resource model to manage resources.

[0069] (1) Improved scheduling framework

[0070] The improved scheduling framework adopts a master / slave architecture, with the work completed by the control node (Master node) and the computing node (Worker node). The Master receives the job request submitted by the client (Client node), parses and verifies the request, and then decomposes the job into tasks. The task information is stored in the database. After the storage is completed, the task is passed to the Worker through the task queue and the task completion information sent by the Worker is collected at the same time.

[0071] The improved framework differs from the MapReduce framework in that it empowers workers to make decisions, rather than relying solely on the master to dispatch tasks. Workers collect resource and task information on their local machines to decide whether to execute the next task and simultaneously send this information to the master via the heartbeat mechanism. When resources are insufficient on the worker side, they may send time-consuming tasks to the master, which then dispatches them to nodes in the cluster with more idle resources. This establishes a task-game dispatching algorithm, based on two types of task queues: a public task-game queue and a private task-dispatch queue. Both queues store pending tasks. The master is the producer of task messages, and the workers are the consumers. The task queue is responsible for task storage and serves as a channel for task communication between the master and workers. When there are multiple producers and consumers, multi-threaded synchronization is necessary. Both task queues are implemented using queues in ActiveMQ, Apache's open source messaging middleware. The public task-game queue and the private task-dispatch queue have different uses and scenarios, corresponding to the task-game and task-dispatch modes, respectively.

[0072] 1. Public task game queue

[0073] When used in task game mode, there is only one in the entire system. There is only one message producer for the public queue, which is the Master, and there are multiple consumers, namely the Workers. When a message is consumed by a consumer, other consumers will no longer receive this message. The public task game queue is set as a priority queue. When the Master receives a job manually submitted by the client operator, it puts the job into the job queue with a higher priority. After taking out the job, it decomposes the job to obtain tasks, passes the job priority to the tasks, and puts the tasks into the task queue with a higher priority so that the tasks can be executed first. This solves the problem of untimely response to client-submitted jobs when the task volume is large. All unexecuted tasks will enter the public task game queue and be executed after negotiation by each Worker node. Only when the Worker side has not executed the task for some reason will it return it to the Master side, and the Master may dispatch the task to the private task dispatch queue.

[0074] 2. Private task dispatch queue

[0075] Used in task dispatch mode, in the scheduling system, the number of private task dispatch queues is consistent with the number of Worker nodes. Each Worker corresponds to a private task dispatch queue and is responsible for completing all tasks in this queue. The tasks in this queue can only be taken out and executed by its corresponding Worker, and other Worker nodes have no access. Each private task dispatch queue has only one message producer, which is the Master, and only one consumer, which is the Worker that owns this queue. The priority of the private queue is higher than the public queue. When there are tasks in both the private queue and the public queue, the tasks in the private queue are executed first. Only when there are no tasks in the private queue will the tasks be taken out from the public queue for execution.

[0076] 3. Resource Model

[0077] Instead of using the concept of slots established for resources in MapReduce, resources are remodeled based on their multidimensionality. The resource model established reflects the CPU, memory, disk I / O, and number of tasks of the current computing node. Different resource entities are constructed to be comparable and additive. The resource type (Resource class) established is an abstract class with several attributes such as the number of CPU cores, memory size, and I / O bandwidth, as well as methods for adding, subtracting, and comparing resources. Subclasses inherit and override these methods.

[0078] The comparability of resource entities is between the resources required for the task and the remaining resources of the computing nodes, as well as between different computing nodes. When the Master node assigns a task, it compares the resource status of each computing node and determines whether the node resources are sufficient to perform the current task. By comparing, the most suitable node that meets the requirements is selected to assign the task. When the Worker node receives the task, it performs task evaluation to check whether the resources required by the task can be met by the current node. Assume that the vector R = (r1, r2, ..., r n ) represents the various dimensions of resources. Consider the number of CPU cores and memory size (Mb). Assume R1 = (10, 2000) and R2 = (12, 1500). R1 and R2 are superior in CPU and memory, respectively, making it difficult to determine which of these two resource entities possesses more resources. The following strategies address this issue.

[0079] 4. Resource Node Pool Model

[0080] The node pool is a container that stores the IP and resource information of all computing nodes. The Master is responsible for management and maintenance. The node pool manages the computing nodes. When computing node resource information is sent, the node pool is responsible for updating this information. When the Master is dispatching tasks, the node pool selects appropriate computing nodes for task dispatch based on the task's resource requirements. This application establishes two node pool models: a convenient set model and a multi-hop space model.

[0081] (1) Convenient Collection Model

[0082] Put all nodes into the hash table, calculate the primary key through the hash function, get the Hashcode, and then determine the position of the key-value pair in the table based on the Hashcode. Use the node IP address as the primary key and store it in the hash table. When new resource information arrives, directly find the node object of the corresponding IP and update its resource information. This process will be very fast. When a task arrives, traverse the entire hash table and find all nodes that meet the resources required for the task through comparative calculation. Select a node to perform the task. For different needs, two methods are constructed: one is to only One method is to select any one of all nodes that meet the requirements, and stop traversing when a node that meets the requirements is found during the traversal; the other method is to select the best one among all nodes that meet the requirements according to a certain rule, and all nodes must be traversed. During the traversal, the current best node that meets the requirements is recorded while traversing. Finally, the node that needs to be found is obtained after the traversal is completed. After obtaining the node, the node is deleted from the table or the node resource information is subtracted from the task information as the latest resource information of the node to prevent too many tasks from being assigned to the same node in a short period of time, thereby causing the problem of overload on a certain node. The task dispatching process is as follows Figure 1 shown.

[0083] (2) Multi-hop spatial model

[0084] Within the node pool, multiple skip tables are used to store computing nodes. The number of skip tables is equal to the number of resource dimensions. Each skip table stores and manages nodes according to a certain dimension of the resource. The node skip table corresponding to the number of CPU cores (referred to as the CPU skip table) stores all nodes in ascending order (or descending order) based on the number of CPU cores. This rule is maintained when adding, updating, or deleting elements.

[0085] When new node resource information arrives, it is determined whether the node exists in the CPU jump table (compared according to the IP or object address, using a hash table to achieve this). If the node does not exist in the jump table, the node is inserted into the CPU jump table, otherwise the original node information is updated and moved to the appropriate position. The resource information update process of the multi-hop space model is as follows: Figure 2 shown.

[0086] When a task arrives, the node pool dispatches nodes to execute according to the resources required for the task. For a task, the dispatched nodes must meet the requirements in all dimensions of resources. The nodes must ensure that the CPU, memory, and IO bandwidth resources are sufficient to complete the task. In order to find all nodes that meet the requirements, first construct a set to store all the nodes, and traverse the jump tables in the node pool one by one. The traversal should start from the node with the least number of resources and end at the first node whose resources meet the requirements. When a node in the jump table whose resources do not meet the task requirements is found, it is removed from the set. After processing the last jump table, the remaining elements in the set are all the nodes that meet the task requirements. The task dispatching process of the multi-hop space model is as follows: Figure 3 shown.

[0087] (2) Scheduling process

[0088] Completing remote sensing data production operations through a distributed scheduling system requires a series of processing procedures, including: job submission and decomposition, task queuing, task execution, result collection, and task exception handling.

[0089] 1. Assignment submission and decomposition

[0090] The job is submitted by the client, such as Figure 4 As shown, the submitted jobs are sent to the server through Web Service, which provides a WebService interface for job submission. External programs can submit tasks regularly through this interface, reducing labor and labor costs.

[0091] When a client submits a job, it sets the job priority. The job priority attribute is passed to its decomposed tasks. The task priority not only determines the order in which tasks are executed, but also determines whether a task has the right to preempt other tasks.

[0092] Job information via the Web The service passes the job to the master. After the job is submitted, the master receives it and verifies whether the submitted request complies with the interface specifications, including whether the request content is complete, the data path for request processing exists, and whether the data format is correct. If verification passes, the job is placed in the job queue according to priority and submission time. The master also checks whether the user exists in the queuing user table queuingUsers maintained by the master. If not, the user, their jobCapacity, and their concurrently executable jobs are added to the table. At this point, the system's total concurrently executable jobs (jobCapacity) equals the sum of the jobCapacity and the jobCapacities of all users in queuingUsers. In another thread, the master monitors the entry and exit of jobs in the queue. When a job enters the queue, it checks the runningJobs and jobCapacity of the user corresponding to the job. If the former is less than the latter, it checks the jobs submitted by the user in the queue from the front to the back and removes them for execution. When a job is removed from the queue, it checks whether there are other jobs submitted by the user in the queue. If not, the user is removed from the queuingUsers table. jobCapacity corresponds to the concurrently executable number of jobs submitted by external programs through the WebService interface.

[0093] When the Master takes a job from the job queue, it first preprocesses the job and then decomposes the job. Job preprocessing requires parsing the process and data information to be executed for processing the current data. The process records the steps of processing the data, and the data information records the data name, specific path, and number of data frames of the data to be processed. After the preprocessing is completed, the job is decomposed. Job decomposition first obtains multiple steps according to the process of processing data. The order between each step determines the dependency relationship of the decomposed tasks. The tasks corresponding to the subsequent steps need to be executed after all the tasks corresponding to the previous steps are completed. The job is decomposed into tasks, and the input and output data paths and execution parameter information are configured for the tasks.

[0094] Use XML files (called orders) to record information, pass the XML path as a parameter to the job program, each XML order corresponds to a data processing, the XML order and its corresponding processing program information are encapsulated to obtain the corresponding task, and tasks that have no dependencies between each other will be placed in the public task game queue. The job submission and decomposition process is as follows Figure 5 shown.

[0095] 2. Task Queuing

[0096] After being decomposed by the Master, tasks are placed in task queues. Task queues include two types: public queues and private queues. Jobs submitted by users are decomposed into tasks and placed in public queues. Tasks returned to the Master by Workers that have not been executed due to insufficient resources or other abnormal situations and need to be re-queued are placed in private queues. The number of private queues is equal to the number of Worker nodes. Each private queue has a corresponding Worker node. The Master node decides which private queue to place the task based on the resource status of the collected Worker nodes.

[0097] The resource information collected by the Master is sent by the Worker node through the Heartbeat. The Master node maintains a node pool for all computing nodes based on the resource and task conditions. Initially, the node pool is empty. When the Master node starts to collect the first resource information sent by the Worker node, it puts this node and resource information into the node pool. Later, when it receives new resource information, it compares the resources and inserts them into the appropriate position in the node pool or updates the elements in the node pool. It selects the node from the node pool and puts the task into the private queue corresponding to the selected node. It immediately updates the resource information corresponding to the node and updates the position of the node in the node pool at the same time. It determines that the tasks put into the private queue have used the required resources and the number of remaining resources is reduced. This prevents some nodes in the node pool from being assigned too many tasks in a short period of time, so that before the new resource information arrives, load imbalance problems may occur due to unreasonable task allocation. Figure 6 The process of queuing tasks.

[0098] 3. Task Execution

[0099] This is done by Worker nodes. All Worker nodes monitor the public task game queue and their own private task dispatch queues. When a new task is available in the public queue, all Worker nodes take the task from the public queue. Each node waits a short time after taking out a task before trying to take a new task. This increases the probability of other nodes taking new tasks, making the number of tasks executed by each node more balanced, and preventing one or several nodes from taking out too many tasks in a short period of time.

[0100] After the Worker takes out the task, it first evaluates the task. If the number of resources required by the task is too large and the current node resources are completely unable to meet the needs of completing the task (this situation may occur in a heterogeneous environment, when some nodes have poor computing performance and fewer resources and the taken tasks require more resources), the Worker node returns the task to the Master node. The Master node then dispatches the task to the node's private queue based on the resource status of each node. If the number of resources required by the task is temporarily not met by the current node due to executing other tasks, but it is expected to be met after waiting for a period of time, the Worker node will wait until the resources are sufficient before executing this task. During the waiting period, the Worker will not take out new tasks from the public task game queue for execution. If, due to some abnormal reasons, the Worker cannot wait for the required resources, the task will be returned to the Master node after the set timeout period.

[0101] If the number of resources required by the task can be met by the current node, the task will be executed, and the task pool will be updated to add the task to the task pool. The task pool maintains the status of all tasks being executed by the current node. During the task execution, if an exception occurs that causes the task to fail or time out, the task will be re-executed or reported directly to the Master node based on user needs or task characteristics. After the task is completed, a task completion notification will be sent to the Master node, and the task pool will be updated at the same time to remove the task from the task pool.

[0102] For tasks in the private queue, the Worker node monitors, retrieves and executes them. The private queue has a higher priority than the public queue. When tasks appear in the private queue, the Worker node stops retrieving tasks from the public queue and only retrieves tasks from the private queue. The processing flow after retrieving tasks is the same as that of the public queue. When all tasks in the private queue have been retrieved, the Worker node resumes retrieving tasks from the public queue.

[0103] 4. Results Collection

[0104] The Master node collects task result information and decides the process execution based on the result information. After the Worker node completes the task, it writes the task result to the database and sends the task ID to the Master through different queues based on different task results. The processed tasks will be placed in one queue, and the unprocessed tasks will be placed in another queue. For the unprocessed task queue, the Master will take the unprocessed tasks out of the queue and then dispatch the tasks to the private queue for re-processing by the Worker node. For the processed task queue, after receiving the task ID, the Master determines which step of the process the task belongs to, and then determines whether all tasks of the step have been completed. If the execution is completed, the step is marked as finished and the next step is continued. If the step is the last step of the process, the process execution is marked as completed, that is, the current job execution is completed. The correspondence between data production jobs, steps and tasks is as follows: Figure 7 shown.

[0105] 5. Task exception handling

[0106] A timeout is set in advance for each task. For the Worker side, when the task being executed is not completed after the timeout period, the task will be terminated and returned to the Master side. The Master side can dispatch the timed-out task to a private task dispatch queue for redoing, or send the task status to the client, and the user will analyze the reason for the timeout and decide whether to redo it. On the other hand, when a task fails, the Worker side immediately redoes the failed task to prevent task failure due to unexpected circumstances. Compared with returning the failed task to the Master node and then the Master node deciding whether to redo and dispatch the node to execute the task, the Worker immediately redoing the failed task is not only more efficient, but also saves network bandwidth resources and reduces the burden on the Master node.

[0107] (3) Scheduling algorithm

[0108] In response to the needs of remote sensing data production operations, a task game dispatching algorithm is constructed. When the resources of each computing node are sufficient, the task game strategy is used to improve the efficiency of task distribution. When resources are insufficient, the task dispatching strategy is used to improve the load balancing rate and resource utilization, and avoid resource waste.

[0109] 1. Task Game

[0110] When no jobs are submitted to the production system, all computing nodes are idle and have sufficient resources. If a job is submitted to the system, all computing nodes are capable of executing the task, allowing all workers to bid for tasks from the public task bidding queue.

[0111] The task negotiation mode is applied when computing node resources are idle. When the Worker node detects that there are no tasks running in the private task dispatch queue, it negotiates for tasks from the public task queue. After obtaining the task through negotiation, it evaluates whether the current node resources are sufficient to execute the task. If not, it waits until the resources are sufficient or returns the task to the Master. If sufficient, it executes the task asynchronously and then evaluates whether the current node resources are sufficient to execute the next task (assuming that the next task taken out is of the same type and uses the same resource size). If not, it waits until the resources are sufficient or times out, and then continues to negotiate for tasks from the public queue. If sufficient, it directly negotiates for tasks from the public queue.

[0112] 2. Task Assignment

[0113] Task dispatch is completed by the Master node. The dispatch strategy is related to the computing node resources. When there are insufficient resources or other abnormal situations, the Worker node returns unprocessed tasks to the Master node. The Master node dispatches tasks when it receives unprocessed tasks.

[0114] In the task dispatching mode, the Master maintains a node pool that manages global computing node information, including IP addresses and resource information. When resource information is sent to the Master by the Worker node through the Heartbeat message, the Master will update the node information in the pool. When a task needs to be dispatched, the Master takes a node that meets the task resource requirements from the node pool, dispatches the task to the node for execution, and updates the node resource information.

[0115] It is stipulated that tasks assigned by the Master have a higher priority than ordinary tasks, that is, the priority of each Worker's private queue is higher than the public queue as a whole. When a task appears in a Worker's private queue, the Worker will stop taking tasks from the public queue and only take tasks from the private queue until all tasks in the private queue are taken and there are idle resources, and then resume taking tasks from the public queue.

[0116] 3. Algorithm Performance Analysis

[0117] The algorithm uses two modes in two situations to achieve flexible system processing for different tasks and resources. The task game mode allows Worker nodes to be partially autonomous and reduces the load on the Master node. This can greatly reduce the pressure on the Master when there are many nodes and tasks. The task dispatch mode is similar to the traditional MapReduce strategy, where the Master node dispatches tasks to the Worker nodes for execution. This is because tasks returned to the Master for dispatch are generally tasks that have not been processed by a Worker node. Reasons for non-processing include requiring too many resources, which may not be met by the Worker node, or some abnormality in the task. Therefore, a Master node with global resource information is needed to coordinate and dispatch the appropriate node to complete this task. At the same time, because tasks dispatched to private queues have a relatively high priority, dispatched tasks can be executed first, which is important for maintaining the overall efficiency of the system.

[0118] Since the algorithm's task game mode does not allocate resources through the Master node, the algorithm has a greater advantage when handling a large number of small tasks (tasks that use fewer resources). However, when handling a large number of large tasks, it is very likely that the computing node resources are insufficient and cannot meet the task's resource requirements, so the task will be returned to the Master for allocation. If such situations occur too often, it will increase the burden on the Master and affect scheduling efficiency.

[0119] 3. Experimental Results and Analysis

[0120] Implement the model and algorithm proposed in this application to verify their correctness and effectiveness. Build a cluster environment and run remote sensing data production jobs multiple times. Analyze the correctness of the results, job time, job parallelism, and load balancing to draw final experimental conclusions.

[0121] (1) Experimental environment configuration

[0122] The server cluster environment used for experimental verification consists of seven servers, one of which serves as the master node, deploying the master program, and the remaining six as worker nodes, deploying the worker program. This is also the environment used in actual production. The cluster is interconnected by high-speed Gigabit Ethernet, and all data is stored on a scalable 40TB disk array. Several client nodes, each deploying client programs, are also located within the local area network. Each client node can access the server cluster, and all server and client nodes can access data on the disk array. All master and worker nodes in the server cluster run the 64-bit Red Hat Enterprise Linux 6.5 operating system. The server programs are developed in Java, running JDK 1.8.

[0123] (2) Analysis of experimental results

[0124] In order to verify the correctness and effectiveness of the algorithm proposed in this application, and to prove that the task scheduling system can improve production efficiency while ensuring the quality of data production, a real remote sensing data production operation is used for testing. The operation used in the test is the processing operation of the data transmitted by a certain video satellite. The operation process includes the following steps: LO-level data cataloging and processing, image cloud judgment processing, precise attitude calculation, L1 product production, L1 radiation quality inspection, L1 geometric quality inspection and L2 product production. The initial data processed is binary data with the suffix DAT. Each DAT data corresponds to a video frame. The ultimate goal is to produce the initial data into an L2-level product. The experiment uses multi-track data for processing and calculation, verifies the correctness of the processed result data, and analyzes the average processing time and load balancing performance of the operation.

[0125] 1. Analysis of the correctness of the result data

[0126] The correctness of the result data is a prerequisite for the availability of the scheduling system. When processing tasks, the scheduling system must first ensure that the dependencies between tasks within the job are not destroyed, and secondly, ensure the normal execution of all data processing programs to ensure the correctness of the result data. Many factors affect the normal operation of data processing programs. Internal system factors include insufficient resource allocation, unreasonable input parameters, incorrect intermediate data, path errors, etc. External factors include data processing program errors, operator misoperation that prematurely terminates the program, operating system errors, hardware errors, etc. In addition, some data-related anomalies, such as incomplete initial data and data occupied by other programs, can also cause incorrect result data. To measure the correctness of the result data, statistics on the errors in each job task and the accuracy of the result data are counted for analysis.

[0127] Figure 8Statistics were collected for the production results of 20 data production jobs, including the total number of tasks, number of errors, error rate, and total number of frames, error frames, and error rate for each job. The experimental results show that the task error rate and data error rate for most jobs are below 1%. Furthermore, an inspection of the erroneous jobs revealed that the cause of the errors is generally due to incomplete initial data in the first frame (possibly related to the image capture method). The data processing program reports an error when processing incomplete data, resulting in task failure and incorrect final data production. Excluding failures caused by initial data errors, the task failure rate caused by the scheduling system is very low, meeting production requirements in terms of data correctness, demonstrating that the scheduling framework is reasonable and effective.

[0128] 2. Job processing time analysis

[0129] The ultimate goal of designing a scheduling system is to improve the efficiency of remote sensing data processing and reduce the average processing time. Therefore, it is of great significance to examine and analyze the processing time.

[0130] We analyzed the processing time of multiple jobs for data of varying sizes. The resulting data shows that different jobs can be processed in parallel, and within each job, different tasks also run in parallel to some extent. This can shorten the overall processing time and improve job processing efficiency. To demonstrate the scheduling system's improved data processing efficiency, we manually tested the time it takes each data processing program to process a single frame of data. We then calculated the time it takes to process multiple frames of data in a serial manner and compared this time with the actual processing time of the jobs in the experiment. The comparison results show that the actual job execution time is much shorter than the theoretical serial execution time, demonstrating the significant advantages of distributed and parallel processing.

[0131] In order to more accurately reflect the parallel execution efficiency of the computing nodes, a time statistical analysis is performed on a single step of the job. The total time taken to complete all tasks in the step and the average time taken for each task are counted and compared with the time taken by the data processing program to process a single frame of data. The experimental results are as follows: Figure 9 As shown;

[0132] The step time here refers to the time spent on executing the entire step, the total time is the total time obtained by testing this step several times, the average time refers to the average time relative to the number of tasks, and the total task time and average time refer to the total time spent on executing the task relative to the average time of the number of tasks. Comparing the step time and the task time, it can be seen that the average time of some steps is much shorter than the task time. This is because some steps have multiple tasks in parallel, so the average time for each task is much shorter, which also reflects the advantage of parallelism.

[0133] 3. Load balancing analysis

[0134] Several jobs were submitted for execution over a period of time. After completion, the number of tasks processed by each compute node was analyzed to determine the load balancing status of each compute node. Resource usage of the master node was also measured, and the changes in master node load as the number of jobs increased were analyzed. The node load balancing generally met requirements, with no significant resource waste.

[0135] The CPU and memory usage of the master node remained relatively stable over time, and both CPU and memory resource consumption were very low. Although the node's memory usage approached 9000MB, monitoring the Java virtual machine revealed that the master program consumed only tens of MB of memory, which is consistent with theoretical expectations. The low master node load further ensures the stability of the master program, demonstrating the improvements that the proposed framework offers over MapReduce.

[0136] The framework proposed in this application can improve the efficiency of remote sensing data production to a certain extent while ensuring production quality.

Claims

1. An efficient scheduling method for cloud computing tasks in remote sensing big data production, characterized by: The MapReduce framework was improved to establish a new scheduling framework suitable for remote sensing data production. The improved scheduling framework still adopts a master / slave architecture, with the control Master node and the computing Worker node completing the work. The Master receives job requests submitted by the client, parses and verifies the requests, and then breaks the job into tasks. The tasks are then passed to the Worker through the task queue, and the task status information sent by the Worker is collected. The improvement to the MapReduce framework is to give the Worker decision-making power, rather than relying entirely on the Master to dispatch tasks, thus overcoming the problem of overloaded control nodes in the MapReduce framework. The Worker collects resource and task information on the local machine to decide whether to execute the next task, and also sends this information to the Master through the Heartbeat mechanism. When the Worker is short of resources, some tasks requiring more resources are sent to the Master, which dispatches them to nodes with more idle resources in the cluster. The task game dispatch algorithm is constructed. The algorithm framework is based on two types of task queues: a public task game queue and a private task dispatch queue to improve the MapReduce framework resource model. 1) Based on the production characteristics of remote sensing data, MapReduce is used to process distributed tasks. Each data processing step of the remote sensing data processing plug-in is abstractly encapsulated into a task. Remote sensing data processing does not involve Reduce work and does not execute ReduceTasks. A Master node decomposes the job into tasks and then dispatches the tasks to multiple Worker nodes for execution. 2) In terms of scheduling strategy, a task game dispatching algorithm is established based on the characteristics of satellite data processing tasks. That is, the two scheduling modes of task game and task dispatch are coordinated to perform overall task scheduling. First, the control node places the task into the public task game queue. When the resources of each computing node are sufficient, the task game mode is adopted. Each computing node bids for a task from the public task game queue and immediately executes the task obtained through the game. As the number of tasks executed simultaneously by a computing node increases, the remaining resources of the computing node gradually decreases until it is insufficient to complete the execution of the next task. At this time, the computing node chooses to wait until the resources are sufficient or return the task to the control node based on the required resources and the total available resources of the node. If it chooses to return the task to the control node, the task is scheduled according to the task dispatching mode. The control node dispatches the task to the appropriate computing node through the computing node's private queue based on the resource information collected from each computing node. At this time, the computing node that has obtained the task has priority to execute the assigned task, that is, it stops bidding for new tasks from the public queue until all tasks in the private queue are taken out.

2. The method for efficiently scheduling cloud computing tasks for remote sensing big data production according to claim 1 is characterized in that: Public task game queue: used in task game mode, there is only one in the entire system. There is only one message producer for the public queue, which is the Master, and there are multiple consumers, namely the Workers. When a message is consumed by a consumer, other consumers will no longer receive this message. The public task game queue is set as a priority queue. When the Master receives a job manually submitted by the client operator, it puts the job into the job queue with a higher priority. After taking out the job, it decomposes the job to obtain tasks, passes the job priority to the tasks, and puts the tasks into the task queue with a higher priority so that the tasks can be executed first. This solves the problem of untimely response to client-submitted jobs when the task volume is large. All unexecuted tasks will enter the public task game queue and be executed by each Worker node after game. Only when the Worker side has not executed the task for some reason will it be returned to the Master side by the Worker side. The Master may dispatch the task to the private task dispatch queue.

3. The efficient scheduling method for cloud computing tasks for remote sensing big data production according to claim 1 is characterized in that: Private task dispatch queue: used in task dispatch mode. In the scheduling system, the number of private task dispatch queues is consistent with the number of Worker nodes. Each Worker corresponds to a private task dispatch queue and is responsible for completing all tasks in this queue. Tasks in this queue can only be taken out and executed by its corresponding Worker. Other Worker nodes have no access. Each private task dispatch queue has only one message producer, which is the Master, and only one consumer, which is the Worker that owns this queue. The priority of the private queue is higher than the public queue. When there are tasks in both the private queue and the public queue, the tasks in the private queue are executed first. Only when there are no tasks in the private queue will the tasks be taken out from the public queue for execution.

4. The method for efficiently scheduling cloud computing tasks for remote sensing big data production according to claim 1 is characterized in that: Resource model: Instead of using the slot concept established by MapReduce for resources, resources are remodeled based on their multidimensionality. The established resource model reflects the CPU, memory, disk I / O, and task quantity information of the current computing node. Different resource entities are constructed to be comparable and additive. The established resource type Resource class is an abstract class with several attributes such as the number of CPU cores, memory size, and I / O bandwidth, as well as resource addition, subtraction, and comparison methods. Subclasses inherit and override these methods. The comparability of resource entities is between the resources required for the task and the remaining resources of the computing nodes, as well as between different computing nodes. When the Master node assigns a task, it compares the resource status of each computing node and determines whether the node resources are sufficient to perform the current task. By comparing, the most suitable node that meets the requirements is selected to assign the task. When the Worker node receives the task, it performs task evaluation to check whether the resources required by the task can be met by the current node. Assume that the vector R = (r1, r2, ..., r n ) represents the dimensions of resources. Considering the number of CPU cores and memory size (Mb), let R1 = (10, 2000) and R2 = (12, 1500). Then R1 and R2 are better in CPU and memory, respectively.

5. The efficient scheduling method for cloud computing tasks for remote sensing big data production according to claim 1 is characterized in that: Resource node pool model: The node pool is a container that stores the IP and resource information of all computing nodes. The Master is responsible for management and maintenance. The node pool manages the computing nodes. When computing node resource information is sent, the node pool is responsible for updating this information. When the Master is dispatching tasks, the node pool selects appropriate computing nodes for task dispatch based on the task's resource requirements. This application establishes two node pool models: a convenient set model and a multi-hop space model. (1) Convenient Collection Model Put all nodes into the hash table, calculate the primary key through the hash function, get the Hashcode, and then determine the position of the key-value pair in the table based on the Hashcode. Use the node IP address as the primary key and store it in the hash table. When new resource information arrives, directly find the node object of the corresponding IP and update its resource information. When a task arrives, traverse the entire hash table and find all nodes that meet the resources required for the task through comparison and calculation. Select a node to perform the task. For different needs, two methods are constructed: one is to only select all nodes. Among any of the nodes that meet the requirements, the traversal stops when a node that meets the requirements is found during the traversal; the other method requires that the best one be selected according to a certain rule among all the nodes that meet the requirements. All nodes must be traversed. During the traversal, the current best node that meets the requirements is recorded while traversing. Finally, the node to be found is obtained after the traversal is completed. After obtaining the node, the node is deleted from the table or the node resource information is subtracted from the task information as the latest resource information of the node to prevent too many tasks from being assigned to the same node in a short period of time, thereby causing the problem of overload of a certain node. (2) Multi-hop spatial model Within the node pool, multiple skip tables are used to store computing nodes. The number of skip tables is equal to the number of resource dimensions. Each skip table stores and manages nodes according to a certain dimension of the resource. The node skip table corresponding to the number of CPU cores stores all nodes in ascending order of CPU core count. This rule remains unchanged when adding, updating, or deleting elements. When new node resource information arrives, it is determined whether the node exists in the CPU skip list. If the node does not exist in the skip list, the node is inserted into the CPU skip list. Otherwise, the original node information is updated and moved to the appropriate position. When a task arrives, the node pool dispatches nodes to execute according to the resources required for the task. For a task, the dispatched nodes must meet the requirements in all dimensions of resources. The nodes must ensure that the CPU, memory, and IO bandwidth resources are sufficient to complete the task. In order to find all nodes that meet the requirements, first construct a set to store all the nodes, and traverse the jump tables in the node pool one by one. The traversal should start from the node with the least number of resources and end at the first node whose resources meet the requirements. When a node in the jump table whose resources do not meet the task requirements is found, it is removed from the set. After processing the last jump table, the remaining elements in the set are all the nodes that meet the task requirements.

6. The method for efficiently scheduling cloud computing tasks for remote sensing big data production according to claim 1 is characterized in that: Job submission and decomposition: Jobs are submitted by the client and sent to the server through the Web Service. A Web Service interface for job submission is provided, and external programs submit tasks regularly through this interface. When a client submits a job, it sets the job priority. The job priority attribute is passed to its decomposed tasks. The task priority not only determines the order in which tasks are executed, but also determines whether a task has the right to preempt other tasks. Job information via the Web The service passes the job to the Master. After the job is submitted, the Master receives and verifies whether the submitted request complies with the interface specifications, including whether the request content is complete, the data path for request processing exists, and whether the data format is correct. If the verification passes, the job is placed in the job queue according to priority and submission time. The Master also checks whether the user exists in the queuing user table queuingUsers maintained by the Master. If not, the user and their jobCapacity and runningJobs information are added to the table. At this time, the total number of jobs that can be executed in parallel in the system (jobCapacity) is equal to the sum of jobCapacity and the jobCapacities of all users in queuingUsers. In another thread, the Master monitors the entry and exit of jobs in the queue at any time. When a job enters the queue, it checks the runningJobs and jobCapacity of the user corresponding to the job. If the former is less than the latter, it checks the jobs submitted by the user in the queue from the front to the back and removes them for execution. When a job is removed from the queue, it checks whether there are other jobs submitted by the user in the queue. If not, the user is removed from the queuingUsers table. jobCapacity corresponds to the number of jobs that can be executed in parallel by external programs submitted through the WebService interface. When the Master takes a job from the job queue, it first preprocesses the job and then decomposes the job. Job preprocessing requires parsing the process and data information to be executed for processing the current data. The process records the steps of processing the data, and the data information records the data name, specific path, and number of data frames of the data to be processed. After the preprocessing is completed, the job is decomposed. Job decomposition first obtains multiple steps according to the process of processing data. The order between each step determines the dependency relationship of the decomposed tasks. The tasks corresponding to the subsequent steps need to be executed after all the tasks corresponding to the previous steps are completed. The job is decomposed into tasks, and the input and output data paths and execution parameter information are configured for the tasks.

7. The method for efficiently scheduling cloud computing tasks for remote sensing big data production according to claim 1 is characterized in that: Task queuing: Tasks are decomposed by the Master and placed in task queues. Task queues include two types: public queues and private queues. Jobs submitted by users are decomposed into tasks and placed in public queues. Tasks returned to the Master by Workers that are not executed due to insufficient resources or other abnormal conditions and need to be re-queued are placed in private queues. The number of private queues is equal to the number of Worker nodes. Each private queue has a corresponding Worker node. The Master node decides which private queue to place the task based on the resource status of the collected Worker nodes. The resource information collected by the Master is sent by the Worker node through the Heartbeat. The Master node maintains a node pool for all computing nodes based on the resource and task conditions. Initially, the node pool is empty. When the Master node starts to collect the first resource information sent by the Worker node, it puts this node and resource information into the node pool. Later, when it receives new resource information, it compares the resources and inserts them into the appropriate position in the node pool or updates the elements in the node pool. It selects the node from the node pool and puts the task into the private queue corresponding to the selected node. It immediately updates the resource information corresponding to the node and updates the position of the node in the node pool at the same time. It determines that the tasks put into the private queue have used the required resources and the number of remaining resources is reduced. This prevents some nodes in the node pool from being assigned too many tasks in a short period of time, so that before the new resource information arrives, unreasonable task distribution causes load imbalance problems.

8. The method for efficiently scheduling cloud computing tasks for remote sensing big data production according to claim 1 is characterized in that: Task execution: This is completed by Worker nodes. All Worker nodes monitor the public task game queue and their own private task dispatch queue. When a new task is available in the public queue, all Worker nodes take the task from the public queue. Each node waits a short time after taking out a task before trying to take a new task. This increases the probability of other nodes taking new tasks, makes the number of tasks executed by each node more balanced, and prevents one or several nodes from taking out too many tasks in a short period of time. After the Worker takes out the task, it first evaluates the task. If the number of resources required for the task is too large and the current node resources are completely unable to meet the needs of completing the task, the Worker node returns the task to the Master node. The Master node then dispatches the task to the node's private queue based on the resource status of each node. If the number of resources required for the task is temporarily not met by the current node due to executing other tasks, but it is expected to be met after waiting for a period of time, the Worker node will wait until the resources are sufficient before executing this task. During the waiting period, the Worker will not take out new tasks from the public task game queue for execution. If due to some abnormal reasons, the Worker cannot wait for the required resources, the task will be returned to the Master node after the set timeout period. If the number of resources required by the task can be met by the current node, the task will be executed, and the task pool will be updated to add the task to the task pool. The task pool maintains the status of all tasks being executed by the current node. During the task execution, if an exception occurs that causes the task to fail or time out, the task will be re-executed or reported directly to the Master node based on user needs or task characteristics. After the task is completed, a task completion notification will be sent to the Master node, and the task pool will be updated at the same time to remove the task from the task pool. For tasks in the private queue, the Worker node monitors, retrieves and executes them. The private queue has a higher priority than the public queue. When tasks appear in the private queue, the Worker node stops retrieving tasks from the public queue and only retrieves tasks from the private queue. The processing flow after retrieving tasks is the same as that of the public queue. When all tasks in the private queue have been retrieved, the Worker node resumes retrieving tasks from the public queue.

9. The method for efficiently scheduling cloud computing tasks for remote sensing big data production according to claim 1 is characterized in that: Result collection: The Master node collects task result information and decides the process execution based on the result information. After executing the task, the Worker node writes the task result to the database and sends the task ID to the Master through different queues according to different task results. The processed tasks will be placed in one queue, and the unprocessed tasks will be placed in another queue. For the unprocessed task queue, the Master will take the unprocessed tasks out of the queue and then dispatch the tasks to the private queue, which will be taken out and processed again by the Worker node. For the processed task queue, after receiving the task ID, the Master determines which step of the process the task belongs to, and then determines whether all tasks of the step have been completed. If the execution is completed, the step is marked as finished and the next step is continued. If the step is the last step of the process, the process execution is marked as completed, that is, the current job execution is completed.

10. The method for efficiently scheduling cloud computing tasks for remote sensing big data production according to claim 1 is characterized in that: Scheduling algorithm: Build a task game dispatching algorithm. When the resources of each computing node are sufficient, the task game strategy is used to improve the efficiency of task distribution. When the resources are insufficient, the task dispatching strategy is used to improve the load balancing rate and resource utilization. 1-Task Game When no jobs are submitted to the production system, each computing node is idle. If a job is submitted to the system, each computing node has the ability to execute the task, allowing all workers to bid for tasks from the public task bidding queue. The task negotiation mode is applied when computing node resources are idle. When the Worker node detects that there are no tasks running in the private task dispatch queue, it negotiates for tasks from the public task queue. After obtaining the task through negotiation, it evaluates whether the current node resources are sufficient to execute the task. If not, it waits until the resources are sufficient or returns the task to the Master. If sufficient, it executes the task asynchronously and then evaluates whether the current node resources are sufficient to execute the next task. If not, it waits until the resources are sufficient or times out, and then continues to negotiate for tasks from the public queue. If sufficient, it directly negotiates for tasks from the public queue. 2-Task Assignment Task dispatch is completed by the Master node. The dispatch strategy is related to the computing node resources. When there are insufficient resources or other abnormal situations, the Worker node returns unprocessed tasks to the Master node. The Master node dispatches tasks when it receives unprocessed tasks. In the task dispatching mode, the Master maintains a node pool that manages global computing node information, including IP addresses and resource information. When resource information is sent to the Master by the Worker node through the Heartbeat message, the Master will update the node information in the pool. When a task needs to be dispatched, the Master takes a node that meets the task resource requirements from the node pool, dispatches the task to the node for execution, and updates the node resource information. It is stipulated that tasks assigned by the Master have a higher priority than ordinary tasks, that is, the priority of each Worker's private queue is higher than the public queue as a whole. When a task appears in a Worker's private queue, the Worker will stop taking tasks from the public queue and only take tasks from the private queue until all tasks in the private queue are taken and there are idle resources, and then resume taking tasks from the public queue.

Images