Scheduling method and apparatus, electronic device, and storage medium

Abstract

This disclosure provides a scheduling method, apparatus, electronic device, and storage medium. Applicable to the field of remote sensing data processing, the scheduling method includes: responding to a data processing request for remote sensing data, determining at least one target processing flow from a processing flow library based on the data attributes of the remote sensing data; determining a processing algorithm for at least one task based on the task attributes of multiple tasks included in the at least one target processing flow; associating the at least one task with its corresponding processing algorithm to obtain at least one task instance; and allocating the multiple task instances to multiple resource nodes based on the task attributes of the tasks and the resource requirement information of the multiple task instances.

Description

Technical Field

[0001] This disclosure relates to the field of remote sensing data processing, and more specifically, to a scheduling method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of remote sensing data acquisition technology, the amount of various types of remote sensing data, such as optical remote sensing data, microwave remote sensing data, and hyperspectral remote sensing data, has experienced explosive growth. These data are characterized by massive volume, high dimensionality, and complexity, making efficient processing and analysis a major challenge in the field of remote sensing. Traditional single-system computing clusters are no longer sufficient to meet the needs of large-scale remote sensing data processing.

[0003] In realizing the present invention, the inventors discovered that in the related technologies, when processing large-scale and multi-type remote sensing data, the use of computing resources in computing devices to process remote sensing data often results in a waste of computing resources. How to reasonably allocate computing resources to different remote sensing data processing tasks has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the present disclosure provides a scheduling method, apparatus, electronic device and storage medium.

[0005] One aspect of this disclosure provides a scheduling method, comprising:

[0006] According to embodiments of this disclosure, in response to a data processing request for remote sensing data, at least one target processing flow is determined from a processing flow library based on the data attributes of the remote sensing data; a processing algorithm for at least one task is determined based on the task attributes of multiple tasks included in the at least one target processing flow; the at least one task is associated with the corresponding processing algorithm to obtain at least one task instance; and multiple task instances are allocated to multiple resource nodes based on the task attributes of the task and the resource requirement information of the multiple task instances.

[0007] According to embodiments of this disclosure, determining a processing algorithm for at least one task based on the task attributes of multiple tasks included in at least one target processing flow includes: splitting at least one target processing flow to obtain multiple tasks that are dependent on each other; and determining a processing algorithm for at least one task based on the task attributes of the tasks.

[0008] According to embodiments of this disclosure, processing algorithms corresponding to multiple tasks are determined from a processing algorithm library based on the task attributes of the tasks.

[0009] According to embodiments of this disclosure, the data attributes of remote sensing data include data type and processing mode; determining at least one target processing flow from a processing flow library based on the data attributes of remote sensing data includes: filtering from the processing flow library based on multiple processing modes of remote sensing data to obtain multiple first processing flows; and determining at least one target processing flow from the multiple first processing flows based on the data type.

[0010] According to embodiments of this disclosure, multiple work task instances are allocated to multiple resource nodes based on the task attributes of the work tasks and the resource requirement information of multiple work task instances, including: determining the processing order of work task instances using a queue management strategy based on the task attributes of the work tasks; and allocating multiple work task instances to multiple resource nodes according to the processing order of the work task instances and the resource requirement information of the work task instances.

[0011] According to embodiments of this disclosure, allocating multiple task instances to target resource nodes includes: using a centralized allocation strategy, prioritizing the allocation of task instances to the same target resource node while avoiding the target resource node from reaching its load limit, until all task instances have been allocated. According to embodiments of this disclosure, allocating multiple task instances to target resource nodes according to their processing order and required resources includes: comparing the required resources of the current task instance with the available resources of each of the multiple resource nodes to obtain at least one target resource node corresponding to the current task instance; and, based on a load balancing strategy, sequentially and evenly distributing the multiple task instances to the multiple target resource nodes according to their processing order.

[0012] Another aspect of this disclosure provides a scheduling apparatus, comprising: a first determining module, configured to determine at least one target processing flow from a processing flow library based on data attributes of remote sensing data in response to a data processing request for remote sensing data; a second determining module, configured to determine a processing algorithm for at least one task based on task attributes of multiple tasks included in the at least one target processing flow; an associating module, configured to associate the at least one task with a corresponding processing algorithm to obtain at least one task instance; and an allocation module, configured to allocate multiple task instances to multiple resource nodes based on the task attributes of the task and resource requirement information of the multiple task instances.

[0013] Another aspect of this disclosure provides an electronic device comprising:

[0014] One or more processors;

[0015] Memory, used to store one or more programs.

[0016] Specifically, when one or more programs are executed by one or more processors, the one or more processors implement the above method.

[0017] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the methods described above.

[0018] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the methods described above.

[0019] According to embodiments of this disclosure, the corresponding target processing flow is determined by the data attributes of remote sensing data, which can support the processing of various types of remote sensing data. In situations requiring multi-task, massive data processing, the corresponding processing flow can be flexibly determined, improving the processing efficiency of remote sensing data. Furthermore, based on the determined target processing flow, task instances are obtained, and then allocated to corresponding resource nodes for execution according to the processing order and required resources of the task instances. Since task instances are executable sub-steps obtained based on the processing flow, the resource requirements of task instances are more granular. Allocation based on the resource requirement information of task instances reduces the total resources required to complete the target processing flow. Additionally, allocation based on finer-grained resource requirements offers greater flexibility. Therefore, when processing large-scale, multi-type remote sensing data, the flexibility and efficiency of resource scheduling can be improved, computational resources can be saved, and resource waste can be avoided. Attached Figure Description

[0020] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 The illustration schematically depicts an exemplary system architecture to which scheduling methods and apparatus can be applied according to embodiments of the present disclosure.

[0022] Figure 2 A flowchart illustrating a scheduling method according to an embodiment of the present disclosure is shown schematically.

[0023] Figure 3 The schematic illustration shows the determination process of the processing algorithm according to an embodiment of the present disclosure.

[0024] Figure 4A An example of a target processing flow topology diagram according to an embodiment of the present disclosure is illustrated schematically.

[0025] Figure 4B The illustration schematically shows an example of the processing order of a task instance corresponding to a target processing flow according to an embodiment of the present disclosure.

[0026] Figure 5 A flowchart illustrating the assignment of task instances according to an embodiment of this disclosure is shown schematically.

[0027] Figure 6 The illustration shows a schematic diagram of the task instance allocation process according to an embodiment of the present disclosure.

[0028] Figure 7 A schematic diagram illustrating a distributed resource parallel scheduling process according to an embodiment of the present disclosure is shown.

[0029] Figure 8 A schematic diagram of an integrated scheduling framework for remote sensing data processing tasks according to an embodiment of the present disclosure is shown.

[0030] Figure 9 The diagram illustrates a hierarchical architecture of an integrated scheduling framework for remote sensing data processing tasks according to an embodiment of the present disclosure.

[0031] Figure 10 A block diagram of a scheduling apparatus according to an embodiment of the present disclosure is shown schematically.

[0032] Figure 11 A block diagram of an electronic device suitable for implementing a scheduling method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0033] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0034] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0035] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0036] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0037] In related technologies, resources used for processing remote sensing data generally focus on general-purpose virtualized resources such as CPU and memory, designing resource allocation strategies around the goal of maximizing resource utilization. Furthermore, a DAG (Directed Acyclic Graph) model can be built using the partial order relationships in remote sensing data processing to solve for the optimal scheduling order. Alternatively, a workflow task scheduling algorithm based on deep reinforcement learning can be used to obtain the scheduling strategy with the minimum time. Another approach is to build a remote sensing data processing system in a cloud environment to improve cloud system utilization. In general, current remote sensing data processing systems mainly rely on traditional computing resources for scheduling remote sensing data processing tasks, constructing DAGs based on simple dependencies for scheduling the remote sensing data processing flow, focusing on solving optimal scheduling schemes for specific types of remote sensing data processing tasks, lacking scheduling design for multi-type, multi-mode remote sensing data tasks, and struggling to achieve integrated parallel processing of large-scale, massive remote sensing data.

[0038] For remote sensing data to be processed, the types of remote sensing data generally include multiple types such as visible light, infrared, hyperspectral, microwave imaging, and electronic signals. Different types of remote sensing data processing tasks have vastly different requirements for computing resources. Existing methods are difficult to fully consider the characteristic attributes of remote sensing image processing tasks and their differentiated requirements for computing resources. Moreover, the processing flow of multi-type remote sensing data is extremely complex, and traditional scheduling methods are difficult to effectively manage these complex task dependencies, thus affecting the efficiency and accuracy of the entire data processing flow.

[0039] Based on the above, in scenarios where remote sensing data processing tasks are performed, the processing requirements, processing models, and processing processes for massive amounts of remote sensing data vary. Moreover, due to the heterogeneity and diversity of resources of the equipment used to process data, most existing solutions allocate resources at the overall task level, resulting in a large granularity of resource scheduling and waste of resources.

[0040] In view of this, embodiments of the present disclosure provide a scheduling method, including:

[0041] In response to a data processing request for remote sensing data, at least one target processing flow is determined from the processing flow library based on the data attributes of the remote sensing data; based on the task attributes of the multiple tasks included in the at least one target processing flow, a processing algorithm for at least one task is determined; the at least one task is associated with the corresponding processing algorithm to obtain at least one task instance; based on the task attributes of the task and the resource requirement information of the multiple task instances, the multiple task instances are allocated to multiple resource nodes.

[0042] Figure 1 An exemplary system architecture 100 to which scheduling methods and apparatus can be applied according to embodiments of this disclosure is illustrated. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0043] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first slave node 101, a second slave node 102, a third slave node 103, a network 104, and a master node 105. The network 104 serves as a medium for providing communication links between the first slave node 101, the second slave node 102, the third slave node 103, and the master node 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0044] The master node 105 interacts with the first slave node 101, the second slave node 102, and the third slave node 103 through the network 104 to receive or send messages, etc. Various remote sensing data algorithms, such as optical processing algorithms, infrared processing algorithms, hyperspectral processing algorithms, etc., can be deployed on the first slave node 101, the second slave node 102, and the third slave node 103 (for example only).

[0045] The master node 105, the first slave node 101, the second slave node 102, and the third slave node 103 can all be various electronic devices with computing resources, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0046] The master node 105 can store various processing procedures or algorithms for processing remote sensing data. The first slave node 101, the second slave node 102, and the third slave node 103 can analyze and process the received data such as work task instances, and feed back the processed remote sensing data results to the master node 105.

[0047] It should be noted that the scheduling method provided in this embodiment can generally be executed by the master node 105. Correspondingly, the scheduling system provided in this embodiment can generally be set up in the master node 105. The scheduling method provided in this embodiment can also be executed by a server or server cluster that is different from the master node 105 and can communicate with the first slave node 101, the second slave node 102, the third slave node 103, and / or the master node 105. Correspondingly, the scheduling system provided in this embodiment can also be set up in a server or server cluster that is different from the master node 105 and can communicate with the first slave node 101, the second slave node 102, the third slave node 103, and / or the master node 105.

[0048] For example, remote sensing data may originally be stored in the master node 105, or it may be stored on an external storage device and imported into the master node 105. Then, the master node 105 can execute the scheduling method provided in the embodiments of this disclosure locally.

[0049] Figure 2 A flowchart illustrating a scheduling method according to an embodiment of the present disclosure is shown schematically.

[0050] like Figure 2 As shown, the method includes operations S210~S240.

[0051] In operation S210, in response to a data processing request for remote sensing data, at least one target processing flow is determined from the processing flow library based on the data attributes of the remote sensing data.

[0052] According to embodiments of this disclosure, remote sensing data may include visible light data, infrared data, hyperspectral data, microwave imaging data, and electronic signals, etc. The method of this disclosure can perform resource scheduling when processing multiple types of remote sensing data. The data attributes of remote sensing data may include data type, processing mode for the data, and parameters of the data processing request. Data attributes may also include data format, such as character data or image data. The processing mode may include streaming processing mode, batch processing mode, or persistent service mode. The parameters of the data processing request can be used to indicate the processing result for the current remote sensing data.

[0053] According to embodiments of this disclosure, the above-mentioned processing flow library may include multiple processing flows for processing remote sensing data. The processing flow may be a description of a data processing flow that obtains the processing result of remote sensing data based on the acquired remote sensing data. The processing flow may indicate the planning of processing steps for remote sensing data.

[0054] According to embodiments of this disclosure, a target processing flow that can be used to process the corresponding remote sensing data can be obtained by filtering from a flow library based on the data attributes of the remote sensing data described above. For cases where multiple sets of remote sensing data need to be processed simultaneously, this method can also filter out multiple target processing flows accordingly. When a target processing flow cannot be determined, a new processing flow can be planned based on the data attributes of the remote sensing data to process the remote sensing data.

[0055] In operation S220, based on the task attributes of the multiple tasks included in at least one target processing flow, a processing algorithm for at least one task is determined.

[0056] According to embodiments of this disclosure, the processing flow may include multiple work tasks with interdependent relationships. The aforementioned work task may be an indivisible minimum task unit, i.e., the work task does not contain subtasks; a work task may also contain multiple work subtasks with interdependent relationships.

[0057] According to embodiments of this disclosure, a corresponding processing algorithm can be specified for a task during the planning of the processing flow; alternatively, a new algorithm can be constructed based on the task's attributes, and this new algorithm can be used to complete the task. The aforementioned processing algorithms can include general and specialized algorithms for processing various types of remote sensing data, such as visible light, infrared, hyperspectral, microwave imaging, and electronic signals. The general algorithms provide common basic processing functions for various types of images and information, while the specialized algorithms perform targeted analysis and processing for the differentiated processing needs of different remote sensing data. No limitation is made to the processing algorithms described here.

[0058] In operation S230, at least one task is associated with a corresponding processing algorithm to obtain at least one task instance.

[0059] According to embodiments of this disclosure, after determining the processing algorithm corresponding to the task, the two can be associated. The associated task and the corresponding processing algorithm can then be used as a task instance. In this method, if the task instance has corresponding input remote sensing data, the task instance can be executed to obtain the output result of the corresponding remote sensing data.

[0060] In operation S240, based on the task attributes of the task and the resource requirement information of multiple task instances, multiple task instances are allocated to multiple resource nodes.

[0061] According to embodiments of this disclosure, the task attribute of a task can also indicate the priority of the corresponding task, and the processing order of the corresponding task instance can be determined based on the task priority.

[0062] According to embodiments of this disclosure, the attribute information of the processing algorithm included in a task instance may include resource requirement information for executing the processing algorithm. If the resource device executing the task instance cannot meet the resource requirements of the processing algorithm, the task instance cannot be executed successfully. This method can allocate multiple task instances to resource nodes that meet their resource requirement information based on the processing order of the aforementioned task instances, thereby completing the resource scheduling of remote sensing data.

[0063] According to embodiments of this disclosure, the execution status of the task instance can be continuously monitored, so the target processing flow corresponding to the current task instance can be associated, the relationship of the remaining tasks in the target processing flow can be analyzed, at least one new executable task instance can be obtained, and the task sorting and allocation can be performed cyclically until the target processing flow ends.

[0064] According to embodiments of this disclosure, the aforementioned resource nodes can be obtained by partitioning local hardware resource devices or cloud resource devices. Local hardware resource devices may include bare-metal servers, virtual machines, and other devices that can provide computing resources. The resource types included in the resource nodes may include CPU (Central Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), disk, MEM (Computer Memory), ports, or FPGA (Field-Programmable Gate Array), etc.

[0065] According to embodiments of this disclosure, the corresponding target processing flow is determined by the data attributes of remote sensing data, which can support the processing of various types of remote sensing data. In situations requiring multi-task, massive data processing, the corresponding processing flow can be flexibly determined, improving the processing efficiency of remote sensing data. Furthermore, based on the determined target processing flow, task instances are obtained, and then allocated to corresponding resource nodes for execution according to the processing order and required resources of the task instances. Since task instances are executable sub-steps obtained based on the processing flow, the resource requirements of task instances are more granular. Allocation based on the resource requirement information of task instances reduces the total resources required to complete the target processing flow. Additionally, allocation based on finer-grained resource requirements offers greater flexibility. Therefore, when processing large-scale, multi-type remote sensing data, the flexibility and efficiency of resource scheduling can be improved, computational resources can be saved, and resource waste can be avoided.

[0066] According to embodiments of this disclosure, determining a processing algorithm for at least one task based on the task attributes of multiple tasks included in at least one target processing flow includes: splitting at least one target processing flow to obtain multiple tasks that are dependent on each other; and determining a processing algorithm for at least one task based on the task attributes of the tasks.

[0067] According to embodiments of this disclosure, when multiple remote sensing data need to be processed, multiple target processing flows can be determined based on the different data attributes of the remote sensing data or the different data processing requests.

[0068] According to embodiments of this disclosure, after obtaining at least one target processing flow, the obtained target processing flow can be split into sub-flows. However, the dependencies between the tasks within the sub-flows must be preserved. When splitting the target processing flow, if a sub-flow includes sub-tasks, it should be further split until all sub-flows are indivisible individual tasks. The aforementioned dependencies can include strong dependencies, weak dependencies, or data dependencies. A strong dependency means that a subsequent task can only be initiated after the preceding task has been successfully completed. A weak dependency means that a subsequent task can be initiated regardless of whether the preceding task has been successfully completed. A data dependency means that the subsequent task is initiated immediately after the preceding task starts, with quasi-synchronous execution, indicating a data dependency relationship.

[0069] According to embodiments of this disclosure, when splitting a process, all tasks included in the target processing flow may be obtained at once, or only the first or first few tasks in the target processing flow may be obtained. The remaining tasks can only be determined after the task instances corresponding to the first or first few tasks in the target processing flow have been executed.

[0070] According to embodiments of this disclosure, after the task splitting is completed, the corresponding processing algorithm is determined based on the identifiable task attributes. If a task A has undetermined task attributes, the task attributes of task A can only be determined after the execution of other task instances that depend on task A. Then, the processing algorithm corresponding to task A is determined based on the task attributes of task A. Task attributes may include task input data, task output data, and task processing mode. The task processing mode may include streaming processing mode, batch processing mode, or persistent service mode.

[0071] Figure 3 The schematic illustration shows the determination process of the processing algorithm according to an embodiment of the present disclosure.

[0072] According to embodiments of this disclosure, the following is combined with Figure 3The process for determining the processing algorithm is described below. First, multiple target processing flows are broken down into tasks, each with various dependencies on the others. Then, it's determined whether the task attributes of each task can be determined. For tasks with identifiable attributes, their corresponding processing algorithms are determined based on those attributes. If the task attributes of a task cannot be determined, the current remote sensing data processing status is monitored, and the execution status of the target processing flow is updated in real time. The process waits until other dependent tasks have completed their execution before determining their corresponding task attributes and then using those attributes to determine the appropriate processing algorithm. If the task attributes indicate that the task includes subtasks, it is further broken down until the resulting tasks are indivisible. Task scheduling is then performed dynamically until all tasks have corresponding processing algorithms determined.

[0073] According to embodiments of this disclosure, the scheduling method further includes: determining processing algorithms corresponding to multiple tasks in a processing algorithm library based on the task attributes of the tasks.

[0074] According to embodiments of this disclosure, an algorithm library can also be established to deploy processing algorithms for performing tasks. Establishing the algorithm library may include the following stages:

[0075] (1) Algorithm development stage: Developers need to develop algorithms corresponding to various remote sensing data processing according to the processing algorithm integration specifications, and follow the corresponding interface requirements and task attributes to obtain a dynamic library of processing algorithms. The algorithm development process needs to correspond with the operation stage, and the algorithm development process should be constrained according to the integration specifications, specifically involving behaviors such as startup, file access, working directory, and log output.

[0076] (2) Algorithm packaging stage: Personnel or developers handling algorithm integration generate corresponding algorithm packages according to the packaging specifications in the algorithm integration specifications and certain file organization rules. The algorithm packaging specifications involve the compilation and packaging requirements of the algorithm, the directory rules of the algorithm file package, and the naming rules of different versions of the algorithm, which facilitates subsequent file package management, partial updates and multi-version management of the algorithm.

[0077] (3) Algorithm Deployment Stage: Algorithm integrators use registration tools to deploy algorithm packages with different functions into the system, forming an algorithm library. Each algorithm package deployed into the system is an independent processing algorithm in the process flow construction, and can correspond to an indivisible work task. The algorithm package entity file is then deployed to a designated location in the system for easy subsequent use.

[0078] The registration specifications for processing algorithms include metadata descriptions and computational resource description requirements:

[0079] Metadata information includes basic algorithm information, resource requirements, algorithm version information, order interface templates, etc. Algorithms can be registered and exported with one click according to metadata description specifications, which facilitates the maintenance and management of algorithm metadata information.

[0080] The computational resource description requirements can be: in addition to the traditional computational resource requirements such as the number of CPU cores and memory size, additional requirements such as scheduling requirements for GPU, NPU, FPGA, timeout, and concurrency are added. It supports heterogeneous resources and custom extensions, providing a more comprehensive description and support for the resource scheduling of algorithms.

[0081] For parallel processing scenarios, the algorithm package can also be designed as a data processing operator that can be computed in parallel, providing a scheduling basis for the construction of batch processing, streaming processing and big data analysis and processing processes for large-scale remote sensing data.

[0082] (4) Algorithm Execution Phase: During the execution phase, the algorithm first combines with the work tasks to form a work task instance. Then, based on the resource requirements at the time of deployment, it is allocated to a suitable resource node to run. After the processing algorithm in the work task instance starts, it interacts with the system according to the interface specifications in the processing algorithm integration specification. The processing algorithm interface specification involves multiple aspects such as order parameters, data interaction, and log management of the processing algorithm. There are eight types of interaction interfaces for the algorithm, namely, job order interface, work directory interface and exit code interface, data packet message interface, status reporting interface, exit command interface, running log interface, and product list interface.

[0083] According to embodiments of this disclosure, processing algorithms for different types of remote sensing data vary in processing modes and resource requirements. When packaging and deploying processing algorithms, it is necessary to select the appropriate processing mode and resource requirements. Common processing modes include batch processing, streaming processing, and persistent service modes. Specifically:

[0084] Batch processing mode: Batch processing mode is the most common processing mode for remote sensing data processing algorithms. It is suitable for efficient processing of large volumes of remote sensing data. Its characteristics include: all input data and configuration parameters are determined at startup; during algorithm operation, runtime logs and status information are output; and upon completion, result files and information are output, returning an exit code according to the interface specification. The registration of batch processing algorithms is constrained by registration and runtime parameters such as timeout. For example, batch processing algorithm processes that have not exited by the timeout period will be forcibly terminated.

[0085] Streaming processing mode: Streaming processing mode is a processing mode for streaming remote sensing data. It is suitable for real-time processing of small-volume, continuous remote sensing data. Its characteristics are that the processing algorithm acquires basic parameters and data information when it starts up, and the input data is continuously read during the algorithm's operation. After all streaming data is processed, the processing algorithm needs to listen for an exit command. The exit command indicates that there is no subsequent data stream, and the processing algorithm will exit the operation.

[0086] Persistent Service Mode: Persistent service mode requires the processing algorithm to run normally in the form of microservices, providing corresponding computing and processing services. The remote sensing data to be processed has the characteristics of being unlimited, intermittent and unpredictable. Once the remote sensing data processing flow is started, the processing algorithm corresponding to the internal task instance will run for a long time until the entire remote sensing data processing flow ends, at which point the processing algorithms corresponding to all internal task instances will stop running.

[0087] According to embodiments of this disclosure, by constructing an algorithm library, the types of remote sensing data that the scheduling algorithm can process can be expanded in real time. Specifically, the algorithm library can include multiple algorithms for processing various types of remote sensing data, and the processing algorithms in the algorithm library can be dynamically adjusted. Furthermore, the processing algorithm is determined based on the task attributes of the task, decoupling the task and the processing algorithm at the matching level, allowing for dynamic updates and flexible adjustments to the algorithm. Therefore, the overall computational efficiency of remote sensing data processing can be improved by increasing the computational efficiency of the processing algorithm.

[0088] According to embodiments of this disclosure, the data attributes of remote sensing data include data type and processing mode; determining at least one target processing flow from a processing flow library based on the data attributes of remote sensing data includes: filtering from the processing flow library based on multiple processing modes of remote sensing data to obtain multiple first processing flows; and determining at least one target processing flow from the multiple first processing flows based on the data type.

[0089] According to embodiments of this disclosure, the data types of the aforementioned remote sensing data may include various types such as visible light, infrared, hyperspectral, microwave imaging, and electronic signals, and the processing modes of remote sensing data may include streaming processing mode, large-scale batch processing mode, resident service mode, and post-event remote sensing big data analysis and processing flow.

[0090] According to embodiments of this disclosure, when determining the target processing flow, a first processing flow matching the processing mode of remote sensing data can be selected from the flow library. Specifically, for a first processing flow suitable for streaming processing mode, intermediate data is generally not written to disk but directly passed to the next task through a cache to improve the timeliness of data processing; for a first processing flow suitable for large-scale batch processing mode, there are generally strong dependencies between tasks, and the parallelism of data processing is improved through pre-allocation of data, and the timeliness of data processing is improved through large-scale parallel processing; for a first processing flow suitable for resident service mode, it runs in a microservice manner for a long time, providing service interfaces for corresponding functions on a regular basis until the overall remote sensing data processing plan is completed; for a first processing flow suitable for post-event remote sensing big data analysis, it uses data segmentation and result aggregation to improve the parallelism of data processing, and provides remote sensing big data analysis processing capabilities through large-scale parallel processing.

[0091] According to embodiments of this disclosure, after determining multiple first processing flows based on the processing mode, a second filtering is performed based on the data type of the remote sensing data to determine the corresponding target processing flow.

[0092] According to embodiments of this disclosure, the processing flow in the above-described process library can be constructed using DWMR (Directed Acyclic Graph Workflow Model for Remote Sensing Data Process).

[0093] The DWMR workflow model uses XML (eXtensible Markup Language) format to describe and define the processing flow of remote sensing data. The specific definition format is shown in Table 1.

[0094] Table 1

[0095]

[0096] In the DWMR workflow model, NodeType is used to describe the type of work task in the workflow. There are three types of work tasks in DWMR: Process, Task, and Fake. As shown in Table 2, these are described in detail.

[0097] Table 2

[0098]

[0099] In the DWMR workflow model, NodeType supports sub-processes, allowing workflows to be nested freely, which improves the scalability of remote sensing data processing workflows.

[0100] In the DWMR workflow model, DepType can represent the dependency relationship between tasks and is divided into Force, Unforce, and Data, as detailed in Table 3.

[0101] Table 3

[0102]

[0103] In the DWMR workflow model, both Force and Unforce belong to sequential flows, representing typical dependencies in batch processing. Data represents data flows, indicating data dependencies in streaming scheduling. The DWMR workflow model, through the DepType field, provides the foundation for defining the integrated streaming and batch processing scheduling in remote sensing data processing workflows.

[0104] Therefore, the processing flow of remote sensing data can be constructed based on the aforementioned DWMR workflow model. During construction, task attributes and dependencies between tasks can be specified; processing algorithms can also be directly assigned to tasks to build the processing flow, and multiple processing flows can then be stored in a flow library. When processing remote sensing data, the target processing flow that best matches the data can be selected based on the data attributes of the remote sensing data to be processed.

[0105] According to embodiments of this disclosure, target processing flows for processing corresponding remote sensing data are selected from a processing flow library based on the data type and processing mode of the remote sensing data. For the processing flow library, a processing flow more suitable for the current remote sensing data processing plan can be constructed using the DWMR workflow model, making the entire processing process more closely matched to the corresponding remote sensing data. Therefore, based on the flow selection from the processing flow library, processing flows can be dynamically matched, improving the efficiency of remote sensing data processing.

[0106] According to embodiments of this disclosure, multiple work task instances are allocated to multiple resource nodes based on the task attributes of the work tasks and the resource requirement information of multiple work task instances, including: determining the processing order of work task instances using a queue management strategy based on the task attributes of the work tasks; and allocating multiple work task instances to multiple resource nodes according to the processing order of the work task instances and the resource requirement information of the work task instances.

[0107] The following is for reference. Figure 4A and Figure 4B The processing order of task instances will be further explained in conjunction with specific embodiments.

[0108] Figure 4A An example of a target processing flow topology diagram according to an embodiment of the present disclosure is illustrated schematically.

[0109] Figure 4B The illustration schematically shows an example of the processing order of a task instance corresponding to a target processing flow according to an embodiment of the present disclosure.

[0110] like Figure 4A As shown, the target processing flow includes 8 tasks: Task 1, Task 2, Task 3, Task 4, Task 6, Task 7, Task 8, and Task 9. The target processing flow also includes a sub-process Task 5, which contains 4 tasks: Task 51, Task 52, Task 53, and Task 54. There are multiple dependencies between the different tasks.

[0111] By analyzing the dependencies between these tasks, subprocess task 5 is broken down into task 51, task 52, task 53 and task 54.

[0112] It is possible to obtain, such as Figure 4B The example shown is for Figure 4A The processing order of the multiple job task instances shown.

[0113] like Figure 4B As shown, task 1 corresponds to task instance 1, task 2 corresponds to task instance 2, task 3 corresponds to task instance 3, task 4 corresponds to task instance 4, task 51 corresponds to task instance 51, task 52 corresponds to task instance 52, task 53 corresponds to task instance 53, task 54 corresponds to task instance 54, task 6 corresponds to task instance 6, task 7 corresponds to task instance 7, task 8 corresponds to task instance 8, and task 9 corresponds to task instance 9. Since there is a data dependency between task instances 1, 2, 3, and 4, they can be processed sequentially in a streaming manner.

[0114] Since both task instance 51 and task instance 6 have strong dependencies on task instance 1, they can only be executed after task instance 1 has been executed successfully.

[0115] Job task instance 52 and job task instance 53 both have a strong dependency on job task instance 51, so they can only be executed after job task instance 51 has been executed successfully.

[0116] Task instance 54 has strong dependencies on both task instances 52 and 53, so it can only be executed after both task instances 52 and 53 have been executed successfully.

[0117] Task 7 has a strong dependency on both Task 54 and Task 6, so it can only be executed after both Task 54 and Task 6 have been executed successfully.

[0118] Task Instance 8 has a strong dependency on Task Instance 6, so it can only be executed after Task Instance 6 has been executed successfully.

[0119] Task instance 9 has a weak dependency on task instance 8, so it can be executed after task instance 8 has finished executing.

[0120] According to embodiments of this disclosure, the resource requirement information of a task instance can be determined through the resource requirement information of the corresponding processing algorithm within the task instance. After obtaining the processing order of the task instance based on the task attributes and queue management strategy of the corresponding task, allocation is performed according to the processing order. If the resource requirement of the current task instance is less than the resources possessed by a certain resource node, the task instance can be allocated to the corresponding resource node, thus completing resource scheduling.

[0121] According to embodiments of this disclosure, resource nodes can be obtained from a uniformly constructed resource pool. For processing resources in the resource pool, resources can be merged from devices possessing processing resources. Specifically, resource modeling must first be performed on the devices possessing processing resources. Resource modeling can involve describing the processing resources contained within the devices. Resource descriptions can include scalars, floating-point numbers, integers, ranges, sets, and text; alternatively, only scalars, ranges, and sets can be selected to describe the processing resources.

[0122] According to embodiments of this disclosure, the queue management strategy described above may include a priority strategy, such as prioritizing the scheduling of the highest-priority task instance to an idle resource node. The priority strategy can be used to allocate task instances requiring priority processing to idle resource nodes, providing scheduling possibilities for prioritizing the production of key areas or target products.

[0123] According to embodiments of this disclosure, the queue management strategy may further include a time-based strategy, such as sorting work task instances according to the order in which they are submitted, with each instance being submitted to the most available computing resource at the front of the queue. This time-based strategy favors utilizing resource nodes at the front of the queue, ensuring that backend queuing computing resources operate at low energy consumption when the task load is low, thus saving system resources.

[0124] The following is for reference. Figure 5 The allocation of task instances will be further explained in conjunction with specific embodiments.

[0125] Figure 5 A flowchart illustrating the assignment of task instances according to an embodiment of this disclosure is shown schematically.

[0126] According to the embodiments of this disclosure, after multiple target processing flows are determined from the process library, multiple sets of work task instances can be obtained by splitting operations and associating operations based on the multiple target processing flows. Moreover, there may be no association between different sets of work task instances. Then, multiple job queues can be constructed for the multiple sets of work task instances. The job queues include multiple work task instances with processing order.

[0127] According to embodiments of this disclosure, such as Figure 5 As shown, during the execution of the scheduling method, the system continuously listens for work task instance processing instructions. When a new work task instance needs to be assigned, the processing order of the corresponding work task instances is first determined, and then they are sent to the job queue for queuing, awaiting execution. Next, the job queue management system first pops the first work task instance to be assigned from the matched computation queue according to the queue management strategy. If the queue management strategy is based on time sequence, the work task instance that arrived first is popped in chronological order; if the queue management strategy is based on priority, the work task instance with the highest priority is popped. After the job queue management system pops the first work task instance to be processed, it selects an allocation strategy from the allocation strategy library and polls the resource pool for resource nodes that can meet the resource requirements of the work task instance. Then, the work task instance is assigned to the corresponding resource node, and the resource node then executes the corresponding work task instance.

[0128] According to embodiments of this disclosure, the processing order of task instances can be determined based on the task attributes of the task and a preset queue management strategy. Then, multiple task instances are allocated to multiple resource nodes according to this processing order and the resource requirements of the task instances. Task instances with higher processing orders are preferentially allocated to resource nodes for execution. Furthermore, allocation based on resource requirements ensures that resource nodes have the capacity to execute the corresponding task instances. This allocation method, while confirming that task instances can be executed, also guarantees that task instances are executed in sequence, thereby improving the execution efficiency of task instances and successfully completing the resource scheduling of remote sensing data.

[0129] According to embodiments of this disclosure, allocating work task instances to target resource nodes includes: using a centralized allocation strategy, prioritizing the allocation of work task instances to the same target resource node while avoiding the target resource node from reaching its load limit, until all work task instances have been allocated.

[0130] According to embodiments of this disclosure, if the allocation strategy is centralized allocation, for the current task instance, all resource nodes are polled in a fixed node order to find the first resource node that can meet the computing resource requirements for allocation, until all task instances are allocated. For example, multiple resource nodes have different weights for the task instances to be processed, and the resource nodes are queued in a certain order. Task instances are preferentially allocated to resource nodes with higher priority, until the task instances running on that resource node are saturated, and then the jobs are allocated to subsequent resource nodes in sequence. This mode can keep some computing resources idle in a normal state, which can play a role in resource reservation and provide the possibility of timely scheduling for individual task instances with large computing resource requirements, preventing the situation where resource equalization causes some task instances to be in a waiting state for a long time without being able to obtain a single sufficient computing resource.

[0131] According to embodiments of this disclosure, the allocation of work task instances to target resource nodes according to the processing order of the work task instances and the required resources of the work task instances includes: comparing the required resources of the current work task instance with the available resources of each of the multiple resource nodes to obtain at least one target resource node corresponding to the current work task instance; and based on a load balancing strategy, the multiple work task instances are evenly distributed to multiple target resource nodes according to the processing order of the work task instances.

[0132] According to embodiments of this disclosure, if the allocation strategy is a load balancing strategy, then based on the load status of each resource node in the current resource pool, idle nodes are prioritized, and the first processing node that can meet the computing resource requirements is matched. For example, multiple resource nodes have the same weight for the pending task instances, and the task instances are allocated to multiple resource nodes for scheduling and execution according to resource availability. This mode can be used for processing large-scale remote sensing data in daily situations, where task instances are evenly distributed across multiple resource nodes, fully leveraging the overall processing capacity of the system.

[0133] The following is for reference. Figure 6 The allocation of task instances will be further explained in conjunction with specific embodiments.

[0134] Figure 6 The illustration shows a schematic diagram of the task instance allocation process according to an embodiment of the present disclosure.

[0135] According to embodiments of this disclosure, Figure 6 The task (N) shown can represent a group of work task instances composed of multiple work task instances. In task scheduling within a single work task instance group, the processing order of the work task instances in the group is obtained based on the task attributes of the corresponding work task instances, under a preset queue management strategy. Then, a computation queue is formed based on this processing order. Furthermore, two types of computation queues can be used to queue multiple work task instances to be executed. If the computation queue is a priority queue, jobs with higher priority among the current work task instances will be allocated resources first. If the computation queue is a time-based strategy queue, the work task instances will be managed according to their time order. Then, when allocating computational resources, the allocation strategy is first determined, and then the work task instances are allocated to the corresponding resource nodes. The above allocation strategies include centralized allocation strategies and load balancing strategies. Finally, the executor is used on the corresponding resource node to execute the work task instance.

[0136] The following is for reference. Figure 7 The following is a further explanation of the parallel scheduling of distributed resources, with reference to specific embodiments.

[0137] Figure 7 A schematic diagram illustrating a distributed resource parallel scheduling process according to an embodiment of the present disclosure is shown.

[0138] According to embodiments of this disclosure, distributed resource parallel scheduling can also be used for the aforementioned resource pool. Distributed resource parallel scheduling is based on a unified computing resource pool, providing unified management, allocation, and integrated scheduling of computing resources. At the architectural level, distributed resource parallel scheduling is divided into a scheduling service (resource scheduler) and a proxy service (job executor). The scheduling service is generally deployed on a scheduling server, while the proxy service is deployed on each resource node. Multiple resource nodes with common attributes constitute a logical partition of a certain type of computing resource. This logical partition supports dynamic resource expansion and automatic removal of faulty nodes, etc. Figure 7 Resource node 7 was removed due to a fault, and resource node 5 was used as an expansion resource node.

[0139] According to embodiments of this disclosure, the resource pool can be divided into different logical partitions based on business needs, corresponding to different computing queues. Each resource node notifies the scheduler of its own liveness status via heartbeat, providing a basis for scheduling work task instances. The scheduling server receives remote sensing data processing tasks, decomposes them into work task instances, and performs parallel scheduling. Based on the resource requirements of the work task instances, it maps different resource partitions to the work task instances, selects corresponding resource nodes from the logical partitions of computing resources, and assigns the work task instances to the corresponding resource nodes. The resource nodes are responsible for starting the work task instances corresponding to this task.

[0140] According to embodiments of this disclosure, such as Figure 7 As shown, the scheduling server also features hot standby. Through real-time data synchronization and rapid failover, it ensures that the backup server can immediately take over the work when the primary server fails, minimizing business interruption time. Resource node 1 and resource node 2 can be assigned to logical partition 1. Meanwhile, resource node 2 and resource node 4 can also be assigned to logical partition 3, and resource node 3 can be assigned to logical partition 2.

[0141] According to embodiments of this disclosure, distributed resource scheduling utilizes Redis cache (a high-performance key-value store based on memory) to cache job status information. The scheduling service operates in a stateless mode, thus enabling stateless horizontal scaling during software service deployment and operation, preventing single points of failure, and providing reliability and stability for the scheduling service.

[0142] According to embodiments of this disclosure, using methods such as centralized allocation and load balancing to allocate task instances can distribute task instances to target resource nodes through reasonable strategies, avoiding overload of single resource nodes and improving the overall resource utilization efficiency of the resource scheduling method. It also ensures the flexibility and reliability of task instance allocation, thereby enhancing system stability and availability. Furthermore, distributed resource parallel scheduling can flexibly handle various resources in the resource pool, enabling resource nodes in the pool to better process task instances.

[0143] The following is for reference. Figure 8 The integrated scheduling framework for remote sensing data processing tasks will be further explained with reference to specific embodiments.

[0144] Figure 8 A schematic diagram of an integrated scheduling framework for remote sensing data processing tasks according to an embodiment of the present disclosure is shown.

[0145] According to embodiments of this disclosure, such as Figure 8 As shown, for the scheduling method described above, an integrated scheduling framework for remote sensing data processing tasks can also be established, including processing algorithm integration, business process scheduling, business process modeling, task instance allocation, processing resource pooling, data caching, and corresponding specifications and models, specifically:

[0146] (1) Business process modeling. Based on the remote sensing data processing process definition model, different types of remote sensing data processing processes are defined and modeled. A unified process modeling language is used to describe the complex remote sensing data processing workflows of various types, and the dependencies and scheduling logic of the business processes are determined, which is the basis for business process scheduling.

[0147] (2) Processing algorithm integration. In accordance with the processing algorithm integration specification, we will carry out the integrated registration and integration of various types of remote sensing data processing algorithms, such as visible light, infrared, hyperspectral, microwave imaging, and electronic signals. We will agree on the order parameters and operating resource requirements of the processing algorithms, and carry out unified integration and management of various processing algorithms.

[0148] (3) Business process scheduling. Based on dynamic workflow technology, business logic is controlled and driven in real time. Business process orders are processed through remote sensing data, the process modeling configuration of production processing flow is interpreted in real time, the data flow to be processed is tracked to dynamically trigger job scheduling, and the creation, startup, operation and task initiation of work task instances are realized according to business requirements.

[0149] (4) Processing resource pooling. Processing resource pooling is responsible for accessing and adapting various heterogeneous resources such as bare metal servers, virtual machines, containers, cloud systems, and dedicated equipment based on a unified processing resource description, forming a computing resource pool, providing a unified resource service interface for processing resource scheduling and computing job scheduling, and monitoring the health and usage status of various processing resources.

[0150] (5) Data caching. Data caching makes full use of large-capacity caching devices, in-memory databases, message buses and other hardware and software infrastructure to achieve high-speed reading and writing of remote sensing data, distributed caching of scheduling information, and rapid interaction of task information, providing data support for efficient parallel scheduling of large-scale and massive remote sensing data processing tasks.

[0151] (6) Work task instance allocation. Based on the resource pooling process, the allocation and scheduling of work task instances are implemented. To meet diverse scheduling needs, resources are dynamically requested and allocated from the resource pool according to the scheduling strategy configuration to form resource nodes. Work task instances are then scheduled to resource nodes that meet the resource requirements on demand.

[0152] (7) Specifications and Models. The specifications and models design provides standards and specifications for the integration and construction of various processing algorithms, business processes, and computing resources. The specifications and models design includes processing algorithm integration specifications, remote sensing data processing workflow definition models, and processing resource description models. The algorithm integration specifications are used to constrain the registration and integration of processing algorithms, including encapsulation specifications, registration specifications, operation specifications, and interface specifications, covering all stages of processing algorithm development and integration; the remote sensing data processing workflow definition model is oriented towards complex and diverse multi-type remote sensing data processing logic, and constructs a unified workflow description mechanism; the processing resource description model uses a unified resource description language to uniformly describe various heterogeneous resources, laying the foundation for resource allocation.

[0153] Figure 9The diagram illustrates a hierarchical architecture of an integrated scheduling framework for remote sensing data processing tasks according to an embodiment of the present disclosure.

[0154] According to embodiments of this disclosure, such as Figure 9 As shown, the layered architecture described above includes a resource layer, a logic layer, an algorithm layer, and a scheduling layer. The scheduling layer is further divided into process flow scheduling and instance scheduling. The advantage of this layered architecture design is that it achieves loose coupling between layers, allowing each layer to be independently upgraded and expanded. This enables dynamic adjustment and rolling development of remote sensing data processing services. Specifically:

[0155] (1) Resource layer. The resource layer contains various basic hardware and software resources required for remote sensing data processing, including computing equipment, high-density acceleration processing equipment, high-speed caching equipment, cloud computing, special equipment, network environment, database, operating system, message middleware, etc., providing a hardware and software support environment for efficient computing of remote sensing data processing tasks.

[0156] (2) Algorithm layer. The algorithm layer includes general and special algorithms for processing various types of remote sensing data, such as visible light, infrared, hyperspectral, microwave imaging, and electronic signals. General algorithms provide common basic functions for processing various types of images and information, while special algorithms perform targeted analysis and processing for the different processing needs of different remote sensing data.

[0157] (3) Logic layer. The logic layer is used to define complex processing business processes for different types of remote sensing data. It can flexibly arrange the processing flow according to different data processing needs, and supports various execution logics such as sequential, branching, merging, parallel, and nesting, as well as multiple modes of association logic such as data dependency, strong dependency, and weak dependency.

[0158] (4) Scheduling layer. The scheduling layer is responsible for the integrated scheduling of remote sensing data processing tasks. It is divided into business process scheduling and work task instance scheduling. Business process scheduling parses the processing requirements, generates work task instances, executes them according to the target processing flow, and tracks and controls the execution status of the flow. Instance scheduling is responsible for scheduling work task instances to the corresponding resource nodes for execution according to the scheduling strategy and the current load of processing resources.

[0159] According to embodiments of this disclosure, resource pooling can be performed using either general-purpose processing equipment or dedicated processing equipment.

[0160] According to embodiments of this disclosure, for general-purpose processing devices, general-purpose resource pooling has the ability to apply for basic environment resources from various cloud platforms and schedule and use corresponding computing, storage, network, and other processing resources. It can combine the processing resources of general-purpose processing devices into a resource pool that supports dynamic allocation and scheduling of processing resources, thus masking the hardware differences between different types of processing resources. Specifically, this includes:

[0161] (1) Processing resource registration management: Provides the ability to register processing resources to the scheduling system, supports the horizontal expansion of processing resources, and can automatically monitor and report the original information of processing resources after registration, such as resource type, network information, storage information, CPU information, and memory information; supports management operations such as adding and removing processing resources.

[0162] (2) Cloud platform basic environment resource application: Based on the unified access interface for processing resources provided by the cloud platform, submit resource access applications to realize the application, query and access of various processing resources and storage resources, and have the ability to schedule and use basic environment resources.

[0163] (3) Resource pooling construction: All processing resources are combined into a processing resource pool that supports dynamic allocation and scheduling of processing resources. Reasonable resource allocation and balance control are performed on these processing resources to achieve transparency of specific processing nodes to business processing units. The resource pool defines a unified calling interface for various software programs to access processing resources, shielding the hardware differences between different types of computing devices and providing unified resource services for job scheduling.

[0164] According to embodiments of this disclosure, dedicated processing devices are characterized by integrated hardware and software, high-density computing, and independent operation. They provide specialized data processing functions to the outside world through service interfaces, and are a type of service-oriented dedicated processing resource. The method of this disclosure can integrate and schedule dedicated processing devices by encapsulating them to a certain extent and through service calls. The integration of dedicated processing devices mainly includes the following steps:

[0165] (1) Standardized management of equipment service interfaces: A unified management specification for dedicated processing equipment service interfaces should be established, including protocol and path specifications, parameter design specifications, authentication mechanisms, data format specifications, and monitoring management specifications. The service API (Application Programming Interface) of dedicated processing equipment should preferentially adopt the HTTP / HTTPS protocol and use the REST (Representational State Transfer) style to design interface paths. Path parameters are used to identify unique resources, and the validation rules for all parameters need to be clearly defined. All interface responses can use JSON (JavaScript Object Notation) format, and error codes with clear meanings should be defined. Health check interfaces and status monitoring interfaces can be provided to provide timely feedback on equipment load and monitoring status, facilitating adjustments to scheduling strategies.

[0166] (2) Standardized encapsulation of dedicated services: Standardize the service interface of dedicated processing equipment and implement standardized HTTP request calls based on the curl tool (command line data transmission tool). Clearly define the call address, authentication key, and request timeout; extract key parameters from the task context (such as task ID and execution time), and dynamically concatenate and assemble the parameters according to the format required by the device's service interface API; judge the status code of the returned result, and synchronize the status code, call result, and response time to the external calling system.

[0167] (3) Registration and configuration of dedicated services: Considering the limited number of dedicated processing devices, a centralized node configuration file can be maintained to record the IP, port, weight, and status (enabled / disabled) of the dedicated processing device service nodes. For multiple dedicated processing devices, referencing the pooling strategy of general computing resources, the dedicated processing devices can be registered and pooled based on the dedicated device pooling proxy service module to facilitate unified invocation in the future.

[0168] (4) Load balancing call, the scheduling strategy is specified through the configuration file: First, weighted round-robin, which allocates call requests according to the node weight to adapt to the differences in node performance; second, random algorithm, which is suitable for the scenario of balanced node performance and avoids the concentration of requests; third, least connections algorithm, which allocates requests based on the current number of connections of the node. The sampling interval needs to be configured to ensure the accuracy of load allocation.

[0169] (5) Fault tolerance and status monitoring: For call failure scenarios (such as timeout), differentiated retry strategies can be set, with different retry counts for different operations. During the retry process, the system automatically switches to the next available node, and the retry interval is exponentially backed off, reducing the service call pressure on the dedicated processing equipment. The monitoring interface is used to obtain the load and service status of the dedicated processing equipment nodes in a timely manner, and the call strategy is adjusted accordingly.

[0170] Figure 10 A block diagram of a scheduling apparatus according to an embodiment of the present disclosure is shown schematically.

[0171] like Figure 10 As shown, the scheduling device 1000 includes a first determining module 1010, a second determining module 1020, an association module 1030, and an allocation module 1040.

[0172] The first determining module 1010 is used to determine at least one target processing flow from a processing flow library based on the data attributes of the remote sensing data in response to a data processing request for remote sensing data.

[0173] The second determining module 1020 is used to determine the processing algorithm for at least one task based on the task attributes of the multiple tasks included in the at least one target processing flow.

[0174] The association module 1030 is used to associate at least one task with a corresponding processing algorithm to obtain at least one task instance.

[0175] The allocation module 1040 is used to allocate multiple work task instances to multiple resource nodes based on the task attributes of the work tasks and the resource requirement information of multiple work task instances.

[0176] According to embodiments of this disclosure, the second determining module 1020 includes a first determining submodule and a second determining submodule.

[0177] The first determination submodule is used to break down at least one target processing flow into multiple work tasks that are dependent on each other.

[0178] The second determination submodule is used to determine the processing algorithm for at least one task based on the task attributes of the task.

[0179] According to embodiments of this disclosure, the scheduling device 1000 further includes a third determining module.

[0180] The third determination module is used to determine the processing algorithms corresponding to multiple tasks in the processing algorithm library based on the task attributes of the tasks.

[0181] According to embodiments of this disclosure, the first determining module 1010 may include a third determining submodule and a fourth determining submodule.

[0182] The third determination submodule is used to filter from the processing flow library based on the processing modes of multiple remote sensing data to obtain multiple first processing flows.

[0183] The fourth determination submodule is used to determine at least one target processing flow from multiple first processing flows based on the data type.

[0184] According to embodiments of this disclosure, the allocation module 1040 may include a first allocation submodule and a second allocation submodule.

[0185] The first allocation submodule is used to determine the processing order of task instances based on the task attributes of the task and using a queue management strategy.

[0186] The second allocation submodule is used to allocate multiple work task instances to multiple resource nodes according to the processing order of the work task instances and the resource requirement information of the work task instances.

[0187] According to embodiments of this disclosure, the second allocation submodule may include a first allocation unit.

[0188] The first allocation unit is used to employ a centralized allocation strategy to prioritize the allocation of worker task instances to the same target resource node, while avoiding the target resource node from reaching its load limit, until all worker task instances have been allocated.

[0189] According to embodiments of this disclosure, the second allocation submodule may further include a second allocation unit and a third allocation unit.

[0190] The second allocation unit is used to compare the required resources of the current task instance with the available resources of each of the multiple resource nodes to obtain at least one target resource node corresponding to the current task instance.

[0191] The third allocation unit is used to distribute multiple task instances equally among multiple target resource nodes according to the processing order of the task instances based on the load balancing strategy.

[0192] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0193] For example, any plurality of the first determining module 1010, the second determining module 1020, the association module 1030, and the allocation module 1040 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the first determining module 1010, the second determining module 1020, the association module 1030, and the allocation module 1040 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first determining module 1010, the second determining module 1020, the association module 1030, and the allocation module 1040 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0194] It should be noted that the data processing system part in the embodiments of this disclosure corresponds to the data processing method part in the embodiments of this disclosure. The specific description of the data processing system part is referred to in the data processing method part, and will not be repeated here.

[0195] Figure 11 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Figure 11 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0196] like Figure 11 As shown, an electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage portion 1108 into a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0197] RAM 1103 stores various programs and data required for the operation of electronic device 1100. Processor 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Processor 1101 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1102 and / or RAM 1103. It should be noted that the programs may also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0198] According to embodiments of this disclosure, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to a bus 1104. The electronic device 1100 may also include one or more of the following components connected to the input / output (I / O) interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the input / output (I / O) interface 1105 as needed. A removable medium 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1110 as needed so that computer programs read from it can be installed into the storage section 1108 as needed.

[0199] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by processor 1101, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0200] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0201] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0202] For example, according to embodiments of this disclosure, a computer-readable storage medium may include one or more memories other than the ROM 1102 and / or RAM 1103 described above and / or ROM 1102 and RAM 1103.

[0203] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the scheduling methods provided in the embodiments of this disclosure.

[0204] When the computer program is executed by the processor 1101, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0205] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1109, and / or installed from the removable medium 1111. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0206] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be constructed using any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0207] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0208] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A scheduling method, characterized in that, include: In response to a data processing request for remote sensing data, at least one target processing flow is determined from a processing flow library based on the data attributes of the remote sensing data, wherein the data attributes of the remote sensing data include data type and processing mode; The step of determining at least one target processing flow from the processing flow library based on the data attributes of the remote sensing data includes: Multiple first processing flows are obtained by filtering from the processing flow library based on the processing modes of multiple remote sensing data; At least one target processing flow is determined from a plurality of the first processing flows based on the data type; Based on the task attributes of the multiple tasks included in at least one of the target processing flows, determine the processing algorithm for at least one of the tasks. Associating at least one of the work tasks with the corresponding processing algorithm yields at least one work task instance; Based on the task attributes of the work tasks and the resource requirement information of the multiple work task instances, the multiple work task instances are allocated to multiple resource nodes.

2. The scheduling method according to claim 1, characterized in that, The step of determining a processing algorithm for at least one of the multiple tasks included in at least one target processing flow, based on the task attributes of those tasks, includes: At least one of the target processing flows is broken down to obtain multiple work tasks that are dependent on each other; Based on the task attributes of the task, determine at least one processing algorithm for the task.

3. The scheduling method according to claim 2, characterized in that, The method further includes: Based on the task attributes of the task, multiple processing algorithms corresponding to the task are determined from the processing algorithm library.

4. The method according to claim 1, characterized in that, The step of allocating multiple work task instances to multiple resource nodes based on the task attributes of the work task and the resource requirement information of multiple work task instances includes: Based on the task attributes of the task, a queue management strategy is used to determine the processing order of the task instances. According to the processing order of the task instances and based on the resource requirement information of the task instances, multiple task instances are allocated to multiple resource nodes.

5. The method according to claim 4, characterized in that, The step of allocating multiple work task instances to target resource nodes includes: Using a centralized allocation strategy, while avoiding the target resource node from reaching its load limit, the work task instances are preferentially allocated to the same target resource node until all work task instances have been allocated.

6. The method according to claim 4, characterized in that, The step of allocating the multiple work task instances to target resource nodes according to the processing order of the work task instances and the resources required by the work task instances includes: The required resources of the current task instance are compared with the available resources of each of the multiple resource nodes to obtain at least one target resource node corresponding to the current task instance. Based on the load balancing strategy, according to the processing order of the work task instances, multiple work task instances are evenly distributed to multiple target resource nodes.

7. A scheduling device, comprising: A first determining module is configured to, in response to a data processing request for remote sensing data, determine at least one target processing flow from a processing flow library based on the data attributes of the remote sensing data, wherein the data attributes of the remote sensing data include data type and processing mode; the step of determining at least one target processing flow from the processing flow library based on the data attributes of the remote sensing data includes: filtering from the processing flow library based on multiple processing modes of the remote sensing data to obtain multiple first processing flows; and determining at least one target processing flow from the multiple first processing flows based on the data type; The second determining module is used to determine the processing algorithm of at least one of the work tasks based on the task attributes of the multiple work tasks included in at least one of the target processing flows. An association module is used to associate at least one of the work tasks with the corresponding processing algorithm to obtain at least one work task instance; The allocation module is used to allocate multiple work task instances to multiple resource nodes based on the task attributes of the work task and the resource requirement information of multiple work task instances.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 6.

Images