Remote sensing big data automatic processing method based on cloud computing

Abstract

The invention discloses a remote sensing big data automatic processing method based on cloud computing, and relates to the technical field of remote sensing data processing, and the method comprises the steps: obtaining the meta-task features of a remote sensing big data to-be-processed task; constructing a collaborative relation graph of the cloud computing cluster; obtaining a task adaptation degree between each meta-task in the meta-task features of the remote sensing big data task to be processed and all task processing nodes in a cloud computing cluster; screening out a plurality of primary screening task processing node topological structures meeting the processing requirements of the remote sensing big data to-be-processed tasks; determining an optimal task processing node topological structure combination of all remote sensing big data to-be-processed tasks; and allocating a meta-task corresponding to each optimal task processing node topological structure combination to a corresponding task processing node for automatic processing. The method has the advantages that intelligent topology screening is realized by dynamically matching meta-task features and calculating node attributes in combination with the collaborative relation graph, and the cloud resource utilization rate and the task processing efficiency are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing data processing, and specifically relates to an automated processing method for remote sensing big data based on cloud computing. Background Art

[0002] The rapid development of remote sensing technology has led to an explosive growth in remote sensing data. The traditional single-machine processing mode is difficult to meet the real-time processing requirements of massive data. With the popularization of cloud computing technology, distributed computing provides computing power support for remote sensing big data processing. However, existing cloud platforms often adopt a static resource allocation strategy in task scheduling, and do not fully consider the logical relevance between remote sensing data processing tasks and the collaborative characteristics between computing nodes. Especially in the multi-task concurrent scenario, there are problems such as coarse task scheduling granularity, low node collaboration efficiency, and unbalanced resource utilization, resulting in insufficient data processing timeliness and waste of computing resources.

[0003] When processing remote sensing data tasks, the existing technology usually adopts a simple task queue scheduling mechanism, lacking the intelligent parsing of task topology structure and the dynamic adaptation ability of computing resources. This extensive scheduling method is prone to problems such as uneven node load and excessive cross-node communication overhead, and it is difficult to meet the collaborative requirements of complex processing processes such as remote sensing data preprocessing, feature extraction, and intelligent interpretation. Especially when processing a task chain with data dependency relationships, the traditional method cannot effectively quantify the collaborative efficiency between nodes, restricting the overall scheduling optimization effect of cloud cluster resources. Summary of the Invention

[0004] To solve the above technical problems, an automated processing method for remote sensing big data based on cloud computing is provided. This technical solution solves the problem that the existing technology usually adopts a simple task queue scheduling mechanism when processing remote sensing data tasks, lacking the intelligent parsing of task topology structure and the dynamic adaptation ability of computing resources.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An automated processing method for remote sensing big data based on cloud computing, comprising: Obtain all the remote sensing big data tasks to be processed in the current cloud, disassemble the remote sensing big data tasks to be processed into several meta-tasks, and obtain the meta-task characteristics of the remote sensing big data tasks to be processed; Obtain the attributes of all task processing nodes in the cloud computing cluster, and construct a collaborative relationship graph of the cloud computing cluster based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster; Match the task attributes of each meta-task in the meta-task features of the remote sensing big data to-be-processed tasks with the attributes of the task processing nodes, and obtain the task adaptation degrees of each meta-task in the meta-task features of the remote sensing big data to-be-processed tasks to all task processing nodes in the cloud computing cluster; Based on the task adaptation degrees of each meta-task in the meta-task features of the remote sensing big data to-be-processed tasks to all task processing nodes in the cloud computing cluster, and the collaborative relationship graph of the cloud computing cluster, screen out several initial screening task processing node topological structures that meet the processing requirements of the remote sensing big data to-be-processed tasks; Based on the overall analysis of all remote sensing big data to-be-processed tasks in the cloud, determine the optimal task processing node topological structure combination of all remote sensing big data to-be-processed tasks; Based on the optimal task processing node topological structure combination, allocate the meta-tasks corresponding to each optimal task processing node topological structure combination to the corresponding task processing nodes for automated processing.

[0006] Preferably, the construction of the collaborative relationship graph of the cloud computing cluster based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster specifically includes: Based on the communication distances between the task processing nodes in the cloud computing cluster, calculate the initial task collaboration indicators between the task processing nodes; Based on the initial task collaboration indicators between the task processing nodes, combined with the task collaboration processing records between the task processing nodes in the historical computing tasks, calculate the task collaboration indicators between the task processing nodes in the cloud computing cluster; Based on the task collaboration indicators between all task processing nodes in the cloud computing cluster, form the collaborative relationship graph of the cloud computing cluster.

[0007] Preferably, the matching of the task attributes of each meta-task in the meta-task features of the remote sensing big data to-be-processed tasks with the attributes of the task processing nodes, and obtaining the task adaptation degrees of each meta-task in the meta-task features of the remote sensing big data to-be-processed tasks to all task processing nodes in the cloud computing cluster specifically includes: Based on the experience of historical remote sensing big data processing tasks, determine the node hardware requirements during the processing of each meta-task; Based on the fitting analysis of the actual hardware attributes of the task processing nodes and the node hardware requirements during the processing of the meta-tasks, determine the satisfaction degree of the actual hardware attributes of the task processing nodes to the node hardware requirements during the processing of the meta-tasks, as the task adaptation degree.

[0008] Preferably, the specific process of screening out several initial screening task processing node topologies that meet the processing requirements of remote sensing big data tasks based on the meta-task features of the remote sensing big data tasks to be processed, the task adaptation degree between each meta-task and all task processing nodes in the cloud computing cluster, and the collaborative relationship graph of the cloud computing cluster is as follows: Record the task processing nodes with a task adaptation degree exceeding the threshold as the target nodes of the meta-task; Select one task processing node from the target nodes of each meta-task to form a single-point satisfied task processing node topology for the remote sensing big data task to be processed; Based on the meta-task processing chain, extract the collaborative node chain from the single-point satisfied task processing node topology. The task processing nodes in the collaborative node chain are the task processing nodes for processing the meta-tasks in the meta-task processing chain; Based on the collaborative relationship graph of the cloud computing cluster, extract the task collaboration metrics between the nodes in the collaborative node chain; Based on the comprehensive analysis of the task adaptation degree of each task processing node in the collaborative node chain and the task collaboration metrics between the nodes, obtain the collaborative processing metrics of the collaborative node chain for the meta-task processing chain; Summarize the collaborative processing metrics of all collaborative node chains, and comprehensively analyze the task collaboration ability of the single-point satisfied task processing node topology; Extract the single-point satisfied task processing node topology with a task collaboration ability that meets the collaboration requirements as the initial screening task processing node topology.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows: By dynamically matching the meta-task features with the computing node attributes and combining the collaborative relationship graph, the present invention realizes intelligent topology screening, effectively improving the utilization rate of cloud resources and the task processing efficiency; optimizing the task chain scheduling path based on the collaborative metrics between nodes, significantly reducing the cross-node communication delay; through the multi-task load balancing strategy and the global optimal topology combination, while ensuring the timeliness of single-task processing, realizing the optimal allocation of the overall system resources, which can not only reduce the comprehensive time consumption of complex remote sensing data processing tasks, but also improve the utilization efficiency of cluster resources, and at the same time adapt to the dynamic load changes in the multi-task concurrent scenario, taking into account both the processing speed and the system stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of the remote sensing big data automatic processing method based on cloud computing proposed in Embodiment 1; Figure 2 It is a flowchart of the method for constructing the collaborative relationship graph of the cloud computing cluster proposed in Embodiment 2; Figure 3 It is a flowchart of the method for obtaining the task adaptation degree between the meta-task and the task processing node proposed in Embodiment 3; Figure 4 Flow chart of the method for screening the topological structure of the preliminary screening task processing nodes proposed in the third embodiment; Figure 5 Flow chart of the method for determining the optimal combination of task processing node topological structures proposed in the fourth embodiment; Figure 6 Architecture diagram of the electronic device in this solution; Figure 7 Schematic diagram of the structure of the computer-readable storage medium in this solution.

[0011] The labels in the figure are: 500 - Electronic device; 501 - Bus; 502 - CPU; 503 - ROM; 504 - RAM; 505 - Communication port; 506 - Input / output component; 507 - Hard disk; 508 - User interface; 600 - Computer-readable storage medium. Detailed implementation manners

[0012] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations.

[0013] Embodiment 1:

[0014] Referring to Figure 1 as shown, this embodiment proposes a remote sensing big data automated processing method based on cloud computing, including: Obtain all remote sensing big data tasks to be processed in the current cloud, disassemble the remote sensing big data tasks to be processed into several meta-tasks, and obtain the meta-task features of the remote sensing big data tasks to be processed; Specifically, it is implemented through the following steps: Based on the processing logic of remote sensing big data tasks, disassemble the remote sensing big data tasks to be processed into several meta-tasks step by step. The meta-task types at least include data preprocessing tasks, feature extraction tasks, object recognition tasks, and result generation tasks; If there are data dependencies between meta-tasks, build a meta-task processing chain based on the data dependencies between meta-tasks. Meta-tasks with data dependencies need to perform data interaction during execution. By identifying these data dependencies during task disassembly, a structured basis is provided for subsequent screening of task processing nodes based on data interaction requirements; The meta-tasks disassembled from the remote sensing big data tasks to be processed and the meta-task processing chain form the meta-task features of the remote sensing big data tasks to be processed.

[0015] By analyzing the logical process of remote sensing data processing tasks, complex tasks are decomposed into atomic meta-task units, and characteristic parameters such as dependencies between tasks, data magnitudes, and calculation types are extracted. This decomposition method facilitates subsequent matching of computing resources according to the characteristics of different meta-tasks and provides a structured basis for constructing a task processing chain; Obtain the attributes of all task processing nodes in the cloud computing cluster, and construct a collaborative relationship graph of the cloud computing cluster based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster; By collecting the hardware configurations of nodes, such as information on CPU / GPU computing power, memory capacity, network bandwidth, geographical location, etc., and combining dynamic data such as the collaboration frequency and data transmission efficiency between nodes in historical tasks, construct a relationship network reflecting the collaboration ability between nodes. The collaborative relationship graph quantifies the communication cost and collaborative efficiency of node combinations, providing data support for subsequent topology optimization; Match the task attributes of each meta-task in the meta-task characteristics of the remote sensing big data to-be-processed task with the attributes of task processing nodes, and obtain the task adaptation degrees of each meta-task in the meta-task characteristics of the remote sensing big data to-be-processed task to all task processing nodes in the cloud computing cluster; By dynamically matching the resource requirements of meta-tasks, such as high parallel computing requirements and large memory dependencies, with the hardware characteristics of nodes, quantify the adaptation degree of each node to a specific meta-task.

[0016] Based on the task adaptation degrees of each meta-task in the meta-task characteristics of the remote sensing big data to-be-processed task to all task processing nodes in the cloud computing cluster and the collaborative relationship graph of the cloud computing cluster, screen out several initial screening task processing node topologies that meet the processing requirements of the remote sensing big data to-be-processed task; Comprehensively consider the single-node adaptation degree and the collaborative efficiency between nodes, and screen out candidate node combinations that simultaneously meet the independent processing requirements of meta-tasks and the collaborative requirements of the task chain. For example, preferentially select node clusters with high adaptation degrees and short communication distances to reduce cross-node communication latency and avoid task chain blockages caused by single-point performance bottlenecks.

[0017] Based on the overall analysis of all remote sensing big data to-be-processed tasks in the cloud, determine the optimal task processing node topology combination for all remote sensing big data to-be-processed tasks; Based on the optimal task processing node topology combination, allocate the meta-tasks corresponding to each optimal task processing node topology combination to the corresponding task processing nodes for automated processing.

[0018] This solution searches for a combination plan with the highest overall resource utilization rate and the least conflicts between tasks in the preliminary screening topology of multiple tasks. This process needs to balance the timeliness of single-task processing, the load balance of the cluster, and the cross-task resource competition relationship, and finally achieve the system-level optimal allocation of multi-task concurrency.

[0019] Embodiment 2:

[0020] Referring to Figure 2 As shown, on the basis of Embodiment 1, this embodiment further proposes: constructing a collaborative relationship graph of the cloud computing cluster based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster, which specifically includes: Calculating the initial task collaboration index between each task processing node based on the communication distance between each task processing node in the cloud computing cluster; The determination method of the initial task collaboration index is:

[0021] Wherein, is the initial task collaboration index of the i-th task processing node for the j-th task processing node, is the communication distance between the i-th task processing node and the j-th task processing node, is the maximum value among the communication distances between all pairs of task processing nodes, is the minimum value among the communication distances between all pairs of task processing nodes; When there is no historical task collaboration execution between nodes, at this time, due to the lack of collaboration samples between nodes, only the communication distance between nodes can be used to initialize the task collaboration index. Since the farther the communication distance between nodes, the longer the communication delay between nodes, therefore After performing positive normalization on the communication distance between nodes, and is compared to obtain the initial task collaboration index. In this way, the initial task collaboration index between the task processing nodes corresponding to the minimum communication distance is assigned as 1, and the initial task collaboration index is assigned to other nodes in turn based on the proportional relationship with the minimum communication distance; Constructing the initial task collaboration index through the positive normalization of the communication distance solves the cold start problem of node collaboration efficiency evaluation in the scenario without historical collaboration samples, enabling the physical communication characteristics between nodes to be quantified into comparable collaborative ability reference values. By mapping the minimum communication distance to the highest collaboration index, it intuitively reflects the low-latency communication advantage between nodes and avoids the decline in the execution efficiency of the task chain caused by too far communication distance; at the same time, gradient indexes are dynamically generated based on the proportional relationship, providing an initial optimization basis for subsequent task allocation, which not only ensures the rationality of the collaborative relationship graph in the initialization stage but also reserves an expansion space for dynamic optimization after historical data accumulation.

[0022] Based on the initial task collaboration metrics among task processing nodes, and combining the task collaboration processing records among task processing nodes in historical computing tasks, calculate the task collaboration metrics among task processing nodes in the cloud computing cluster; In this embodiment, a time forgetting mechanism is adopted to dynamically adjust the task collaboration metrics among task processing nodes. Based on the latest task collaboration processing record among task processing nodes, evaluate the task collaboration score. Then, perform a weighted sum of the task collaboration score based on the latest task collaboration processing record and the task collaboration metrics among task processing nodes before update to obtain the updated task collaboration metrics among task processing nodes. If the time interval between the latest task collaboration processing record and the previous task collaboration processing record is relatively long, increase the weight of the task collaboration score of the latest task collaboration processing record; Among them, the calculation formula for the task collaboration score is:

[0023] Among them, F is the task collaboration score, is the task requirement data volume, is the actual output data volume, is the task requirement data accuracy, is the actual output data accuracy, is the average communication delay of this collaboration, is the system maximum allowable delay threshold, usually set to 500ms.

[0024] Dynamically fuse historical and real-time collaboration data through the time forgetting mechanism, effectively improving the timeliness and adaptability of task collaboration metrics, enabling the system to quickly respond to node performance fluctuations or network environment changes. Evaluate the task collaboration score from multiple dimensions such as data volume matching degree, accuracy deviation, and communication delay in the latest collaboration record, and accurately quantify the collaboration effectiveness between nodes; by dynamically adjusting the weights of new and old data, both retain historical collaboration rules and strengthen the reference value of recent collaboration performance, avoiding index distortion caused by node performance degradation or upgrade. This mechanism can adaptively optimize the collaboration relationship graph between nodes, ensure that node combinations with high data fidelity and low communication delay are preferentially scheduled, thereby maintaining the execution efficiency of high-efficiency task chains in a complex and changing cloud environment, while enhancing the system's elastic response ability to sudden loads, and achieving a double improvement in resource utilization rate and task processing reliability.

[0025] Based on the task collaboration metrics among all task processing nodes in the cloud computing cluster, form the collaboration relationship graph of the cloud computing cluster.

[0026] By integrating the physical constraints of communication distance with the dynamic efficiency of historical collaboration, a multi-dimensional collaborative relationship graph is constructed, significantly enhancing the scientific nature and adaptability of task scheduling decisions. The initial index quantization based on communication distance solves the cold start problem of no historical collaboration data, ensuring the basic evaluation of node collaboration capabilities; the analysis of historical collaboration records combined with the time forgetting mechanism dynamically corrects the collaboration indicators between nodes, which can not only capture the impact of network environment changes and node performance fluctuations, but also avoid the misleading of stale data for current scheduling. The finally generated collaborative relationship graph provides an optimization basis for task chain allocation by quantifying the communication efficiency and collaboration stability of node combinations, taking into account both the real-time network state and long-term collaboration rules, enabling high-precision and low-latency node combinations to preferentially match data-intensive tasks, reducing cross-node communication overhead while ensuring the quality of task processing, and comprehensively improving the resource scheduling efficiency of the cloud cluster and the execution reliability of complex remote sensing task chains.

[0027] Embodiment 3:

[0028] Referring to Figure 3 As shown, on the basis of Embodiment 2, this embodiment further proposes specific steps for matching the task attributes of each meta-task in the meta-task features of the remote sensing big data to-be-processed task with the attributes of the task processing nodes, and obtaining the task adaptation degree of each meta-task in the meta-task features of the remote sensing big data to-be-processed task and all task processing nodes in the cloud computing cluster, including: Based on the historical processing task experience of remote sensing big data, determine the node hardware requirements during the processing of each meta-task; Based on the fitting analysis of the actual hardware attributes of the task processing nodes and the node hardware requirements during the processing of the meta-task, determine the satisfaction degree of the actual hardware attributes of the task processing nodes for the node hardware requirements during the processing of the meta-task, as the task adaptation degree; More specifically, the task adaptation degree in this embodiment is calculated in the following manner:

[0029] Wherein, is the task adaptation degree, is the requirement for the hardware performance of the nth node during the processing of the meta-task, is the actual hardware attribute of the nth node hardware performance of the task processing node, and m is the total number of types of node hardware performance required during the processing of the meta-task.

[0030] Through a dual-verification mechanism that matches historical experience with real-time hardware, the dynamic adaptation accuracy of tasks and node resources is significantly improved. Based on the hardware requirement portrait modeled from historical task data, the multi-dimensional requirements of meta-tasks for performance such as computing power, storage, and bandwidth are accurately extracted; by calculating the actual satisfaction of each hardware indicator through normalization and weighting, the impact of a single performance shortcoming on the overall adaptability is avoided, ensuring that node resources meet task requirements at the comprehensive performance level. This mechanism can not only adapt to performance changes brought about by node hardware upgrades or expansions, but also dynamically shield faulty nodes or low-performance devices, thereby optimizing resource utilization and reducing the risk of task execution delays or failures caused by insufficient hardware performance, providing a highly reliable and elastic underlying resource guarantee for complex remote sensing data processing.

[0031] Referring to Figure 4 As shown, further, this embodiment also proposes a method for screening out several initial screening task processing node topologies that meet the processing requirements of remote sensing big data to be processed based on the task adaptability of each meta-task in the meta-task characteristics of the remote sensing big data to be processed task and the collaborative relationship map of the cloud computing cluster, as follows: Record the task processing nodes with task adaptability exceeding the threshold as the target nodes of the meta-task, and the threshold is set to 0.85; Select one task processing node from the target nodes of each meta-task to form a single-point satisfaction task processing node topology for the remote sensing big data to be processed task; Based on the meta-task processing chain, extract the collaborative node chain from the single-point satisfaction task processing node topology, and the task processing nodes in the collaborative node chain are the task processing nodes for processing the meta-tasks in the meta-task processing chain; Based on the collaborative relationship map of the cloud computing cluster, extract the task collaboration indicators between the nodes in the collaborative node chain; Based on the comprehensive analysis of the task adaptability of each task processing node in the collaborative node chain and the task collaboration indicators between the nodes, obtain the collaborative processing indicators of the collaborative node chain for the meta-task processing chain; since there are data dependencies between the meta-tasks in the meta-task processing chain, when analyzing the task processing nodes corresponding to the meta-task processing chain, the task collaboration capabilities between these nodes need to be considered. Therefore, in this solution, the task collaboration indicators between these nodes and the adaptability between the meta-task and the nodes are extracted for comprehensive evaluation of the comprehensive matching degree between the collaborative node chain and the meta-task processing chain. The specific analysis formula is:

[0032] Wherein, is the collaborative processing indicator of the collaborative node chain for the meta-task processing chain, is the number of nodes in the collaborative node chain, is a set composed of nodes in the collaborative node chain, is the task fitness corresponding to the o-th element in U, is the task collaboration index between the o-th element and the p-th element in U, and are both emphasis weights, and the emphasis weights are dynamically adjusted based on the data interaction requirements between the meta-task processing chains. Under the constraint of , when the data interaction requirements between the meta-task processing chains are high, increase the value of , otherwise decrease the value of ; Summarize the collaborative processing indicators of all collaborative node chains, and comprehensively analyze the task collaboration ability of a single point to meet the task processing node topology structure; Extract the single-point task processing node topology structure whose task collaboration ability meets the collaboration requirements as the preliminary screening task processing node topology structure.

[0033] By summarizing the average value of the collaborative processing indicators of all collaborative node chains in the single-point task processing node topology structure, it is used as the task collaboration ability of the single-point task processing node topology structure, and the single-point task processing node topology structure with a task collaboration ability greater than the set collaboration threshold is used as the preliminary screening task processing node topology structure.

[0034] Through the dynamic weight adjustment and multi-dimensional collaborative evaluation mechanism, effectively balance the correlation between the node performance independence and the collaboration efficiency in the meta-task processing chain, and significantly improve the scientificity and robustness of the complex task chain scheduling. Based on the screening of the fitness threshold to ensure that the basic performance of a single node meets the standard, avoiding the task execution risk caused by hardware shortcomings; by introducing a dynamic weight allocation strategy sensitive to data interaction requirements, adaptively adjust the contribution ratio of the node fitness and the collaboration index in the comprehensive evaluation, so that the high-data-dependent task chain preferentially matches the node combination with high communication efficiency and strong collaboration stability, reducing the cross-node data transmission bottleneck. At the same time, taking the average value of the overall efficiency of the collaborative node chain as the evaluation standard for the topology structure collaboration ability, avoiding the drag of local inefficient paths on the overall execution efficiency of the task chain, ensuring that the preliminary screening topology structure achieves the optimal balance among the single-node performance, the collaboration ability between nodes and the global requirements of the task chain, so as to maximize the resource utilization efficiency and system throughput while ensuring the reliability of task processing.

[0035] Example 4:

[0036] Referring to Figure 5 as shown, on the basis of Example 3, this example further proposes a method for overall analysis of all remote sensing big data tasks to be processed in the cloud and determining the optimal task processing node topology structure combination for all remote sensing big data tasks to be processed as follows: Randomly combine the topological structures of several preliminary screening task processing nodes for each remote sensing big data task to be processed to obtain all combinations of task processing node topological structures; Select the combination of task processing node topological structures with the highest comprehensive score based on the TOPSIS method as the optimal combination of task processing node topological structures; Set evaluation factors based on each combination of task processing node topological structures. Among them, the evaluation factors need to include the load balancing attribute of the task processing nodes to ensure the load balancing of the cloud computing cluster. On this basis, the evaluation factors can also be set, such as task processing speed attributes, data security attributes, etc. Set at least one evaluation index based on each evaluation factor, and construct an evaluation matrix composed of the evaluation indexes of all combinations of task processing node topological structures. The optimal value of each evaluation index forms an ideal optimal solution, and the optimal value of each evaluation index forms an ideal worst solution. Calculate the TOPSIS index of the combination of task processing node topological structures based on the vector distance between the evaluation indexes of each combination of task processing node topological structures and the ideal optimal solution and the ideal worst solution, and select the maximum value of the TOPSIS index as the optimal combination of task processing node topological structures.

[0037] Through the combination of a multi-dimensional evaluation system and a TOPSIS decision-making model, global optimal screening of task topology combinations is achieved, taking into account key objectives such as load balancing, processing efficiency, and data security. Based on the load balancing attribute, a core constraint is constructed to effectively avoid node overload or idle problems and ensure the long-term stability of cluster resources; by introducing extended evaluation indexes such as processing speed and data security, a multi-dimensional evaluation matrix is constructed to comprehensively quantify the comprehensive effectiveness of different topological combinations. The TOPSIS method is used to dynamically calculate the closeness of each combination to the ideal solution, which can not only avoid the local optimal trap caused by single-index optimization but also scientifically balance the conflict relationship between multiple objectives through vector distance analysis. This method supports dynamic adjustment of index weights to adapt to different task scenarios. For example, high-real-time tasks focus on processing speed, and sensitive data tasks strengthen security indicators to ensure that the optimal topological combination always fits the actual needs, ultimately realizing the coordinated optimization of task execution efficiency, resource utilization rate, and system reliability, and providing efficient and flexible scheduling decision support for complex remote sensing big data processing.

[0038] Furthermore, the method according to the embodiment of the present application can also be implemented with the help of Figure 6 the architecture of the electronic device shown. As Figure 6As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a ROM 503, a RAM 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, etc. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store a method for automatically processing remote sensing big data based on cloud computing provided by this application. The electronic device 500 may also include a user interface 508. Of course, Figure 6 the architecture shown is only exemplary. When implementing different devices, one or more components in the electronic device shown may be omitted according to actual needs. Figure 6

[0039] Figure 7 It is a schematic diagram of the structure of a computer-readable storage medium provided by an embodiment of this application. As Figure 7 shown, it is a computer-readable storage medium 600 according to an embodiment of this application. Computer-readable instructions are stored on the computer-readable storage medium 600. When the computer-readable instructions are run by a processor, a method for automatically processing remote sensing big data based on cloud computing according to the embodiment of this application described with reference to the above drawings can be executed. The storage medium 600 includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory, etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0040] In summary, the advantages of the present invention are as follows: By dynamically matching the meta-task characteristics with the computing node attributes and combining the collaborative relationship graph to achieve intelligent topology screening, the utilization rate of cloud resources and the task processing efficiency are effectively improved; Based on the collaborative metrics between nodes, the task chain scheduling path is optimized, significantly reducing the cross-node communication delay; Through the multi-task load balancing strategy and the global optimal topology combination, while ensuring the timeliness of single-task processing, the optimal allocation of the overall system resources is achieved. It can not only reduce the comprehensive time consumption of complex remote sensing data processing tasks, but also improve the utilization efficiency of cluster resources. At the same time, it adapts to the dynamic load changes in the multi-task concurrent scenario, taking into account both the processing speed and the system stability.

[0041] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art of this industry should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification is only the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A remote sensing big data automated processing method based on cloud computing, characterized in that, Including: Obtain all the remote sensing big data tasks to be processed in the current cloud, disassemble the remote sensing big data tasks to be processed into several meta-tasks, and obtain the meta-task features of the remote sensing big data tasks to be processed; Obtain the attributes of all task processing nodes in the cloud computing cluster, and build a collaborative relationship graph of the cloud computing cluster based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster; Match the task attributes of each meta-task in the meta-task features of the remote sensing big data tasks to be processed with the attributes of the task processing nodes, and obtain the task adaptation degrees of each meta-task in the meta-task features of the remote sensing big data tasks to be processed and all task processing nodes in the cloud computing cluster; Based on the task adaptation degrees of each meta-task in the meta-task features of the remote sensing big data tasks to be processed and all task processing nodes in the cloud computing cluster, and the collaborative relationship graph of the cloud computing cluster, screen out several initial screening task processing node topologies that meet the processing requirements of the remote sensing big data tasks to be processed; Conduct overall analysis based on all the remote sensing big data tasks to be processed in the cloud, and determine the optimal task processing node topology combination of all the remote sensing big data tasks to be processed; Based on the optimal task processing node topology combination, allocate the meta-tasks corresponding to each optimal task processing node topology combination to the corresponding task processing nodes for automated processing.

2. The automated processing method for remote sensing big data based on cloud computing according to claim 1, characterized in that, The disassembling the remote sensing big data tasks to be processed into several meta-tasks specifically includes: Based on the processing logic of the remote sensing big data processing tasks, disassemble the remote sensing big data tasks to be processed into several meta-tasks step by step; If there is a data dependency relationship between the meta-tasks, build a meta-task processing chain based on the data dependency relationship between the meta-tasks; The meta-tasks disassembled from the remote sensing big data tasks to be processed and the meta-task processing chain form the meta-task features of the remote sensing big data tasks to be processed.

3. A method for automatically processing remote sensing big data based on cloud computing according to claim 2, characterized in that The building the collaborative relationship graph of the cloud computing cluster based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster specifically includes: Based on the communication distances between the task processing nodes in the cloud computing cluster, calculate the initial task collaboration indicators between the task processing nodes; Based on the initial task collaboration indicators between the task processing nodes, combined with the task collaboration processing records between the task processing nodes in the historical computing tasks, calculate the task collaboration indicators between the task processing nodes in the cloud computing cluster; Based on the task collaboration indicators between all task processing nodes in the cloud computing cluster, form the collaborative relationship graph of the cloud computing cluster.

4. A remote sensing big data automatic processing method based on cloud computing according to claim 3, characterized in that The matching the task attributes of each meta-task in the meta-task features of the remote sensing big data tasks to be processed with the attributes of the task processing nodes, and obtaining the task adaptation degrees of each meta-task in the meta-task features of the remote sensing big data tasks to be processed and all task processing nodes in the cloud computing cluster specifically includes: Based on the experience of historical remote sensing big data processing tasks, determine the node hardware requirements for each meta-task processing; Based on the fitting analysis of the actual hardware attributes of the task processing nodes and the node hardware requirements during the processing of the meta-tasks, determine the degree to which the actual hardware attributes of the task processing nodes meet the node hardware requirements during the processing of the meta-tasks, and use it as the task adaptation degree.

5. A method for automatically processing remote sensing big data based on cloud computing according to claim 4, characterized in that, The specific process of screening out several initial screening task processing node topologies that meet the processing requirements of the remote sensing big data to be processed based on the task adaptation degree of each meta-task in the meta-task characteristics of the remote sensing big data to be processed task and all task processing nodes in the cloud computing cluster, as well as the collaborative relationship graph of the cloud computing cluster, includes: Record the task processing nodes with a task adaptation degree exceeding the threshold as the target nodes of the meta-tasks; Select one task processing node from the target nodes of each meta-task to form a single-point satisfaction task processing node topology for the remote sensing big data to be processed task; Based on the meta-task processing chain, extract the collaborative node chain from the single-point satisfaction task processing node topology, and the task processing nodes in the collaborative node chain are the task processing nodes for processing the meta-tasks in the meta-task processing chain; Based on the collaborative relationship graph of the cloud computing cluster, extract the task collaboration indicators between the nodes in the collaborative node chain; Based on the comprehensive analysis of the task adaptation degree of each task processing node in the collaborative node chain and the task collaboration indicators between the nodes, obtain the collaborative processing indicators of the collaborative node chain for the meta-task processing chain; Summarize the collaborative processing indicators of all collaborative node chains and comprehensively analyze the task collaboration ability of the single-point satisfaction task processing node topology; Extract the single-point satisfaction task processing node topology with a task collaboration ability that meets the collaboration requirements as the initial screening task processing node topology.

6. A method for automatically processing remote sensing big data based on cloud computing according to claim 5, characterized in that, The specific process of determining the optimal task processing node topology combination for all remote sensing big data to be processed tasks based on the overall analysis of all remote sensing big data to be processed tasks in the cloud includes: Based on the random combination of several initial screening task processing node topologies for each remote sensing big data to be processed task, obtain all task processing node topology combinations; Select the task processing node topology combination with the highest comprehensive score as the optimal task processing node topology combination based on the TOPSIS method.

7. An automated processing method for remote sensing big data based on cloud computing according to claim 6, characterized in that At least the load balancing of the task processing nodes is included in the evaluation factors of the TOPSIS method.

8. An electronic device, characterized in that, Includes: At least one processor; And a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method for automatically processing remote sensing big data based on cloud computing as described in any one of claims 1-7.

9. A computer-readable storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by the processor, a method for automatically processing remote sensing big data based on cloud computing as described in any one of claims 1-7 is implemented.

Images