A Remote Sensing Big Data Automatic Processing Method Based on Cloud Computing

Abstract

The present invention discloses a method for automated processing of remote sensing big data based on cloud computing, which relates to the field of remote sensing data processing technology, including: obtaining meta-task features of remote sensing big data tasks to be processed; constructing a collaborative relationship map of a cloud computing cluster; obtaining the task adaptability of each meta-task in the meta-task features of the remote sensing big data tasks to be processed and all task processing nodes in the cloud computing cluster; screening out a number of preliminary screening task processing node topologies that meet the processing requirements of the remote sensing big data tasks to be processed; determining the optimal task processing node topology structure combination of all remote sensing big data tasks to be processed; and allocating the meta-task corresponding to each optimal task processing node topology structure combination to the corresponding task processing node for automated processing. The advantages of the present invention are: by dynamically matching meta-task features with computing node attributes, combined with the collaborative relationship map to achieve intelligent topology screening, effectively improving cloud resource utilization and task processing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing data processing, and in particular to a method for automatic processing of remote sensing big data based on cloud computing. Background Art

[0002] The rapid development of remote sensing technology has led to an explosive growth in remote sensing data. Traditional single-machine processing models are no longer able to cope with the real-time processing demands of this massive data. With the widespread adoption of cloud computing, distributed computing has provided computing power for remote sensing big data processing. However, existing cloud platforms often employ static resource allocation strategies when scheduling tasks, failing to fully consider the logical dependencies between remote sensing data processing tasks and the collaborative nature of computing nodes. In particular, in multi-tasking concurrent scenarios, problems such as coarse task scheduling granularity, low node collaboration efficiency, and uneven resource utilization arise, resulting in inefficient data processing and wasted computing resources.

[0003] Existing technologies for processing remote sensing data tasks typically employ simple task queue scheduling mechanisms, lacking the ability to intelligently analyze task topologies and dynamically adapt computing resources. This crude scheduling approach can easily lead to problems such as uneven node loads and excessive cross-node communication overhead, making it difficult to meet the collaborative requirements of complex processing flows such as remote sensing data preprocessing, feature extraction, and intelligent interpretation. Particularly when processing task chains with data dependencies, traditional methods are unable to effectively quantify the collaborative efficiency between nodes, hindering the overall scheduling optimization of cloud cluster resources. Summary of the Invention

[0004] In order to solve the above technical problems, a cloud computing-based remote sensing big data automated processing method is provided. This technical solution solves the problem that the above-mentioned existing technologies usually adopt a simple task queue scheduling mechanism when processing remote sensing data tasks, lacking the ability to intelligently analyze the task topology structure and dynamically adapt computing resources.

[0005] In order to achieve the above objects, the technical solution adopted by the present invention is:

[0006] A cloud computing-based remote sensing big data automated processing method, comprising:

[0007] Obtain all the remote sensing big data tasks to be processed on the current cloud, decompose the remote sensing big data tasks to be processed into several meta-tasks, and obtain the meta-task features of the remote sensing big data tasks to be processed;

[0008] Obtain the attributes of all task processing nodes in the cloud computing cluster, and build a collaborative relationship map of the cloud computing cluster based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster;

[0009] Based on the task attributes of each meta-task in the meta-task features of the remote sensing big data to be processed and the attributes of the task processing nodes, the task compatibility of each meta-task in the meta-task features of the remote sensing big data to be processed and all the task processing nodes in the cloud computing cluster is obtained;

[0010] Based on the task compatibility between each meta-task in the meta-task characteristics of the remote sensing big data to be processed and all task processing nodes in the cloud computing cluster, as well as the collaborative relationship map of the cloud computing cluster, several preliminary task processing node topologies that meet the processing requirements of the remote sensing big data to be processed are screened out;

[0011] Conduct a comprehensive analysis of all remote sensing big data tasks to be processed on the cloud and determine the optimal task processing node topology combination for all remote sensing big data tasks to be processed;

[0012] Based on the optimal task processing node topology structure combination, the meta-task corresponding to each optimal task processing node topology structure combination is assigned to the corresponding task processing node for automatic processing.

[0013] Preferably, the constructing of the collaborative relationship graph of the cloud computing cluster based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster specifically includes:

[0014] Based on the communication distance between each task processing node in the cloud computing cluster, the initial task coordination index between each task processing node is calculated;

[0015] Based on the initial task coordination indicators between task processing nodes and the task coordination processing records between task processing nodes in historical computing tasks, the task coordination indicators between task processing nodes in the cloud computing cluster are calculated;

[0016] Based on the task coordination indicators among all task processing nodes in the cloud computing cluster, a coordination relationship map of the cloud computing cluster is formed.

[0017] Preferably, the task attributes of each meta-task in the meta-task features of the remote sensing big data task to be processed are matched with the attributes of the task processing node, and the task compatibility of each meta-task in the meta-task features of the remote sensing big data task to be processed and all the task processing nodes in the cloud computing cluster is obtained, specifically comprising:

[0018] Based on the historical experience of remote sensing big data processing tasks, determine the node hardware requirements for each meta-task processing;

[0019] Based on the fitting analysis of the actual hardware attributes of the task processing node and the node hardware requirements during meta-task processing, the degree to which the actual hardware attributes of the task processing node meet the node hardware requirements during meta-task processing is determined as the task adaptation degree.

[0020] Preferably, the task compatibility between each meta-task in the meta-task characteristics of the remote sensing big data to-be-processed task and all task processing nodes in the cloud computing cluster, as well as the collaborative relationship map of the cloud computing cluster, is used to screen out several preliminary screening task processing node topologies that meet the processing requirements of the remote sensing big data to-be-processed task. Specifically, the topologies include:

[0021] The task processing node whose task fitness exceeds the threshold is recorded as the target node of the meta-task;

[0022] A task processing node is selected from the target node of each meta-task to form a single point satisfying task processing node topology structure for remote sensing big data processing tasks to be processed;

[0023] Based on the meta-task processing chain, a collaborative node chain is extracted from the single-point satisfied task processing node topology structure, wherein the task processing nodes in the collaborative node chain are task processing nodes that process meta-tasks in the meta-task processing chain;

[0024] Based on the collaborative relationship graph of the cloud computing cluster, the task collaboration indicators between nodes in the collaborative node chain are extracted;

[0025] Based on the comprehensive analysis of the task adaptability of each task processing node in the collaborative node chain and the task collaboration indicators between nodes, the collaborative processing indicators of the collaborative node chain for the meta-task processing chain are obtained;

[0026] Summarize the collaborative processing indicators of all collaborative node chains and comprehensively analyze the task collaboration capabilities of a single point that meets the task processing node topology structure;

[0027] The single point task processing node topology structure that satisfies the task collaboration requirements is extracted as the primary screening task processing node topology structure.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention realizes intelligent topology screening by dynamically matching meta-task features with computing node attributes and combining collaborative relationship maps, effectively improving cloud resource utilization and task processing efficiency; optimizes task chain scheduling paths based on inter-node collaborative indicators, significantly reducing cross-node communication delays; and achieves optimal allocation of overall system resources while ensuring the timeliness of single-task processing through a combination of multi-task load balancing strategies and global optimal topology. This can not only reduce the overall time consumption of complex remote sensing data processing tasks, but also improve cluster resource utilization efficiency. At the same time, it adapts to dynamic load changes in multi-task concurrent scenarios, taking into account both processing speed and system stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1This is a flow chart of the automated processing method for remote sensing big data based on cloud computing proposed in Example 1;

[0031] Figure 2 This is a flow chart of the method for constructing a collaborative relationship map of a cloud computing cluster proposed in Example 2;

[0032] Figure 3 This is a flow chart of the method for obtaining the task adaptability between a meta-task and a task processing node proposed in Example 3;

[0033] Figure 4 This is a flow chart of the method for screening out the topological structure of the primary screening task processing nodes proposed in Example 3;

[0034] Figure 5 This is a flow chart of the method for determining the optimal task processing node topology combination proposed in Example 4;

[0035] Figure 6 This is a diagram of the architecture of the electronic equipment in this solution;

[0036] Figure 7 This is a schematic diagram of the computer-readable storage medium structure in this solution.

[0037] The numbers in the figure are:

[0038] 500 - electronic device; 501 - bus; 502 - CPU; 503 - ROM; 504 - RAM; 505 - communication port; 506 - input / output component; 507 - hard disk; 508 - user interface; 600 - computer-readable storage medium. DETAILED DESCRIPTION

[0039] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are merely examples, and those skilled in the art may conceive of other obvious variations.

[0040] Example 1:

[0041] Reference Figure 1 As shown, this embodiment proposes a cloud computing-based remote sensing big data automatic processing method, including:

[0042] Obtain all the remote sensing big data tasks to be processed on the current cloud, decompose the remote sensing big data tasks to be processed into several meta-tasks, and obtain the meta-task features of the remote sensing big data tasks to be processed;

[0043] This is achieved through the following steps:

[0044] Based on the remote sensing big data processing task processing logic, the remote sensing big data processing task is decomposed into several meta-tasks in steps. The meta-task types include at least data preprocessing tasks, feature extraction tasks, object recognition tasks and result generation tasks;

[0045] If there are data dependencies between meta-tasks, a meta-task processing chain is constructed based on the data dependencies between the meta-tasks. Meta-tasks with data dependencies need to interact with each other during execution. By identifying these data dependencies during task decomposition, a structural foundation is provided for the subsequent screening of task processing nodes based on data interaction requirements.

[0046] The meta-tasks and meta-task processing chains derived from the remote sensing big data to be processed constitute the meta-task features of the remote sensing big data to be processed.

[0047] By analyzing the logical flow of remote sensing data processing tasks, complex tasks are decomposed into atomic meta-task units, and characteristic parameters such as inter-task dependencies, data levels, and computation types are extracted. This decomposition method facilitates the subsequent matching of computing resources to different meta-task characteristics and provides a structural foundation for building task processing chains.

[0048] Obtain the attributes of all task processing nodes in the cloud computing cluster, and build a collaborative relationship map of the cloud computing cluster based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster;

[0049] By collecting node hardware configuration information, such as CPU / GPU computing power, memory capacity, network bandwidth, and geographic location, and combining it with dynamic data such as the frequency of node collaboration and data transmission efficiency in historical tasks, a relationship network reflecting the collaborative capabilities of nodes is constructed. The collaborative relationship map quantifies the communication cost and collaborative efficiency of node combinations, providing data support for subsequent topology optimization.

[0050] Based on the task attributes of each meta-task in the meta-task features of the remote sensing big data to be processed and the attributes of the task processing nodes, the task compatibility of each meta-task in the meta-task features of the remote sensing big data to be processed and all the task processing nodes in the cloud computing cluster is obtained;

[0051] By dynamically matching the resource requirements of meta-tasks, such as high parallel computing requirements and large memory dependencies with the hardware characteristics of the nodes, the adaptability of each node to a specific meta-task is quantified.

[0052] Based on the task compatibility between each meta-task in the meta-task characteristics of the remote sensing big data to be processed and all task processing nodes in the cloud computing cluster, as well as the collaborative relationship map of the cloud computing cluster, several preliminary task processing node topologies that meet the processing requirements of the remote sensing big data to be processed are screened out;

[0053] By combining the adaptability of individual nodes and the effectiveness of inter-node collaboration, we select candidate node combinations that meet both the independent processing requirements of meta-tasks and the collaborative needs of task chains. For example, we prioritize node clusters with high adaptability and short communication distances to reduce cross-node communication latency and avoid task chain blockages caused by single-point performance bottlenecks.

[0054] Conduct a comprehensive analysis of all remote sensing big data tasks to be processed on the cloud and determine the optimal task processing node topology combination for all remote sensing big data tasks to be processed;

[0055] Based on the optimal task processing node topology structure combination, the meta-task corresponding to each optimal task processing node topology structure combination is assigned to the corresponding task processing node for automatic processing.

[0056] This solution searches for combinations that maximize overall resource utilization and minimize inter-task conflicts within a pre-screened topology of multiple tasks. This process balances single-task processing timeliness, cluster load balancing, and inter-task resource competition to ultimately achieve optimal system-level allocation of concurrent tasks.

[0057] Example 2:

[0058] Reference Figure 2 As shown, based on the first embodiment, this embodiment further proposes: based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster, building a collaborative relationship map of the cloud computing cluster specifically includes:

[0059] Based on the communication distance between each task processing node in the cloud computing cluster, the initial task coordination index between each task processing node is calculated;

[0060] The initial indicators of task coordination are determined as follows:

[0061]

[0062] in, is the initial task coordination indicator of the i-th task processing node and the j-th task processing node, is the communication distance between the i-th task processing node and the j-th task processing node, is the maximum value of the communication distance between all task processing node pairs, The minimum value of the communication distance between all pairs of task processing nodes;

[0063] When there is no task collaborative execution history between nodes, due to the lack of collaborative samples between nodes, the task collaborative index can only be initialized based on the communication distance between nodes. Since the longer the communication distance between nodes, the longer the communication delay between nodes, After the communication distance between nodes is processed positively, By comparing, we can get the initial task collaboration index. In this way, the task collaboration initial index between the task processing nodes corresponding to the minimum communication distance is assigned to 1, and the task collaboration initial index between other nodes is assigned in turn based on the proportional relationship with the minimum communication distance.

[0064] By constructing initial task collaboration indicators through forward processing of communication distance, the cold start problem of node collaboration performance evaluation in scenarios without historical collaboration samples is solved, allowing the physical communication characteristics between nodes to be quantified as comparable collaborative capability reference values. By mapping the minimum communication distance to the highest collaboration indicator, the advantage of low-latency communication between nodes is intuitively reflected, avoiding the decline in task chain execution efficiency caused by excessive communication distance. At the same time, gradient indicators are dynamically generated based on proportional relationships, providing an initial optimization basis for subsequent task allocation. This not only ensures the rationality of the collaboration relationship map during the initialization phase, but also reserves expansion space for dynamic optimization after the accumulation of historical data.

[0065] Based on the initial task coordination indicators between task processing nodes and the task coordination processing records between task processing nodes in historical computing tasks, the task coordination indicators between task processing nodes in the cloud computing cluster are calculated;

[0066] In this embodiment, a time forgetting mechanism is used to dynamically adjust the task collaboration index between each task processing node. The task collaboration score is evaluated based on the latest task collaboration processing record between the task processing nodes. Then, a weighted sum is taken of the task collaboration score based on the latest task collaboration processing record and the task collaboration index between the task processing nodes before the update to obtain the updated task collaboration index between the task processing nodes. If the time interval between the latest task collaboration processing record and the previous task collaboration processing record is long, the weight of the task collaboration score of the latest task collaboration processing record is increased.

[0067] The calculation formula for task collaboration score is:

[0068]

[0069] Among them, F is the task collaboration score, The amount of data required for the task, is the actual output data volume, To meet the data accuracy requirements of the task, is the actual output data accuracy, is the average communication delay of this collaboration, The maximum allowable delay threshold of the system, usually set to 500ms.

[0070] By dynamically integrating historical and real-time collaborative data through the time-forgetting mechanism, the timeliness and adaptability of task collaboration indicators are effectively improved, enabling the system to quickly respond to node performance fluctuations or changes in the network environment. Task collaboration scores are evaluated based on multiple dimensions such as data volume matching, accuracy deviation, and communication delay in the latest collaboration records to accurately quantify the collaboration efficiency between nodes; by dynamically adjusting the weights of new and old data, it not only retains historical collaboration rules but also strengthens the reference value of recent collaboration performance, avoiding indicator distortion caused by node performance degradation or upgrades. This mechanism can adaptively optimize the collaborative relationship map between nodes, ensuring that node combinations with high data fidelity and low communication latency are scheduled first, thereby maintaining efficient task chain execution efficiency in complex and changing cloud environments, while enhancing the system's elastic response capabilities to sudden loads, and achieving a dual improvement in resource utilization and task processing reliability.

[0071] Based on the task coordination indicators among all task processing nodes in the cloud computing cluster, a coordination relationship map of the cloud computing cluster is formed.

[0072] By integrating the physical constraints of communication distance with the dynamic efficiency of historical collaboration, a multi-dimensional collaborative relationship map is constructed, significantly improving the scientific nature and adaptability of task scheduling decisions. The quantification of initial indicators based on communication distance solves the cold start problem without historical collaboration data, ensuring a basic assessment of node collaboration capabilities. Combined with the analysis of historical collaboration records using the time-forgetting mechanism, the collaborative indicators between nodes are dynamically corrected, capturing the impact of network environment changes and node performance fluctuations while preventing outdated data from misleading current scheduling. The resulting collaborative relationship map quantifies the communication efficiency and collaboration stability of node combinations, providing an optimized basis for task chain allocation that balances real-time network status and long-term collaboration patterns. This prioritizes high-precision, low-latency node combinations for data-intensive tasks, ensuring task processing quality while reducing cross-node communication overhead. This comprehensively improves the resource scheduling efficiency of cloud clusters and the execution reliability of complex remote sensing task chains.

[0073] Example 3:

[0074] Reference Figure 3 As shown, based on the second embodiment, this embodiment further proposes matching the task attributes of each meta-task in the meta-task features of the remote sensing big data task to be processed with the attributes of the task processing node. The specific steps of obtaining the task compatibility of each meta-task in the meta-task features of the remote sensing big data task to be processed with all task processing nodes in the cloud computing cluster include:

[0075] Based on the historical experience of remote sensing big data processing tasks, determine the node hardware requirements for each meta-task processing;

[0076] Based on the fitting analysis of the actual hardware attributes of the task processing node and the node hardware requirements during meta-task processing, the degree to which the actual hardware attributes of the task processing node meet the node hardware requirements during meta-task processing is determined as the task adaptation degree;

[0077] More specifically, the task suitability in this embodiment is calculated as follows:

[0078]

[0079] in, is task suitability, The requirement for the hardware performance of the nth node when processing meta-tasks. is the actual hardware attribute of the nth node hardware performance of the task processing node, and m is the total number of node hardware performance types required for meta-task processing.

[0080] Through the dual verification mechanism of historical experience and real-time hardware matching, the dynamic adaptation accuracy of tasks and node resources is significantly improved. Based on the hardware demand portrait modeled by historical task data, the multi-dimensional requirements of meta-tasks for computing power, storage, bandwidth and other performance are accurately extracted; through normalized weighted calculation of the actual satisfaction of each hardware indicator, the impact of a single performance shortcoming on overall adaptability is avoided, ensuring that node resources meet task requirements at the comprehensive performance level. This mechanism can not only adapt to performance changes caused by node hardware upgrades or expansions, but also dynamically shield faulty nodes or low-performance devices, thereby optimizing resource utilization and reducing the risk of task execution delays or failures due to insufficient hardware performance, providing highly reliable and flexible underlying resource guarantees for complex remote sensing data processing.

[0081] Reference Figure 4 As shown, further, this embodiment also proposes a method for screening several preliminary task processing node topologies that meet the processing requirements of remote sensing big data tasks to be processed based on the task compatibility of each meta-task in the meta-task characteristics of the remote sensing big data tasks to be processed and all task processing nodes in the cloud computing cluster, as well as the collaborative relationship map of the cloud computing cluster:

[0082] The task processing node whose task fitness exceeds the threshold is recorded as the target node of the meta-task, and the threshold is set to 0.85;

[0083] A task processing node is selected from the target node of each meta-task to form a single point satisfying task processing node topology structure for remote sensing big data processing tasks to be processed;

[0084] Based on the meta-task processing chain, a collaborative node chain is extracted from the single-point satisfied task processing node topology structure. The task processing nodes in the collaborative node chain are the task processing nodes that process the meta-tasks in the meta-task processing chain.

[0085] Based on the collaborative relationship graph of the cloud computing cluster, the task collaboration indicators between nodes in the collaborative node chain are extracted;

[0086] Based on the comprehensive analysis of the task adaptability of each task processing node in the collaborative node chain and the task coordination index between nodes, the collaborative processing index of the collaborative node chain for the meta-task processing chain is obtained; since there is a data dependency relationship between the meta-tasks in the meta-task processing chain, when analyzing the task processing nodes corresponding to the meta-task processing chain, it is necessary to consider the task coordination ability between these nodes. Therefore, in this scheme, the task coordination index between these nodes and the adaptability between meta-tasks and nodes are extracted to comprehensively evaluate the comprehensive matching degree between the collaborative node chain and the meta-task processing chain. The specific analysis formula is:

[0087]

[0088] in, is the collaborative processing index of the collaborative node chain for the meta-task processing chain, is the number of nodes in the collaborative node chain, is a collection of nodes in the collaborative node chain. is the task fitness corresponding to the oth element in U, is the task collaboration index between the oth element and the pth element in U, 、 Both focus on weights, which are dynamically adjusted based on the data interaction requirements between meta-task processing chains. Under the constraints of Otherwise, reduce The value of

[0089] Summarize the collaborative processing indicators of all collaborative node chains and comprehensively analyze the task collaboration capabilities of a single point that meets the task processing node topology structure;

[0090] The single point task processing node topology structure that satisfies the task collaboration requirements is extracted as the primary screening task processing node topology structure.

[0091] The average value of the collaborative processing indicators of all collaborative node chains in the single-point task processing node topology structure is summarized as the task collaboration capability of the single-point task processing node topology structure, and the single-point task processing node topology structure with a task collaboration capability greater than the set collaboration threshold is used as the initial screening task processing node topology structure.

[0092] Through dynamic weight adjustment and multi-dimensional collaborative evaluation mechanism, the correlation between node performance independence and collaborative efficiency in the meta-task processing chain is effectively balanced, significantly improving the scientificity and robustness of complex task chain scheduling. Based on the adaptation threshold screening, the basic performance of a single node is ensured to meet the standards, avoiding the task execution risks caused by hardware shortcomings; by introducing a dynamic weight allocation strategy that is sensitive to data interaction requirements, the contribution ratio of node adaptation and collaborative indicators in the comprehensive evaluation is adaptively adjusted, so that high data-dependent task chains give priority to matching node combinations with high communication efficiency and strong collaborative stability, reducing cross-node data transmission bottlenecks. At the same time, the overall performance average of the collaborative node chain is used as the evaluation standard for the collaborative capability of the topology structure to avoid the drag of local inefficient paths on the overall execution efficiency of the task chain, ensuring that the initial screening topology structure achieves the optimal balance between single-node performance, inter-node collaboration capabilities and the global requirements of the task chain, thereby maximizing resource utilization efficiency and system throughput while ensuring task processing reliability.

[0093] Example 4:

[0094] Reference Figure 5 As shown, based on the third embodiment, this embodiment further proposes a method for comprehensively analyzing all remote sensing big data tasks to be processed on the cloud and determining the optimal task processing node topology combination for all remote sensing big data tasks to be processed:

[0095] Based on the random combination of several preliminary screening task processing node topologies of each remote sensing big data task to be processed, all task processing node topology combinations are obtained;

[0096] Based on the TOPSIS method, the task processing node topology structure combination with the highest comprehensive score is selected as the optimal task processing node topology structure combination;

[0097] Evaluation factors are set based on each task processing node topology structure combination, wherein the evaluation factors need to include the load balancing attributes of the task processing nodes to ensure the load balancing of the cloud computing cluster. On this basis, the evaluation factors can also be set, such as task processing speed attributes, data security attributes, etc. At least one evaluation index is set based on each evaluation factor, and an evaluation matrix composed of the evaluation indexes of all task processing node topology structure combinations is constructed. The ideal optimal solution is formed by screening the optimal value of each evaluation index, and the ideal worst solution is formed by screening the optimal value of each evaluation index. The TOPSIS index of the task processing node topology structure combination is calculated based on the vector distance between the evaluation index of each task processing node topology structure combination and the ideal optimal solution and the ideal worst solution, and the maximum value of the TOPSIS index is screened out as the optimal task processing node topology structure combination.

[0098] By combining a multi-dimensional evaluation system with the TOPSIS decision-making model, a globally optimal selection of task topology combinations is achieved, balancing key objectives such as load balancing, processing efficiency, and data security. Core constraints are established based on load balancing properties to effectively avoid node overload or idleness and ensure the long-term stability of cluster resources. By introducing extended evaluation metrics such as processing speed and data security, a multi-dimensional evaluation matrix is constructed to comprehensively quantify the overall performance of different topology combinations. The TOPSIS method dynamically calculates the closeness of each combination to the ideal solution, avoiding the local optimality trap caused by single-metric optimization and scientifically balancing conflicts between multiple objectives through vector distance analysis. This method supports dynamic adjustment of metric weights to adapt to different task scenarios, such as prioritizing processing speed for high-real-time tasks and strengthening security metrics for sensitive data tasks. This ensures that the optimal topology combination always meets actual requirements, ultimately achieving the coordinated optimization of task execution efficiency, resource utilization, and system reliability, providing efficient and flexible scheduling decision support for complex remote sensing big data processing.

[0099] Furthermore, the method according to the embodiment of the present application can also be used with the aid of Figure 6 The electronic device architecture shown in FIG. Figure 6 As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a ROM 503, a RAM 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, etc. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store a method for automatically processing remote sensing big data based on cloud computing provided by the present application. The electronic device 500 may also include a user interface 508. Of course, Figure 6 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 6 One or more components of an electronic device are shown.

[0100] Figure 7 This is a schematic diagram of the computer-readable storage medium structure provided by an embodiment of the present application. Figure 7 , a computer-readable storage medium 600 according to one embodiment of the present application is shown. Computer-readable instructions are stored on the computer-readable storage medium 600. When the computer-readable instructions are executed by a processor, the method for automated processing of remote sensing big data based on cloud computing according to an embodiment of the present application described with reference to the above figures can be executed. The storage medium 600 includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc.

[0101] In summary, the advantages of the present invention are: by dynamically matching meta-task features with computing node attributes, combined with collaborative relationship maps to achieve intelligent topology screening, effectively improving cloud resource utilization and task processing efficiency; optimizing task chain scheduling paths based on inter-node collaborative indicators, significantly reducing cross-node communication delays; through the combination of multi-task load balancing strategies and global optimal topology, while ensuring the timeliness of single task processing, the optimal allocation of overall system resources is achieved, which can not only reduce the comprehensive time consumption of complex remote sensing data processing tasks, but also improve the efficiency of cluster resource utilization, while adaptively adapting to dynamic load changes in multi-task concurrent scenarios, taking into account both processing speed and system stability.

[0102] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A cloud computing-based remote sensing big data automated processing method, characterized in that: include: Obtain all the remote sensing big data tasks to be processed on the current cloud, decompose the remote sensing big data tasks to be processed into several meta-tasks, and obtain the meta-task features of the remote sensing big data tasks to be processed; Obtain the attributes of all task processing nodes in the cloud computing cluster, and build a collaborative relationship map of the cloud computing cluster based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster; Based on the task attributes of each meta-task in the meta-task features of the remote sensing big data to be processed and the attributes of the task processing nodes, the task compatibility of each meta-task in the meta-task features of the remote sensing big data to be processed and all the task processing nodes in the cloud computing cluster is obtained; Based on the task compatibility between each meta-task in the meta-task characteristics of the remote sensing big data to be processed and all task processing nodes in the cloud computing cluster, as well as the collaborative relationship map of the cloud computing cluster, several preliminary task processing node topologies that meet the processing requirements of the remote sensing big data to be processed are screened out; Conduct a comprehensive analysis of all remote sensing big data tasks to be processed on the cloud and determine the optimal task processing node topology combination for all remote sensing big data tasks to be processed; Based on the optimal task processing node topology structure combination, the meta-task corresponding to each optimal task processing node topology structure combination is assigned to the corresponding task processing node for automatic processing; The task adaptability of each meta-task in the meta-task features of the remote sensing big data to be processed task and all task processing nodes in the cloud computing cluster, as well as the collaborative relationship map of the cloud computing cluster, is used to screen out several preliminary task processing node topologies that meet the processing requirements of the remote sensing big data to be processed task. Specifically, the topologies include: The task processing node whose task fitness exceeds the threshold is recorded as the target node of the meta-task; A task processing node is selected from the target node of each meta-task to form a single point satisfying task processing node topology structure for remote sensing big data processing tasks to be processed; Based on the meta-task processing chain, a collaborative node chain is extracted from the single-point satisfied task processing node topology structure, wherein the task processing nodes in the collaborative node chain are task processing nodes that process meta-tasks in the meta-task processing chain; Based on the collaborative relationship graph of the cloud computing cluster, the task collaboration indicators between nodes in the collaborative node chain are extracted; Based on the comprehensive analysis of the task adaptability of each task processing node in the collaborative node chain and the task collaboration indicators between nodes, the collaborative processing indicators of the collaborative node chain for the meta-task processing chain are obtained; Summarize the collaborative processing indicators of all collaborative node chains and comprehensively analyze the task collaboration capabilities of a single point that meets the task processing node topology structure; The single point task processing node topology structure that satisfies the task collaboration requirements is extracted as the primary screening task processing node topology structure.

2. The method for automatic processing of remote sensing big data based on cloud computing according to claim 1, characterized in that: The decomposition of the remote sensing big data processing task into several meta-tasks specifically includes: Based on the remote sensing big data processing task processing logic, the remote sensing big data processing task is decomposed into several meta-tasks in steps; If there is data dependency between meta-tasks, a meta-task processing chain is constructed based on the data dependency between meta-tasks; The meta-tasks and meta-task processing chains derived from the remote sensing big data to be processed constitute the meta-task features of the remote sensing big data to be processed.

3. The method for automatic processing of remote sensing big data based on cloud computing according to claim 2, characterized in that: The construction of the collaborative relationship map of the cloud computing cluster based on the attributes of all task processing nodes in the cloud computing cluster and the historical computing tasks of the cloud computing cluster specifically includes: Based on the communication distance between each task processing node in the cloud computing cluster, the initial task coordination index between each task processing node is calculated; Based on the initial task coordination indicators between task processing nodes and the task coordination processing records between task processing nodes in historical computing tasks, the task coordination indicators between task processing nodes in the cloud computing cluster are calculated; Based on the task coordination indicators among all task processing nodes in the cloud computing cluster, a coordination relationship map of the cloud computing cluster is formed.

4. The method for automatic processing of remote sensing big data based on cloud computing according to claim 3, characterized in that: The task attributes of each meta-task in the meta-task features of the remote sensing big data to be processed are matched with the attributes of the task processing nodes to obtain the task compatibility of each meta-task in the meta-task features of the remote sensing big data to be processed with all the task processing nodes in the cloud computing cluster. Specifically, the task attributes of each meta-task in the meta-task features of the remote sensing big data to be processed are matched with the attributes of the task processing nodes. Based on the historical experience of remote sensing big data processing tasks, determine the node hardware requirements for each meta-task processing; Based on the fitting analysis of the actual hardware attributes of the task processing node and the node hardware requirements during meta-task processing, the degree to which the actual hardware attributes of the task processing node meet the node hardware requirements during meta-task processing is determined as the task adaptation degree.

5. The method for automatic processing of remote sensing big data based on cloud computing according to claim 4, characterized in that: The above-mentioned comprehensive analysis of all remote sensing big data tasks to be processed on the cloud to determine the optimal task processing node topology combination for all remote sensing big data tasks to be processed specifically includes: Based on the random combination of several preliminary screening task processing node topologies of each remote sensing big data task to be processed, all task processing node topology combinations are obtained; Based on the TOPSIS method, the task processing node topology structure combination with the highest comprehensive score is selected as the optimal task processing node topology structure combination.

6. The method for automatic processing of remote sensing big data based on cloud computing according to claim 5, characterized in that: The evaluation factors of the TOPSIS method include at least the load balance of the task processing nodes.

7. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a cloud computing-based remote sensing big data automatic processing method as described in any one of claims 1-6.

8. A computer-readable storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by a processor, a cloud computing-based remote sensing big data automatic processing method according to any one of claims 1 to 6 is implemented.

Images