Distributed computing task cooperative processing method and system based on cloud computing

Abstract

The invention relates to the technical field of task processing, in particular to a distributed computing task cooperative processing method and system based on cloud computing, and the method comprises the steps: obtaining a to-be-processed distributed computing task and task feature data of the distributed computing task; splitting the distributed computing task based on the task feature data to generate a plurality of sub-tasks corresponding to the distributed computing task; acquiring node feature data and historical load data of each computing node in the cloud computing platform; and based on the node feature data and the historical load data of each computing node, in combination with the task feature data of the plurality of sub-tasks, allocating resource nodes to the plurality of sub-tasks to obtain a resource node set corresponding to each sub-task. According to the method, the distributed computing tasks are subjected to feature analysis and splitting, and the resource nodes are reasonably allocated for each sub-task in combination with the features of each computing node and the historical load data, so that the utilization efficiency of resources is improved, and idle and waste of the resources are avoided.

Description

Technical Field

[0001] This invention relates to the field of task processing technology, specifically to a method and system for collaborative processing of distributed computing tasks based on cloud computing. Background Technology

[0002] Currently, traditional methods often employ static resource allocation strategies, which cannot be flexibly adjusted according to real-time load and task characteristics. This results in some computing nodes being idle for extended periods, while other nodes may become inefficient due to excessive load. Furthermore, when faced with sudden load changes, traditional methods typically lack dynamic adjustment mechanisms, making it impossible to migrate tasks or reallocate resources in a timely manner, thereby affecting the stability and high availability of the system.

[0003] Furthermore, traditional methods often lack fine-grained consideration in task splitting and scheduling, which may lead to improper handling of dependencies between tasks, increased data transmission latency, and reduced overall execution efficiency. Moreover, traditional distributed computing systems lack effective real-time monitoring mechanisms, making it impossible to obtain the load status of each computing node in a timely manner, resulting in an inability to make rapid decisions and optimizations. In addition, they usually rely on preset scheduling rules and lack analysis of task characteristics and historical loads, which makes task scheduling unable to adapt to the needs of different scenarios, resulting in poor performance in some cases. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following technical solution: a cloud computing-based distributed computing task collaborative processing method, comprising: Obtain the distributed computing task to be processed and the task feature data of the distributed computing task; Based on the task feature data, the distributed computing task is split to generate multiple sub-tasks corresponding to the distributed computing task. Each sub-task carries some or all of the data in the task feature data. Obtain node characteristic data and historical load data of each computing node in the cloud computing platform; Based on the node characteristic data and historical load data of each computing node, and combined with the task characteristic data of the multiple subtasks, resource nodes are allocated to the multiple subtasks to obtain the resource node set corresponding to each subtask. Based on the resource node set corresponding to each subtask, the logical dependencies between the multiple subtasks are parsed and the communication link characteristics between different resource nodes are constructed. A collaborative processing strategy between the multiple subtasks is generated, and the collaborative processing strategy is used to indicate the timing and path of data interaction between subtasks on different resource nodes. Using the aforementioned collaborative processing strategy, the execution efficiency data corresponding to the distributed computing task is estimated.

[0005] Preferably, the node characteristic data includes computing power level and network bandwidth level; the historical load data includes CPU utilization and memory utilization within a historical time period; Specifically, based on the node characteristic data and historical load data of each computing node, and combined with the task characteristic data of the multiple subtasks, resource nodes are allocated to the multiple subtasks to obtain a resource node set corresponding to each subtask, including: The node feature data and historical load data of each computing node are standardized and represented to generate the node standard feature information of each computing node. Based on the task feature data of the multiple sub-tasks, a computing node matching the multiple sub-tasks is found in the cloud computing platform; Obtain a matching score between the task feature data of any of the subtasks and the node standard feature information of the found computing nodes; The computing node with the highest matching score is determined as the primary resource node for the subtask. If the matching score is greater than or equal to the set allocation threshold, then the primary resource node is added to the resource node set corresponding to the subtask.

[0006] Preferably, based on the node characteristic data and historical load data of each computing node, and combined with the task characteristic data of the multiple subtasks, resource nodes are allocated to the multiple subtasks to obtain a resource node set corresponding to each subtask, further comprising: If the matching score is less than the allocation threshold, then a backup resource node is selected from the backup node pool of the cloud computing platform. Calculate the matching score between the task feature data of the subtask and the node standard feature information of the backup resource node; If the calculated matching score is greater than or equal to the allocation threshold, then the backup resource node is added to the resource node set corresponding to the subtask.

[0007] Preferably, based on the resource node set corresponding to each subtask, the logical dependencies between the multiple subtasks are parsed and the communication link characteristics between different resource nodes are constructed to generate a collaborative processing strategy among the multiple subtasks, including: The logical dependencies between the multiple subtasks are analyzed to obtain task dependency chain data; Based on the resource node set corresponding to each subtask, calculate the communication delay information between any two resource nodes and construct the communication link characteristics; According to the execution order determined by the task dependency link data, operation instructions are assigned to each subtask in sequence; Based on the network bandwidth determined by the communication link characteristics, the data transmission volume between adjacent subtasks is adjusted. The adjusted operation instructions and data transmission volume are combined into the collaborative processing strategy.

[0008] Preferably, after adopting the collaborative processing strategy, the method further includes: During the execution of the distributed computing task using the aforementioned collaborative processing strategy, the current load data of each computing node is obtained in real time. If the current load data indicates that any resource node is overloaded, then select the target subtask running on the overloaded resource node from the multiple subtasks; find an idle resource node with a load below a set load threshold in the resource node set; transfer the execution right of the target subtask from the overloaded resource node to the idle resource node, and update the collaborative processing strategy. After employing the aforementioned collaborative processing strategy to estimate the execution efficiency data corresponding to the distributed computing task, the method further includes: If the execution efficiency data meets the set performance requirements, the collaborative processing strategy will be sent to the scheduling center of the cloud computing platform. The scheduling center is controlled to allocate computing resources and start the distributed computing task according to the collaborative processing strategy.

[0009] Preferably, the collaborative processing strategy is used to estimate the execution efficiency data corresponding to the distributed computing task, including: The collaborative processing strategy is input into the efficiency prediction model, which is used to characterize the mapping relationship between the collaborative strategy and the task execution time. The efficiency prediction model is invoked to output the estimated execution time corresponding to the distributed computing task, which is used as the execution efficiency data; and... After the distributed computing task is completed, obtain the actual execution time; Calculate the deviation between the actual execution time and the estimated execution time; If the deviation value is greater than the set error range, the efficiency prediction model is updated with parameters based on the actual execution time.

[0010] Preferably, the task feature data includes task type, data volume, and processing time requirements; The process of acquiring the distributed computing task to be processed and the task feature data of the distributed computing task includes: Receive computing requests sent by user terminals and extract distributed computing tasks to be processed from the computing requests; Perform semantic analysis on the distributed computing task to identify task attribute tags; The task feature data is generated based on the task attribute tags.

[0011] Preferably, before allocating resource nodes to the multiple subtasks based on the node characteristic data and historical load data of each computing node, combined with the task characteristic data of the multiple subtasks, to obtain the resource node set corresponding to each subtask, the method further includes: Based on the task characteristic information, analyze the task type information of the distributed computing task; Based on the task type information and the node status information, the distributed computing task is divided into multiple task sub-modules; Calculate the communication delay information between any two computing nodes in the computing node cluster; Based on the module requirement information of the task sub-module and the node load capacity information, candidate transmission links are matched in the initial transmission link set. Based on the real-time load information of the candidate transmission links, predict the transmission efficiency fluctuation range of the candidate transmission links within a preset processing period; Determine whether the transmission efficiency fluctuation range information meets the preset efficiency requirement threshold information. If it is determined that the transmission efficiency fluctuation range information meets the preset efficiency requirement threshold information, then the candidate transmission link is determined as the target transmission link.

[0012] Preferably, based on the task type information and the node status information, the distributed computing task is divided into multiple task sub-modules, including: Based on the task type information, determine the logical dependency information of the distributed computing task; Based on the node status information, analyze the computing resource distribution information and network topology information of the computing node cluster; Based on the logical dependency information, identify the critical path tasks and non-critical path tasks in the distributed computing task; Combining the computing power resource distribution information and the network topology information, the critical path tasks and the non-critical path tasks are decomposed into multiple independent task sub-modules according to a preset splitting granularity.

[0013] A cloud computing-based distributed computing task collaborative processing system, applicable to the aforementioned cloud computing-based distributed computing task collaborative processing method, includes: The task acquisition unit is used to acquire the distributed computing task to be processed and the task feature data of the distributed computing task. The task splitting unit is used to split the distributed computing task based on the task feature data to generate multiple subtasks corresponding to the distributed computing task, and each subtask carries some or all of the data in the task feature data. The node acquisition unit acquires node characteristic data and historical load data of each computing node in the cloud computing platform. The task allocation unit is used to allocate resource nodes to the multiple sub-tasks based on the node feature data and historical load data of each computing node, combined with the task feature data of the multiple sub-tasks, so as to obtain the resource node set corresponding to each sub-task. The collaborative processing unit is used to parse the logical dependencies between the multiple subtasks and construct the communication link characteristics between different resource nodes based on the resource node set corresponding to each subtask, and generate a collaborative processing strategy between the multiple subtasks. The collaborative processing strategy is used to indicate the timing and path of data interaction between subtasks on different resource nodes. An execution estimation unit is used to estimate the execution efficiency data corresponding to the distributed computing task using the collaborative processing strategy.

[0014] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention analyzes and decomposes distributed computing tasks by combining the characteristics of each computing node and historical load data, and reasonably allocates resource nodes to each subtask, thereby improving the utilization efficiency of resources and avoiding resource idleness and waste. Moreover, by adopting a collaborative processing strategy and a real-time load monitoring mechanism, the execution path and resource allocation of the task can be adjusted in a timely manner to ensure that efficient computing performance is maintained during execution, reduce execution time, and meet the timeliness requirements of users. (2) This invention ensures the stability and high availability of the entire system by monitoring the load of computing nodes in real time and migrating overloaded nodes. This dynamic adjustment capability enables the system to respond quickly and optimize resource allocation when faced with sudden load changes. Moreover, by parsing the logical dependencies between multiple sub-tasks and constructing reasonable communication link characteristics, it can effectively reduce the data transmission latency between different resource nodes and improve the overall task execution efficiency. Furthermore, by performing fine-grained task splitting based on task type and node status information, tasks can better adapt to the resource characteristics of the cloud computing environment. At the same time, clarifying critical path tasks helps to optimize the entire computing process. Attached Figure Description

[0015] Figure 1 This is a schematic flowchart of the overall method in one embodiment of the present invention; Figure 2 This is a schematic diagram of the overall system architecture in one embodiment of the present invention.

[0016] In the diagram: 1. Task acquisition unit; 2. Task splitting unit; 3. Node acquisition unit; 4. Task allocation unit; 5. Collaborative processing unit; 6. Execution prediction unit. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1, please refer to Figure 1 This invention provides a technical solution: a collaborative processing method for distributed computing tasks based on cloud computing, comprising: S1. Obtain the distributed computing task to be processed and the task feature data of the distributed computing task; S2. Based on the task feature data, the distributed computing task is split to generate multiple sub-tasks corresponding to the distributed computing task. Each sub-task carries some or all of the data in the task feature data. S3. Obtain node characteristic data and historical load data of each computing node in the cloud computing platform; S4. Based on the node characteristic data and historical load data of each computing node, and combined with the task characteristic data of multiple subtasks, resource nodes are allocated to multiple subtasks to obtain the resource node set corresponding to each subtask. S5. Based on the resource node set corresponding to each subtask, parse the logical dependencies between multiple subtasks and construct the communication link characteristics between different resource nodes, and generate a collaborative processing strategy between multiple subtasks. The collaborative processing strategy is used to indicate the timing and path of data interaction between subtasks on different resource nodes. S6. Employ a collaborative processing strategy to estimate the execution efficiency data corresponding to distributed computing tasks.

[0019] It should be noted that the computational tasks to be processed are collected, and the characteristics of these tasks are recorded. Based on the characteristics of the tasks, large computational tasks are broken down into multiple smaller subtasks. Each subtask can be processed independently and may carry some or all of the task characteristic data for subsequent processing. In a cloud computing platform, each computing node (server) has its own characteristics, such as processing power, memory size, network bandwidth, etc., and also has historical load data, showing how many tasks the node has processed in the past time period and what its load status is. By combining the feature data of each subtask with the features and load of each computing node, a suitable resource node will be allocated to each subtask; this step aims to ensure that each subtask can run in the most suitable environment to achieve the best execution efficiency. There may be certain logical dependencies between multiple subtasks, such as the result of one subtask being the input of another. These dependencies will be resolved, and communication link characteristics between different resource nodes will be established based on these relationships to ensure that tasks can run in a coordinated manner. Based on the parsed dependencies and communication link characteristics, a collaborative processing strategy will be generated to indicate when and how subtasks on different resource nodes should interact and transmit data. By leveraging previously established collaborative processing strategies, we can predict the execution efficiency of the entire distributed computing task; this can help optimize resource allocation and task scheduling. Specific examples: Suppose a company is handling a large-scale data analysis task, such as analyzing a large amount of user behavior data to identify potential market trends; this task is very complex and involves a lot of data computation. This data analysis task was submitted to the cloud computing platform, and the task's characteristic data was defined, such as a data volume of 100GB, high computational complexity, and the expectation that machine learning models would be used for analysis. This data analysis task is broken down into 10 sub-tasks, each of which is responsible for analyzing a different part of the user data. For example, sub-task 1 analyzes the data from user A to user E, sub-task 2 analyzes the data from user F to user J, and so on. The cloud computing platform checked the currently available computing nodes and found that node 1 has high CPU performance but is currently under high load, node 2 has average performance but low load, and node 3 has good performance and is almost idle. Subtasks 1 and 2 are assigned to node 3 because it has the lowest load and can complete the tasks faster; while subtasks 3 and 4 are assigned to node 2 because its performance is sufficient to handle these tasks. The output data of subtask 1 and subtask 2 will be used as the input of subtask 5. This dependency relationship is identified, and a communication link is built from node 3 to node 2 to ensure that data can be exchanged smoothly between subtasks. A strategy was developed to instruct that after subtasks 1 and 2 are completed, the results should be immediately transmitted to node 2, and then subtask 5 should be started for subsequent analysis. Predict the overall execution time of the entire data analysis task to help project managers plan subsequent work accordingly.

[0020] In an optional embodiment, node characteristic data includes computing power level and network bandwidth level; historical load data includes CPU utilization and memory utilization within a historical time period; Specifically, based on the node characteristic data and historical load data of each computing node, combined with the task characteristic data of multiple subtasks, resource nodes are allocated to multiple subtasks, resulting in a set of resource nodes corresponding to each subtask, including: The node feature data and historical load data of each computing node are standardized and represented to generate the standard node feature information of each computing node. Based on the task feature data of multiple subtasks, find computing nodes in the cloud computing platform that match the multiple subtasks; Obtain a matching score between the task feature data of any subtask and the standard feature information of the found computing nodes; The computation node with the highest matching score is determined as the primary resource node for the subtask. If the matching score is greater than or equal to the set allocation threshold, the primary resource node will be added to the resource node set corresponding to the subtask.

[0021] It should be noted that each compute node has some characteristic data, including compute capability level (representing the node's processing power, such as high, medium, and low) and network bandwidth level; historical load data records the usage of compute nodes over a period of time, including CPU utilization and memory utilization, reflecting the actual load of the nodes. To make it easier to compare the feature data and load conditions of different nodes, this data is standardized; this means converting the raw data into a directly comparable format, making the feature information between different nodes comparable. Based on the task characteristic data of multiple subtasks, find the computing nodes that match these subtasks; the task characteristic data may include the computing power, memory requirements and network requirements of the task. For any subtask, a matching score is calculated between its task feature data and the standard feature information of the found computing nodes; this score can be based on a variety of factors, such as the degree of matching of processing capabilities, the degree of adaptability of bandwidth, etc. Based on the matching score, the computing node with the highest score is selected as the primary resource node for the subtask; this means that this node is most suitable for processing the subtask. If the matching score is higher than or equal to the set allocation threshold, the primary resource node will be added to the resource node set corresponding to the subtask for subsequent task scheduling. Specific examples: Suppose a cloud computing platform is processing a set of image processing tasks that require complex image recognition and analysis; now, resource nodes need to be allocated to these tasks. Node characteristic data: On this platform, there are three computing nodes: Node A: high computing power level, medium network bandwidth level; Node B: medium computing power level, high network bandwidth level; Node C: low computing power level, low network bandwidth level. Historical load data: In the past time period: Node A's CPU utilization was 70%, and memory utilization was 60%; Node B's CPU utilization was 50%, and memory utilization was 40%; Node C's CPU utilization was 90%, and memory utilization was 80%; Task characteristic data: There are two subtasks: Subtask 1: requires high computing power and medium network bandwidth; Subtask 2: requires medium computing power and high network bandwidth; The characteristics and load data of each node were standardized to obtain corresponding standard characteristic information, which facilitates subsequent comparisons. For subtask 1, node A is found to be the most suitable because it has high computing power and moderate network bandwidth; for subtask 2, node B is the most suitable because it meets the requirements of the subtask in terms of both computing power and network bandwidth. Based on the matching of computing power and network bandwidth, a matching score is calculated for each node in these two subtasks; for example, subtask 1 may get a score of 95 for node A, a score of 85 for node B, and node C may not be considered due to insufficient computing power. The primary resource node for subtask 1 is node A (with the highest matching score), and the primary resource node for subtask 2 is node B. Assuming the allocation threshold is set to 80 points; since the matching scores of nodes A and B are both higher than this threshold, they will be added to the resource node set of their respective subtasks.

[0022] In an optional embodiment, based on the node characteristic data and historical load data of each computing node, and combined with the task characteristic data of multiple subtasks, resource nodes are allocated to multiple subtasks to obtain a resource node set corresponding to each subtask, and the method further includes: If the matching score is less than the allocation threshold, a backup resource node will be selected from the backup node pool of the cloud computing platform. Calculate the matching score between the task feature data of the subtask and the node standard feature information of the backup resource node; If the calculated matching score is greater than or equal to the allocation threshold, then the spare resource node is added to the resource node set corresponding to the subtask.

[0023] It should be noted that when allocating resource nodes to multiple subtasks, a matching score is calculated based on the node's feature data and historical load. If the matching score of the main computing node of a subtask is lower than the set allocation threshold, it means that the node may not be suitable for processing the subtask. Cloud computing platforms typically maintain a pool of backup nodes, which contains additional computing nodes; these nodes may be in standby mode under normal circumstances, but can be quickly put into use when needed. When the matching score of the primary resource node is lower than the threshold, a backup resource node that meets the requirements of the subtask will be searched in the backup node pool. Calculate the matching score between the standard feature information of the backup resource node and the task feature data of the subtask; if this score is higher than or equal to the allocation threshold, the backup node will be added to the resource node set of the subtask to ensure that the task can be executed smoothly. Specific examples: Suppose there are two ongoing image processing subtasks on a cloud computing platform, along with a set of compute nodes and standby nodes; Computational Node Characteristics: Node X: High computing power level, medium network bandwidth level; Node Y: Medium computing power level, low network bandwidth level; Standby Node Characteristics: Standby Node Z: Medium computing power level, high network bandwidth level; Historical Load Data: Node X's CPU utilization is 40%, memory utilization is 30%; Node Y's CPU utilization is 85%, memory utilization is 80% (high load); Subtask Characteristics: Subtask A: Requires high computing power and medium network bandwidth; Subtask B: Requires medium computing power and high network bandwidth; Check nodes X and Y; node X has a high matching score (let's say 90 points), while node Y may have a low matching score, such as only 60 points, due to excessive load. Since node Y's score is below the set allocation threshold (let's say 70 points), it cannot be used as the primary resource node for subtask A. Check the standby node pool; the characteristic data of standby node Z indicates that it has moderate computing power and high network bandwidth, and it can be considered as a standby resource node. The matching score between the backup node Z and subtask A was calculated and found to be 80 points (meeting the requirements of subtask A). Since the matching score of the backup node Z is greater than the allocation threshold, it is added to the resource node set of subtask A. Recheck nodes X and Y; for subtask B, node Y's computing power is insufficient, so it is excluded; although node X has high computing power, its network bandwidth level is medium, which may not fully meet the requirements of subtask B. Suppose node X has a matching score of 65, which is still below the allocation threshold; check the backup node pool again, and backup node Z may be evaluated because of its moderate computing power and high network bandwidth; The matching score of backup node Z is found to be 75 (which meets the requirements of subtask B); because the score is higher than the set threshold, backup node Z is added to the resource node set of subtask B.

[0024] In an optional embodiment, based on the resource node set corresponding to each subtask, the logical dependencies between multiple subtasks are parsed and the communication link characteristics between different resource nodes are constructed to generate a collaborative processing strategy among multiple subtasks, including: Analyze the logical dependencies between multiple subtasks to obtain task dependency chain data; Based on the set of resource nodes corresponding to each subtask, calculate the communication delay information between any two resource nodes and construct the communication link characteristics; According to the execution order determined by the task dependency link data, operation instructions are assigned to each subtask in sequence; Based on the network bandwidth determined by the characteristics of the communication link, adjust the amount of data transmission between adjacent subtasks; The adjusted operation instructions and data transmission volume are combined into a collaborative processing strategy.

[0025] It should be noted that subtasks do not exist independently; there may be some kind of sequential or data dependency relationship between them. For example, subtask B may need to start executing after subtask A. By resolving these logical dependencies, task dependency chain data can be constructed to identify which tasks are prerequisites and which are subsequent tasks. Once the dependencies between subtasks are determined, the next step is to consider the communication latency information between resource nodes. This involves calculating the data transmission time and bandwidth between any two resource nodes to form the communication link characteristics between different resource nodes. These characteristics will help evaluate the efficiency of data transmission between the nodes. Based on the task dependency chain data, an execution order will be defined; in this order, each subtask will be assigned operation instructions one by one according to the dependency relationship to ensure that all necessary data is ready. Based on the characteristics of the communication link, the amount of data transmitted between adjacent subtasks can be determined; for example, if the network bandwidth between two subtasks is low, the amount of data that needs to be transmitted will be reduced to avoid data transmission bottlenecks. By combining the adjusted operation instructions and data transmission volume, a comprehensive collaborative processing strategy is formed; this strategy ensures that the various subtasks can cooperate efficiently, reduce waiting time, and improve overall performance. Specific examples: Suppose there is an image processing project that includes three subtasks: subtask A, subtask B, and subtask C; there are certain dependencies between them; subtask A is responsible for image preprocessing; subtask B depends on the output of subtask A to perform image analysis; subtask C generates a report based on the analysis results after subtask B is completed. It was identified that subtask B depends on the output of subtask A, while subtask C depends on the result of subtask B; therefore, the task dependency chain was determined to be A→B→C. Assume subtask A runs on node X, subtask B on node Y, and subtask C on node Z; calculate the communication latency between node X and node Y as 20 milliseconds, and the latency between node Y and node Z as 15 milliseconds; at the same time, measure the network bandwidth between node X and node Y as 100 MB / s, and the bandwidth between node Y and node Z as 50 MB / s. Based on the dependencies, operation instructions are first assigned to subtask A, and then to subtask B after it is completed, followed by subtask C. Because the bandwidth between node X and node Y is high (100MB / s), it is decided to transmit the entire image preprocessing result; however, between node Y and node Z, due to the low bandwidth (50MB / s), in order to avoid congestion, only a simplified version of the analysis result may be transmitted instead of the complete data. Combining this information, the following collaborative processing strategy is formed: First, subtask A is executed on node X to prepare the image preprocessing results; then, the processing results are transmitted to node Y in complete data form (20 milliseconds, 100 MB / s); once subtask B receives the data and completes its analysis, the simplified analysis results are transmitted to node Z in compressed format (15 milliseconds, 50 MB / s); finally, subtask C is executed on node Z and generates the final report.

[0026] In an optional embodiment, after employing a collaborative processing strategy, the method further includes: During the execution of distributed computing tasks using a collaborative processing strategy, the current load data of each computing node is obtained in real time. If the current load data indicates that any resource node is overloaded, then select the target subtask running on the overloaded resource node from multiple subtasks; find the idle resource node with a load below the set load threshold in the resource node set; transfer the execution right of the target subtask from the overloaded resource node to the idle resource node, and update the collaborative processing strategy. After employing a collaborative processing strategy to estimate the execution efficiency data corresponding to distributed computing tasks, the method also includes: If the execution efficiency data meets the set performance requirements, the collaborative processing strategy will be sent to the scheduling center of the cloud computing platform. The control and scheduling center allocates computing resources and initiates distributed computing tasks based on the collaborative processing strategy.

[0027] It should be noted that during execution, the load of each computing node will be continuously monitored; this means that the computing power, memory usage, and network bandwidth of each node can be understood in real time. If any compute node is detected to be overloaded (e.g., CPU or memory usage exceeds a set threshold), then subtasks that are running on that overloaded node will be filtered out from the running subtasks. Find idle resource nodes in the resource node set whose load is below a set threshold; these nodes can handle more computing tasks without affecting performance. Once suitable idle resource nodes are found, the execution rights of the target subtask will be migrated from the overloaded nodes to these idle nodes, and the collaborative processing strategy will be updated according to the new resource configuration; this process ensures the overall load balance. After migrating subtasks, the execution efficiency under the new collaborative processing strategy will be evaluated; if the execution efficiency meets the expected performance requirements, the strategy will be prepared to be deployed to the scheduling center of the cloud computing platform. The scheduling center allocates computing resources reasonably according to the issued collaborative processing strategy and starts distributed computing tasks to ensure that tasks are completed on time and optimize resource utilization. Specific examples: Suppose there is a video processing application that needs to execute multiple subtasks on different computing nodes, such as video encoding, effects addition, and final rendering; each subtask may run on different computing nodes, such as node A, node B, and node C; The video encoding task begins at node A, the effects addition task begins at node B, and the final rendering task begins at node C. As time progresses, real-time monitoring shows that the CPU load of node A has reached 90%, indicating that it is in an overloaded state. After determining that node A is overloaded, it was found that the video encoding task was still running, so it was marked as the target subtask; Querying other resource nodes reveals that node C is currently using only 30% of the CPU, with a load below the set threshold, making it a potential idle node. The video encoding task is migrated from node A to node C; at this time, the video encoding subtask will be started on node C, and all necessary data will be transferred to node C. After the migration is complete, update its collaborative processing strategy to reflect the new task allocation and ensure that other subtasks can proceed smoothly. A reassessment of the current collaborative processing strategy revealed that the execution efficiency improved to 95% after the migration, meeting the set performance requirements. The updated collaborative processing strategy is sent to the scheduling center of the cloud computing platform; the scheduling center reallocates computing resources according to this strategy and starts other related distributed computing tasks to ensure the smooth progress of the entire video processing process.

[0028] In an optional embodiment, a collaborative processing strategy is employed to estimate the execution efficiency data corresponding to the distributed computing task, including: The collaborative processing strategy is input into the efficiency prediction model, which is used to characterize the mapping relationship between the collaborative strategy and the task execution time. The efficiency prediction model outputs the estimated execution time for the distributed computing task, which serves as the execution efficiency data; and... After the distributed computing task is completed, obtain the actual execution time; Calculate the deviation between the actual execution time and the estimated execution time; If the deviation value is greater than the set error range, the efficiency prediction model parameters will be updated using the actual execution time.

[0029] It should be noted that the collaborative processing strategy is input into the efficiency prediction model; the role of this model is to describe the relationship between the collaborative processing strategy and the task execution time, and to help predict the completion time of the task under different conditions. The efficiency prediction model specifically uses a neural network. After invoking the efficiency prediction model, it will output an estimated execution time based on the input collaborative processing strategy; this estimated time represents the time required for the distributed computing task under ideal conditions. Once a distributed computing task is completed, the actual execution time is recorded. This is the time that the task actually takes to run, which may be affected by various factors, such as resource contention and changes in node load. Compare the actual execution time with the estimated execution time and calculate the deviation between the two. If the deviation exceeds the set error range (for example, the difference between the estimated and actual time exceeds 10%), it indicates that there is a problem with the accuracy of the prediction model. If the deviation exceeds the set range, the parameters of the efficiency prediction model will be updated using the actual execution time; this update process helps to improve the accuracy of future predictions, enabling the model to better reflect the actual situation. Specific examples: Suppose there is an online image processing platform. A user submits a batch processing task containing hundreds of images, which need to be compressed and converted in format. In order to complete this task efficiently, the platform adopts a collaborative processing strategy, which distributes the task across multiple computing nodes. Based on the current collaborative processing strategy, relevant information (such as the number of nodes participating in the processing, the load of each node, etc.) is input into the efficiency prediction model; this model is trained with historical data and can predict the execution time of tasks based on different strategies and conditions. The efficiency prediction model yields an estimated execution time of 30 minutes; this is the best estimate based on the current workload and resource configuration. As the task was executed, image processing began; however, due to a sudden increase in the load on some nodes, the task processing speed slowed down; ultimately, the task took 45 minutes to complete. The deviation between the actual execution time (45 minutes) and the estimated execution time (30 minutes) was calculated, and the result was 15 minutes. Since this deviation exceeds the set error range (e.g., 10 minutes), it indicates that there is a problem with the accuracy of the prediction model. To improve future prediction accuracy, the parameters of the efficiency prediction model are updated based on the actual execution time. This real-time feedback is incorporated into the model to enhance its predictive ability in similar situations, enabling more accurate execution time predictions when handling similar batch image tasks in the future.

[0030] In an optional embodiment, the task characteristic data includes task type, data volume, and processing time requirements; This includes acquiring the distributed computing task to be processed and its task characteristic data, including: Receive computing requests sent by user terminals and extract distributed computing tasks to be processed from the computing requests; Perform semantic analysis on distributed computing tasks to identify task attribute tags; Task feature data is generated based on task attribute tags.

[0031] It should be noted that the system receives computing requests from user terminals; these requests typically contain basic information about the computing tasks the user wishes to perform. Extract the specific distributed computing tasks to be processed from the received computing requests; this means identifying the operations that the user wants to perform, such as data analysis, image processing, or machine learning training. Semantic analysis is performed on the extracted tasks; this process aims to understand the specific requirements and context of the tasks, thereby identifying the attribute labels of the tasks; these labels may include the nature of the tasks (such as computationally intensive or memory intensive), the type of input data, the priority of the tasks, etc. Based on the identified task attribute tags, corresponding task feature data is generated. This feature data typically includes task type, data volume, and processing time requirements, providing necessary information for subsequent task scheduling and resource allocation. Specific examples: Suppose an online image processing service platform receives a computational request from a user who wants to compress a batch of hundreds of photos and convert them to a specific format; A user submitted a request through the platform's application: "Please compress 500 JPEG photos I uploaded into PNG format, and the processing time should not exceed 10 minutes." Upon receiving the request, the system identifies the specific task the user wishes to perform as "image compression and format conversion." At this point, the task is marked as a pending task and is prepared for further analysis. Semantic analysis was performed on the extracted tasks to identify some key attribute labels: Task type: image processing; Data volume: 500 photos; Input format: JPEG; Output format: PNG; Processing time requirement: to be completed within 10 minutes; Based on the identified attribute tags, task feature data was generated. The results show that the task type is image processing, the data volume is medium (500 images, assuming each image is about 2MB in size, totaling about 1GB of data), and the processing time requirement is high (it needs to be completed within 10 minutes). This task feature data will be used for subsequent task scheduling and resource allocation to ensure that computing resources are allocated effectively to meet the user's processing time requirements.

[0032] In an optional embodiment, before allocating resource nodes to multiple subtasks based on node characteristic data and historical load data of each computing node, combined with task characteristic data of multiple subtasks, to obtain a resource node set corresponding to each subtask, the method further includes: Analyze the task type information of distributed computing tasks based on task characteristic information; Based on task type information and node status information, the distributed computing task is divided into multiple task sub-modules; Calculate the communication latency information between any two compute nodes in a compute node cluster; Based on the module requirement information and node load capacity information of the task sub-modules, candidate transmission links are matched from the initial transmission link set. Based on the real-time load information of the candidate transmission links, predict the range of transmission efficiency fluctuations of the candidate transmission links within a preset processing period. Determine whether the transmission efficiency fluctuation range information meets the preset efficiency requirement threshold information. If it is determined that the transmission efficiency fluctuation range information meets the preset efficiency requirement threshold information, then the candidate transmission link is determined as the target transmission link.

[0033] It should be noted that analyzing the type of distributed computing task based on task characteristic data (such as task type, data volume, etc.) helps to understand the nature of the task and the types of resources it may need. Based on the analyzed task type information and the current node status, large distributed computing tasks are broken down into multiple smaller subtasks or modules; this breakdown enables tasks to be processed in parallel, thereby improving computing efficiency. Evaluate the communication latency between any two compute nodes in the compute node cluster; this information is crucial for subsequent resource allocation and task scheduling, as high latency can affect the overall execution time of tasks. Based on the requirements of each submodule (such as data bandwidth, processing capacity, etc.) and the load capacity of the nodes, suitable candidate transmission links will be matched from the initial set of transmission links; these links are potential communication paths used for data transmission. After identifying candidate transmission links, the real-time load information of these links is analyzed to predict the range of transmission efficiency fluctuations within a preset processing period; this step ensures that the load changes that the links may face in the future can be anticipated. Determine whether the predicted transmission efficiency fluctuation range meets the preset efficiency requirement threshold; if it does, then determine the candidate transmission link as the target transmission link for subsequent task execution. Specific examples: Suppose a cloud computing company is handling a complex data analysis task. The user wants to analyze a large dataset containing a lot of sensor data to identify anomalous patterns. The received task feature data indicates that the user submitted a data analysis task, which belongs to the "data mining" type; based on this information, it is understood that the task requires strong computing power and certain data storage resources. Based on the nature of the task and the current load of the computing nodes, it was decided to break down this data analysis task into multiple sub-modules, such as: data preprocessing; feature extraction; model training; anomaly detection; each sub-module can be processed in parallel on different computing nodes, thereby improving execution efficiency; Measure the communication latency between each computing node; for example, the latency from node A to node B is 10 milliseconds, and the latency from node C to node B is 15 milliseconds. This latency information will be used for subsequent link selection. During data preprocessing, it is determined that the processed data needs to be transmitted to node B for feature extraction; based on module requirements (e.g., the amount of data to be transmitted and bandwidth requirements) and the load capacity of the node, candidate links that can support this transmission are selected. Analyzing the real-time load data of the candidate transmission link, it is predicted that the transmission efficiency of the link may fluctuate in the next 15 minutes; calculations show that the predicted efficiency fluctuation range is between 70% and 90%. Assuming the preset efficiency requirement threshold is 60%, and since the predicted efficiency fluctuation range (70% to 90%) is all above this threshold, the candidate transmission link is finally determined as the target transmission link, and the data is prepared to be sent to node B.

[0034] In an optional embodiment, the distributed computing task is divided into multiple task sub-modules based on task type information and node status information, including: Based on the task type information, determine the logical dependency information of the distributed computing tasks; Based on node status information, analyze and compute the distribution of computing resources and network topology of the node cluster; Based on logical dependency information, identify critical path tasks and non-critical path tasks in distributed computing tasks; Combining computing resource distribution information and network topology information, critical path tasks and non-critical path tasks are decomposed into multiple independent task sub-modules according to a preset splitting granularity.

[0035] It should be noted that the logical dependencies within the distributed computing tasks need to be analyzed based on the task type information; this means identifying which task sub-modules have dependencies on each other, for example, the output of one sub-module may be the input of another sub-module. Analyze the resource distribution and network topology information of the computing node cluster; the computing resource distribution information includes the computing power, memory and storage resources of each node, while the network topology information describes the connection method and communication latency between the nodes. After understanding the logical dependencies of tasks, we can identify which tasks are critical path tasks and which are non-critical path tasks. Critical path tasks are those tasks that, if delayed, will directly affect the overall task completion time, while non-critical path tasks are relatively independent, and delaying them will not affect the overall execution progress. Based on the analyzed information on the distribution of computing resources and network topology, critical path tasks and non-critical path tasks are decomposed into multiple independent task sub-modules according to a preset granularity. These sub-modules can be allocated to different computing nodes for parallel execution, thereby improving the overall computing efficiency. Specific examples: Suppose an online image processing platform needs to process a batch of user-uploaded images for style transfer and image enhancement; this task involves multiple steps, as detailed below: Determine logical dependency information: The platform's tasks include the following steps: image upload; image preprocessing; style transfer; image enhancement; image output; the logical dependencies are as follows: image preprocessing must be performed before style transfer, while image enhancement can be performed after style transfer; therefore, there is a dependency between image preprocessing and style transfer. Assessing the current computing node status reveals that node A has strong computing power (e.g., abundant GPU resources), making it suitable for image conversion tasks; while node B has more abundant storage resources, making it suitable for storing and processing large amounts of image data; in addition, the network latency between node A and node B is low, making it suitable for frequent data exchange. Based on logical dependencies, image preprocessing and style transfer are identified as the critical path tasks, because the execution order of these two steps directly affects the time of the final image output; while image enhancement is considered a non-critical path task, because even if this step is delayed, as long as style transfer is completed, the overall task completion time will not be affected. For critical path tasks, it was decided to break down "image preprocessing" into multiple smaller tasks, such as image denoising and image scaling; "style transfer" was broken down into sub-modules, such as style selection and style application; meanwhile, the non-critical path task "image enhancement" was also broken down into brightness adjustment and contrast adjustment. Each submodule is designed as an independent task that can be processed in parallel on different nodes; for example, image denoising can be processed on node A, while image scaling can be performed on node B.

[0036] Example 2, please refer to Figure 2 This invention provides a technical solution: a cloud computing-based distributed computing task collaborative processing system, applicable to the aforementioned cloud computing-based distributed computing task collaborative processing method, comprising: Task acquisition unit 1 is used to acquire the distributed computing task to be processed and the task feature data of the distributed computing task; Task splitting unit 2 is used to split the distributed computing task based on task feature data to generate multiple subtasks corresponding to the distributed computing task. Each subtask carries some or all of the data in the task feature data. Node acquisition unit 3 acquires node characteristic data and historical load data of each computing node in the cloud computing platform; Task allocation unit 4 is used to allocate resource nodes to multiple subtasks based on the node feature data and historical load data of each computing node, combined with the task feature data of multiple subtasks, so as to obtain the resource node set corresponding to each subtask. The collaborative processing unit 5 is used to parse the logical dependencies between multiple subtasks and construct the communication link characteristics between different resource nodes based on the resource node set corresponding to each subtask, and generate a collaborative processing strategy between multiple subtasks. The collaborative processing strategy is used to indicate the timing and path of data interaction between subtasks on different resource nodes. The execution estimation unit 6 is used to estimate the execution efficiency data corresponding to the distributed computing task by adopting a collaborative processing strategy.

[0037] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A method for collaborative processing of distributed computing tasks based on cloud computing, characterized in that, include: Obtain the distributed computing task to be processed and the task feature data of the distributed computing task; Based on the task feature data, the distributed computing task is split to generate multiple sub-tasks corresponding to the distributed computing task. Each sub-task carries some or all of the data in the task feature data. Obtain node characteristic data and historical load data of each computing node in the cloud computing platform; Based on the node characteristic data and historical load data of each computing node, and combined with the task characteristic data of the multiple subtasks, resource nodes are allocated to the multiple subtasks to obtain the resource node set corresponding to each subtask. Based on the resource node set corresponding to each subtask, the logical dependencies between the multiple subtasks are parsed and the communication link characteristics between different resource nodes are constructed. A collaborative processing strategy between the multiple subtasks is generated, and the collaborative processing strategy is used to indicate the timing and path of data interaction between subtasks on different resource nodes. Using the aforementioned collaborative processing strategy, the execution efficiency data corresponding to the distributed computing task is estimated.

2. The cloud computing-based distributed computing task collaborative processing method according to claim 1, characterized in that, The node characteristic data includes computing power level and network bandwidth level; the historical load data includes CPU utilization and memory utilization within a historical time period. Specifically, based on the node characteristic data and historical load data of each computing node, and combined with the task characteristic data of the multiple subtasks, resource nodes are allocated to the multiple subtasks to obtain a resource node set corresponding to each subtask, including: The node feature data and historical load data of each computing node are standardized and represented to generate the node standard feature information of each computing node. Based on the task feature data of the multiple sub-tasks, a computing node matching the multiple sub-tasks is found in the cloud computing platform; Obtain a matching score between the task feature data of any of the subtasks and the node standard feature information of the found computing nodes; The computing node with the highest matching score is determined as the primary resource node for the subtask. If the matching score is greater than or equal to the set allocation threshold, then the primary resource node is added to the resource node set corresponding to the subtask.

3. The cloud computing-based distributed computing task collaborative processing method according to claim 2, characterized in that, Based on the node characteristic data and historical load data of each computing node, and combined with the task characteristic data of the multiple subtasks, resource nodes are allocated to the multiple subtasks to obtain a resource node set corresponding to each subtask, and the allocation also includes: If the matching score is less than the allocation threshold, then a backup resource node is selected from the backup node pool of the cloud computing platform. Calculate the matching score between the task feature data of the subtask and the node standard feature information of the backup resource node; If the calculated matching score is greater than or equal to the allocation threshold, then the backup resource node is added to the resource node set corresponding to the subtask.

4. The cloud computing-based distributed computing task collaborative processing method according to claim 3, characterized in that, Based on the resource node set corresponding to each subtask, the logical dependencies between the multiple subtasks are parsed and the communication link characteristics between different resource nodes are constructed to generate a collaborative processing strategy among the multiple subtasks, including: The logical dependencies between the multiple subtasks are analyzed to obtain task dependency chain data; Based on the resource node set corresponding to each subtask, calculate the communication delay information between any two resource nodes and construct the communication link characteristics; According to the execution order determined by the task dependency link data, operation instructions are assigned to each subtask in sequence; Based on the network bandwidth determined by the communication link characteristics, the data transmission volume between adjacent subtasks is adjusted. The adjusted operation instructions and data transmission volume are combined into the collaborative processing strategy.

5. The cloud computing-based distributed computing task collaborative processing method according to claim 4, characterized in that, After adopting the aforementioned collaborative processing strategy, the method further includes: During the execution of the distributed computing task using the aforementioned collaborative processing strategy, the current load data of each computing node is obtained in real time. If the current load data indicates that any resource node is overloaded, then select the target subtask running on the overloaded resource node from the multiple subtasks; find an idle resource node with a load below a set load threshold in the resource node set; transfer the execution right of the target subtask from the overloaded resource node to the idle resource node, and update the collaborative processing strategy. After employing the aforementioned collaborative processing strategy to estimate the execution efficiency data corresponding to the distributed computing task, the method further includes: If the execution efficiency data meets the set performance requirements, the collaborative processing strategy will be sent to the scheduling center of the cloud computing platform. The scheduling center is controlled to allocate computing resources and start the distributed computing task according to the collaborative processing strategy.

6. The cloud computing-based distributed computing task collaborative processing method according to claim 5, characterized in that, Using the aforementioned collaborative processing strategy, the execution efficiency data corresponding to the distributed computing task is estimated, including: The collaborative processing strategy is input into the efficiency prediction model, which is used to characterize the mapping relationship between the collaborative strategy and the task execution time. The efficiency prediction model is invoked to output the estimated execution time corresponding to the distributed computing task, which is used as the execution efficiency data; and... After the distributed computing task is completed, obtain the actual execution time; Calculate the deviation between the actual execution time and the estimated execution time; If the deviation value is greater than the set error range, the efficiency prediction model is updated with parameters based on the actual execution time.

7. The distributed computing task collaborative processing method based on cloud computing according to claim 6, characterized in that, The task characteristic data includes task type, data volume, and processing time requirements; The process of acquiring the distributed computing task to be processed and the task feature data of the distributed computing task includes: Receive computing requests sent by user terminals and extract distributed computing tasks to be processed from the computing requests; Perform semantic analysis on the distributed computing task to identify task attribute tags; The task feature data is generated based on the task attribute tags.

8. The cloud computing-based distributed computing task collaborative processing method according to claim 7, characterized in that, Before allocating resource nodes to the multiple subtasks based on the node characteristic data and historical load data of each computing node, combined with the task characteristic data of the multiple subtasks, to obtain the resource node set corresponding to each subtask, the method further includes: Based on the task characteristic information, analyze the task type information of the distributed computing task; Based on the task type information and the node status information, the distributed computing task is divided into multiple task sub-modules; Calculate the communication delay information between any two computing nodes in the computing node cluster; Based on the module requirement information of the task sub-module and the node load capacity information, candidate transmission links are matched in the initial transmission link set. Based on the real-time load information of the candidate transmission links, predict the transmission efficiency fluctuation range of the candidate transmission links within a preset processing period; Determine whether the transmission efficiency fluctuation range information meets the preset efficiency requirement threshold information. If it is determined that the transmission efficiency fluctuation range information meets the preset efficiency requirement threshold information, then the candidate transmission link is determined as the target transmission link.

9. The cloud computing-based distributed computing task collaborative processing method according to claim 8, characterized in that, Based on the task type information and the node status information, the distributed computing task is divided into multiple task sub-modules, including: Based on the task type information, determine the logical dependency information of the distributed computing task; Based on the node status information, analyze the computing resource distribution information and network topology information of the computing node cluster; Based on the logical dependency information, identify the critical path tasks and non-critical path tasks in the distributed computing task; Combining the computing power resource distribution information and the network topology information, the critical path tasks and the non-critical path tasks are decomposed into multiple independent task sub-modules according to a preset splitting granularity.

10. A cloud computing-based distributed computing task collaborative processing system, applicable to the cloud computing-based distributed computing task collaborative processing method according to any one of claims 1-9, characterized in that, include: The task acquisition unit is used to acquire the distributed computing task to be processed and the task feature data of the distributed computing task. The task splitting unit is used to split the distributed computing task based on the task feature data to generate multiple subtasks corresponding to the distributed computing task, and each subtask carries some or all of the data in the task feature data. The node acquisition unit acquires node characteristic data and historical load data of each computing node in the cloud computing platform. The task allocation unit is used to allocate resource nodes to the multiple sub-tasks based on the node feature data and historical load data of each computing node, combined with the task feature data of the multiple sub-tasks, so as to obtain the resource node set corresponding to each sub-task. The collaborative processing unit is used to parse the logical dependencies between the multiple subtasks and construct the communication link characteristics between different resource nodes based on the resource node set corresponding to each subtask, and generate a collaborative processing strategy between the multiple subtasks. The collaborative processing strategy is used to indicate the timing and path of data interaction between subtasks on different resource nodes. An execution estimation unit is used to estimate the execution efficiency data corresponding to the distributed computing task using the collaborative processing strategy.

Images