Multi-task collaborative data processing method and system based on cloud computing

Abstract

The embodiment of the invention provides a multi-task collaborative data processing method and system based on cloud computing, and belongs to the technical field of cloud computing. The method comprises the following steps: obtaining an initial task, and decomposing the initial task to obtain a plurality of sub-tasks and task relationships among the sub-tasks; performing clustering processing on the sub-tasks according to the task relationship to obtain a plurality of target tasks; determining a target optimization model according to the plurality of target tasks, and determining a target load and a target execution sequence corresponding to the target tasks according to the target optimization model; and processing target data corresponding to the target task according to the target load and the target execution sequence to obtain a target processing result corresponding to the target task. The problem that when an existing scheduler manages and allocates subtasks, resource waste or task delay may occur is solved, and the task execution efficiency and the resource utilization rate are improved.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a multi-task collaborative data processing method and system based on cloud computing. Background Art

[0002] With the development of cloud computing technology, cloud computing platforms provide powerful computing power and storage resources to support large-scale data processing tasks. At the same time, cloud computing platforms also provide a variety of tools and services, such as schedulers, distributed computing architectures, data sharing and collaboration tools, elastic scaling and automation, data transmission and synchronization, real-time monitoring and optimization, security and permission management, etc., which can support the implementation of multi-task collaborative data processing. However, when managing and allocating subtasks, existing schedulers may cause problems such as resource waste or task delays, and may also affect the performance and efficiency of the system. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to provide a multi-task collaborative data processing method and system based on cloud computing, aiming to solve the problems of resource waste or task delay that may occur when the scheduler in the related technology manages and allocates subtasks.

[0004] In a first aspect, an embodiment of the present invention provides a multi-task collaborative data processing method based on cloud computing, comprising:

[0005] Obtaining an initial task, and decomposing the initial task to obtain a plurality of subtasks and task relationships between the subtasks;

[0006] Clustering the subtasks according to the task relationships to obtain multiple target tasks;

[0007] Determine a target optimization model according to the plurality of target tasks, and determine a target load and a target execution order corresponding to the target tasks according to the target optimization model;

[0008] The target data corresponding to the target task is processed according to the target load and the target execution order to obtain a target processing result corresponding to the target task.

[0009] In a second aspect, an embodiment of the present invention provides a multi-task collaborative data processing system based on cloud computing, comprising:

[0010] A task decomposition module, used to obtain an initial task, and decompose the initial task to obtain a plurality of subtasks and task relationships between the subtasks;

[0011] A task clustering module, used for clustering the subtasks according to the task relationships to obtain multiple target tasks;

[0012] A task optimization module, used to determine a target optimization model according to the plurality of target tasks, and determine a target load and a target execution order corresponding to the target tasks according to the target optimization model;

[0013] The task execution module is used to process the target data corresponding to the target task according to the target load and the target execution order to obtain the target processing result corresponding to the target task.

[0014] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any one of the multi-task collaborative data processing methods based on cloud computing provided in the specification of the present invention are implemented.

[0015] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of the multi-task collaborative data processing methods based on cloud computing provided in the specification of the present invention.

[0016] The embodiment of the present invention provides a multi-task collaborative data processing method and system based on cloud computing, the method comprising: decomposing an initial task into multiple subtasks, so that the complex initial task becomes easier to manage and execute. Each subtask has a clear goal and specific requirements, which helps to refine the work arrangement and facilitates tracking and monitoring the progress of the task. Through this method, a large and complex task can be decomposed into smaller and easier to control parts, so that the execution of the overall task becomes more efficient and controllable. Further, by analyzing the task relationship between each subtask and clustering it, the related subtasks can be combined into multiple target tasks. This clustering process helps to reduce interference between subtasks and enhance the coordination between tasks, thereby improving overall work efficiency. This process ensures that related tasks can be carried out under optimal conditions, thereby avoiding unnecessary conflicts between tasks and improving the organization and execution effect of tasks. A target optimization model is established based on multiple target tasks. Through the target optimization model, the optimal target load and target execution order can be determined for each target task. This optimization process not only helps to effectively allocate resources, but also avoids resource waste and potential bottleneck problems, and significantly improves the operating efficiency of the system. The application of the optimization model ensures the rational use of resources, so that the execution process of the task is smoother and more efficient. Finally, by processing the target data according to the optimized target load and target execution order, the task can be completed more efficiently and the expected target processing results can be obtained, thereby effectively solving the problems of resource waste or task delay that the existing scheduler may face when managing and allocating subtasks. Through scientific task decomposition, reasonable task clustering, accurate optimization model and efficient data processing, the efficiency of task execution and resource utilization are significantly improved, providing a more efficient and systematic solution for the management of complex tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a multi-task collaborative data processing method based on cloud computing provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of the module structure of a multi-task collaborative data processing system based on cloud computing provided by an embodiment of the present invention;

[0020] Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0023] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0024] The embodiment of the present invention provides a multi-task collaborative data processing method and system based on cloud computing. The multi-task collaborative data processing method based on cloud computing can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The terminal device can be a server or a server cluster.

[0025] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0026] Please refer to Figure 1 , Figure 1 A flowchart of a multi-task collaborative data processing method based on cloud computing is provided in an embodiment of the present invention.

[0027] like Figure 1 As shown, the multi-task collaborative data processing method based on cloud computing includes steps S101 to S104.

[0028] Step S101: obtain an initial task, and decompose the initial task to obtain a plurality of subtasks and task relationships between the subtasks.

[0029] For example, cloud computing platforms provide powerful computing power and storage resources to support large-scale data processing tasks. At the same time, cloud computing platforms also provide a variety of tools and services, such as schedulers, distributed computing architectures, data sharing and collaboration tools, elastic scaling and automation, data transmission and synchronization, real-time monitoring and optimization, security and permission management, etc., which can support the implementation of multi-task collaborative data processing.

[0030] For example, the initial task to be processed in the cloud computing platform is uploaded to the cloud computing platform according to actual needs, thereby obtaining the initial task, and then breaking the initial task into multiple subtasks. When breaking down, the complexity and workload of the task need to be considered, and the task is broken down into smaller, more manageable parts.

[0031] For example, tasks are broken down by function or operational steps to obtain multiple subtasks, and each subtask is described in detail, including its objectives, scope, required resources, and expected results. This helps ensure that each subtask is clear and specific.

[0032] Exemplarily, the interrelationships between subtasks are determined. This includes determining which subtasks need to be started after other subtasks are completed, which subtasks can be carried out in parallel, and which subtasks have dependencies. For example, determine which subtasks are predecessor tasks and which are successor tasks. The predecessor task needs to be completed before the successor task can start. Or identify which subtasks can be carried out in parallel to improve overall efficiency. Thus, the task relationship between subtasks is determined based on task dependencies and priorities.

[0033] In some embodiments, decomposing the initial task to obtain multiple subtasks and task relationships between the subtasks includes: obtaining a first task structure and a first task function corresponding to the initial task, and then decomposing the initial task according to the first task structure and the first task function to obtain multiple first intermediate tasks; predicting the satisfaction of the first intermediate task using a task satisfaction model to obtain a first satisfaction level corresponding to the first intermediate task; decomposing the first intermediate task according to the first satisfaction level to obtain the multiple subtasks; determining the corresponding dependency relationships between the multiple subtasks, and determining the task relationships between the subtasks according to the dependency relationships.

[0034] Exemplarily, analyzing the first task structure of the initial task includes the components of the initial task, the hierarchical relationship and the internal organization thereof. For example, the initial task can be broken down into a plurality of main stages or modules. Determine the first task function that needs to be implemented by the initial task. This involves identifying the specific operations or services required to complete the initial task, for example, after implementing function 1, continue to implement function 2 based on function 1.

[0035] Exemplarily, the initial task is decomposed into a plurality of first intermediate tasks according to the first task structure and the first task function of the initial task.

[0036] Exemplarily, a suitable task satisfaction model (such as a regression model based on historical data, a machine learning model, etc.) is used to evaluate the task satisfaction. Thus, the task satisfaction model is applied to predict the satisfaction of each first intermediate task. The input of the task satisfaction model includes but is not limited to task characteristics, requirements, and expected results, etc., so as to output the first satisfaction corresponding to the first intermediate task. The first satisfaction can be a value between 1 and 10.

[0037] Exemplarily, when the first satisfaction level corresponding to each first intermediate task does not meet the preset satisfaction level, the initial task is re-decomposed according to the first task structure and the first task function, so as to obtain multiple first intermediate tasks again, and then the task satisfaction model is reused to predict the satisfaction level of the multiple first intermediate tasks again, so as to obtain multiple new first satisfaction levels, until the first satisfaction levels of the multiple first intermediate tasks obtained all meet the preset satisfaction levels, then the obtained first intermediate tasks are determined as the multiple subtasks obtained by decomposing the initial task.

[0038] For example, analyze the dependencies between the subtasks. Determine which subtasks need to be completed before they can start, and which can be done in parallel. Create a task relationship diagram or network diagram to show the dependencies and execution order between subtasks. This helps visualize the execution flow of tasks and coordinate work. Then determine the task relationships between subtasks based on the dependencies.

[0039] Specifically, by decomposing the initial tasks, predicting task satisfaction, optimizing task decomposition, and clarifying task relationships, the scientificity and efficiency of task management are effectively improved, potential problems are reduced, and the coordination and flexibility of task execution are enhanced, thereby providing support for subsequent improvements in resource utilization and task execution efficiency.

[0040] In some embodiments, decomposing the first intermediate task according to the first satisfaction degree to obtain the multiple subtasks includes: if the first satisfaction degree is less than a preset satisfaction degree, obtaining a second task structure and a second task function corresponding to the first intermediate task, and then decomposing the first intermediate task according to the second task structure and the second task function to obtain a plurality of second intermediate tasks; predicting the satisfaction degree of the second intermediate task using the task satisfaction model to obtain a second satisfaction degree corresponding to the second intermediate task; continuing to decompose the second intermediate task according to the second satisfaction degree until the second satisfaction degree corresponding to the second intermediate task is greater than or equal to the preset satisfaction degree; if the first satisfaction degree is greater than or equal to the preset satisfaction degree, obtaining the multiple subtasks according to the first intermediate task; if the second satisfaction degree is greater than or equal to the preset satisfaction degree, obtaining the multiple subtasks according to the first intermediate task and the second intermediate task.

[0041] Exemplarily, a preset satisfaction level is set based on historical experience or expert experience, and when the first satisfaction level is less than the preset satisfaction level, the second task structure and second task function corresponding to the first intermediate task are obtained, and then the first intermediate task is further decomposed according to the second task structure and the second task function to obtain multiple second intermediate tasks.

[0042] Exemplarily, the task satisfaction model is used to predict the satisfaction of the second intermediate task, and the second satisfaction corresponding to the second intermediate task is obtained. The second satisfaction is then compared with the preset satisfaction. When the second satisfaction is less than the preset satisfaction, the second intermediate task continues to be decomposed until the satisfaction of all decomposed intermediate tasks is greater than or equal to the preset satisfaction.

[0043] Exemplarily, if the first satisfaction level is greater than or equal to the preset satisfaction level, the first intermediate task obtained by decomposition is determined as a plurality of subtasks. If the second satisfaction level is greater than or equal to the preset satisfaction level, the first intermediate task and the second intermediate task obtained by decomposition whose satisfaction level is greater than or equal to the preset satisfaction level are determined as a plurality of subtasks. In other words, the plurality of subtasks are tasks whose satisfaction level is greater than or equal to the preset satisfaction level after the initial task is decomposed. That is, the satisfaction level corresponding to each of the plurality of subtasks is greater than or equal to the preset satisfaction level.

[0044] Specifically, through dynamic task decomposition and optimization based on satisfaction, high-quality completion of subtasks is ensured, resource allocation is optimized, the predictability and flexibility of subtasks are improved, and the coordination and systematicness of subtasks are enhanced. This not only improves the scientific nature of task management, but also effectively reduces potential problems and improves overall work efficiency.

[0045] Step S102: clustering the subtasks according to the task relationships to obtain a plurality of target tasks.

[0046] Exemplarily, task relationships include dependency relationships and parallel relationships, where dependency relationships are used to determine which subtasks must be completed before other subtasks, and there are predecessor and successor dependency relationships between these tasks; parallel relationships are subtasks that can be performed simultaneously, and there is no direct dependency relationship between these tasks, and they are independent of each other.

[0047] Exemplarily, two subtasks with mutual dependence are respectively determined as the first center and the second center, and then the similarity between the subtasks in the parallel relationship and the first center and the second center is calculated using an appropriate similarity measurement method (such as cosine similarity, Euclidean distance, Jaccard coefficient, etc.). For each subtask in the parallel relationship, the similarity value between it and the first center and the second center is calculated. According to the calculation results, the subtasks in the parallel relationship are classified into clusters with higher similarity. Each subtask should be assigned to the cluster corresponding to the center point with the highest similarity. According to the similarity classification results, multiple clusters are formed. Each cluster contains subtasks with a high similarity to the center point. Each cluster is defined as a target task. Each target task includes a group of subtasks with similar functions and independent of each other, and subtasks with high similarity are aggregated together. Different target tasks have dependencies, and different target tasks need to be executed in sequence according to the dependencies.

[0048] In some embodiments, clustering the subtasks according to the task relationship to obtain multiple target tasks includes: obtaining task information corresponding to the subtasks, and establishing a task execution graph according to the task relationship between the subtasks; dividing the subtasks into execution orders according to the task execution graph to obtain a corresponding task set under the same execution order; calculating the functional similarity and execution mode similarity between any two related tasks in the task set according to the task information; determining the target similarity between two related tasks according to the functional similarity and the execution mode similarity; and clustering the task set according to the target similarity to obtain multiple target tasks.

[0049] Exemplarily, task information of each subtask is obtained, and the task information includes but is not limited to task name, task description, input and output, etc. The task relationship between subtasks is determined, especially dependency relationships (such as predecessor tasks and successor tasks) and parallel relationships (such as tasks that can be executed simultaneously). A task execution graph is created based on the task relationships between subtasks. In the task execution graph, each subtask is used as a node, and the task relationships (such as dependency relationships) between subtasks are used as edges to connect the nodes. The edges in the task execution graph represent the dependency order and the possibility of parallel execution, which helps to determine the execution order of tasks.

[0050] Exemplarily, the execution order of the subtasks is determined by using the dependency and parallel relationship in the task execution graph. Methods such as topological sorting can be used to ensure that all dependencies are satisfied. According to the determined execution order, subtasks with the same execution order are grouped into task sets. Each task set represents a group of subtasks with the same execution order.

[0051] Exemplarily, the functional similarity and execution mode similarity between any two related tasks in the task set are calculated based on the task information; the execution mode information of the related tasks, such as execution time, resource consumption, operation frequency, etc., is obtained from the task information. The task description of the related task is obtained from the task information, and then the task descriptions of any two related tasks in the task set are compared, and the functional similarity between any two related tasks is calculated using an appropriate similarity measurement method (such as cosine similarity, Jaccard coefficient, etc.). And the execution modes of any two related tasks are compared, and the similarity in execution time, resource consumption, etc. is calculated to obtain the execution mode similarity between any two related tasks.

[0052] Exemplarily, the functional similarity and execution mode similarity are combined by weighted averaging to calculate the target similarity between any two related tasks, and then a similarity threshold is set to determine whether any two related tasks belong to the same target task, so that related tasks with high similarity are clustered together to form multiple target tasks.

[0053] Specifically, by establishing a task execution graph, the dependencies and parallel relationships between subtasks can be clearly identified, making task management more intuitive and efficient. This graphical approach helps to clarify critical paths and potential bottlenecks, ensuring that tasks are executed in the correct order, thereby reducing delays and conflicts caused by improper task order and improving overall execution efficiency. By calculating the functional similarity between related tasks, functionally similar tasks can be identified and aggregated together to optimize task allocation and resource allocation. This approach helps to improve the efficiency and consistency of task processing. By analyzing the execution mode of tasks, tasks with similar execution methods can be found, which helps to optimize task scheduling, reduce duplication of work, and improve resource utilization. Combining functional similarity and execution mode similarity, the target similarity between tasks can be accurately determined, so that resources can be effectively allocated and managed through task clustering, ensuring that resources are reasonably allocated between related tasks and avoiding resource waste and conflicts.

[0054] Step S103: determining a target optimization model according to the plurality of target tasks, and determining a target load and a target execution order corresponding to the target tasks according to the target optimization model.

[0055] Exemplarily, the optimization goal of the target optimization model is determined, such as minimizing the total execution time, maximizing resource utilization, balancing the load, etc., so as to select appropriate evaluation indicators to measure the achievement of the optimization goal, such as task completion time, resource consumption, etc.

[0056] Exemplarily, detailed information of each target task is obtained, including the execution time and resource requirements of the task. Constraints in the environment corresponding to the execution of cloud computing are collected, such as available resources, system load, network bandwidth, etc. A suitable modeling method is selected according to the optimization goal. For example, linear programming, integer programming, heuristic algorithms, etc. can be used. The optimization goal and constraints are converted into a mathematical model, including the definition of the objective function and constraints, so as to obtain the target optimization model.

[0057] Exemplarily, based on the target optimization model, the load of each target task is calculated. The load may include computing requirements, memory usage, data transmission volume, etc. Thus, the results of the optimization model are used to determine how to distribute the task load to achieve balanced distribution of the load, thereby obtaining the target load corresponding to the target task. For example, computing-intensive tasks are assigned to nodes with strong computing capabilities, and storage-intensive tasks are assigned to nodes with strong storage capabilities.

[0058] Exemplarily, the execution order of target tasks is determined based on the dependencies of target tasks. Ensure that all predecessor tasks are completed before subsequent tasks to avoid execution conflicts. Use optimization models to develop a target execution order for target tasks. For example, high priority tasks are executed first, or resource-intensive tasks are executed when the system load is low. Tasks are scheduled into specific execution time periods or resource nodes based on the calculated target execution order. Use scheduling algorithms to optimize the actual execution order of tasks to ensure that tasks proceed as planned.

[0059] Step S104: Process the target data corresponding to the target task according to the target load and the target execution order to obtain a target processing result corresponding to the target task.

[0060] Exemplarily, according to the determined execution order, the target tasks are executed one by one, and a corresponding target load is assigned to each target task. It is ensured that each target task is completed in the set order and complies with the dependency relationship between the target tasks, so that the target data is processed by the target task under the target load, including operations such as calculation, analysis and conversion, so as to obtain the target processing result corresponding to the target task.

[0061] In some embodiments, the target task includes a first task and a second task, the target load includes a first load corresponding to the first task and a second load corresponding to the second task, the target processing result includes a first processing result corresponding to the first task and a second processing result corresponding to the second task, and the processing of the target data corresponding to the target task according to the target load and the target execution order to obtain the target processing result corresponding to the target task includes: determining a first message queue corresponding to the process of the first load executing the first task and determining a second message queue corresponding to the process of the second load executing the second task; exchanging data through the first message queue and the second message queue to obtain first related data required for the second task in the process of executing the first task and obtaining second related data required for the first task in the process of executing the second task; executing the first task according to the first related data to obtain the first processing result corresponding to the first task; executing the second task according to the second related data to obtain the second processing result corresponding to the second task.

[0062] Exemplarily, the target task includes at least a first task and a second task, the target load includes at least a first load corresponding to the first task and a second load corresponding to the second task, and the target processing result includes at least a first processing result corresponding to the first task and a second processing result corresponding to the second task.

[0063] Exemplarily, a first message queue corresponding to a first load executing a first task and a second message queue corresponding to a second load executing a second task are determined, wherein the first message queue is used to process data transmission required by the first load during the execution of the first task, and the second message queue is used to process data transmission required by the second load during the execution of the second task.

[0064] Exemplarily, the first message queue and the second message queue are configured to ensure that they can send and receive data normally. When executing the first task, the required first relevant data of the second task is sent to the first message queue, and then the demand for the first relevant data is sent to the second message queue through the first message queue, so that after the first relevant data is obtained in the second task, the first relevant data is sent to the first task through the second message queue, and then the first task is executed according to the first relevant data to obtain the first processing result corresponding to the first task.

[0065] Exemplarily, when executing the second task, the required second related data of the first task is sent to the second message queue, and then the demand for the second related data is sent to the first message queue through the second message queue. After obtaining the second related data in the first task, the second related data is sent to the second task through the first message queue, and then the second task is executed according to the second related data to obtain the second processing result corresponding to the second task.

[0066] Specifically, the message queue effectively manages the data dependencies and execution order between tasks, avoiding errors caused by dependency issues and ensuring that tasks are executed in the expected order. The message queue supports real-time data transmission, allowing task processing to reflect the latest data status and improve the timeliness and relevance of the results.

[0067] In some embodiments, the method further includes: determining a target database, storing the target processing result in the target database, and setting access permissions for the target processing result, obtaining a target access permission corresponding to the target processing result, so that a target user can perform data query in the target database according to the target access permission.

[0068] For example, the type of the target database (e.g., relational database, NoSQL database) is first determined, and then a suitable target database is selected according to the requirements. This may include considering factors such as the database's functionality, compatibility, and scalability.

[0069] For example, determine who needs to access the data (i.e., the target user) and the target user's permission level (such as read-only, read-write permission, etc.). Configure user roles and permissions in the target database. Create corresponding user roles according to the requirements of the database management system, and assign required permissions to each role. Associate the target user with the configured role to ensure that the target user can access the data in the target database according to the assigned permissions. Thus, a query interface or tool is provided for the target user to help the target user access and query the data in the target database according to the permissions.

[0070] For example, use the cloud platform's identity and access management services (such as AWS IAM, Google IAM) to control the target tasks and target users' access rights to data and resources to ensure data security. For example, when a target user needs to access a data source, the administrator can assign the target user corresponding target access rights through the IAM service to ensure data security. The above steps are automatically performed on the cloud computing platform without manual intervention, thereby achieving efficient and reliable multi-task collaborative data processing.

[0071] In some embodiments, the method also includes: obtaining the task running time and resource utilization corresponding to the target task; determining the task execution status corresponding to the target task based on the task running time and the resource utilization; and determining the exception handling strategy corresponding to the target task based on the task execution status.

[0072] Exemplarily, a monitoring tool is used to track the task running time and resource utilization of the target task. Define the standard and threshold of the target task execution status. For example, set a normal range of task running time and a normal range of resource utilization. Compare the actually collected task running time and resource utilization data with the set standards and thresholds to analyze whether the task is within the normal range. Determine the task execution status based on the analysis results. The target task may be in a normal, warning, or abnormal state. For example, a task running time that exceeds the expected range or an abnormally high resource utilization may indicate a problem with the task.

[0073] For example, a corresponding exception handling strategy is formulated according to the task execution status. Different task execution status may require different handling measures. When the task execution status is normal, no special handling is usually required and monitoring can continue; when the task execution status is warning, further inspection or optimization measures are required, such as adjusting task configuration, adding resources, etc.; when the task execution status is abnormal, immediate measures need to be taken, which may include restarting the task, notifying operation and maintenance personnel, and performing troubleshooting.

[0074] Exemplarily, after the exception handling strategy corresponding to the target task is obtained, the determined exception handling strategy is applied to the actual operation.

[0075] For example, use the monitoring services provided by the cloud platform (such as AWS CloudWatch, Google Stackdriver) to monitor task execution, resource usage, and performance indicators in real time to obtain the task running time and resource utilization corresponding to the target task. For example, when the target running time of a target task exceeds the scheduled time, the monitoring service will issue an alarm to remind the administrator to handle the exception.

[0076] Specifically, the task running time and resource utilization of the target task are obtained, the task execution status of the target task is accurately determined, and appropriate exception handling strategies are formulated and implemented according to the task execution status, thereby ensuring the stability of the task and the normal operation of the system.

[0077] The embodiment of the present invention provides a multi-task collaborative data processing method and system based on cloud computing, the method comprising: decomposing an initial task into multiple subtasks, so that the complex initial task becomes easier to manage and execute. Each subtask has a clear goal and specific requirements, which helps to refine the work arrangement and facilitates tracking and monitoring the progress of the task. Through this method, a large and complex task can be decomposed into smaller and easier to control parts, so that the execution of the overall task becomes more efficient and controllable. Further, by analyzing the task relationship between each subtask and clustering it, the related subtasks can be combined into multiple target tasks. This clustering process helps to reduce interference between subtasks and enhance the coordination between tasks, thereby improving overall work efficiency. This process ensures that related tasks can be carried out under optimal conditions, thereby avoiding unnecessary conflicts between tasks and improving the organization and execution effect of tasks. A target optimization model is established based on multiple target tasks. Through the target optimization model, the optimal target load and target execution order can be determined for each target task. This optimization process not only helps to effectively allocate resources, but also avoids resource waste and potential bottleneck problems, and significantly improves the operating efficiency of the system. The application of the optimization model ensures the rational use of resources, so that the execution process of the task is smoother and more efficient. Finally, by processing the target data according to the optimized target load and target execution order, the task can be completed more efficiently and the expected target processing results can be obtained, thereby effectively solving the problems of resource waste or task delay that the existing scheduler may face when managing and allocating subtasks. Through scientific task decomposition, reasonable task clustering, accurate optimization model and efficient data processing, the efficiency of task execution and resource utilization are significantly improved, providing a more efficient and systematic solution for the management of complex tasks.

[0078] See also Figure 2 , Figure 2A multi-task collaborative data processing system 200 based on cloud computing is provided for an embodiment of the present application. The multi-task collaborative data processing system 200 based on cloud computing includes a task decomposition module 201, a task clustering module 202, a task optimization module 203, and a task execution module 204, wherein the task decomposition module 201 is used to obtain an initial task, and decompose the initial task to obtain multiple subtasks and task relationships between the subtasks; the task clustering module 202 is used to cluster the subtasks according to the task relationships to obtain multiple target tasks; the task optimization module 203 is used to determine a target optimization model according to the multiple target tasks, and determine the target load and target execution order corresponding to the target task according to the target optimization model; the task execution module 204 is used to process the target data corresponding to the target task according to the target load and the target execution order to obtain the target processing result corresponding to the target task.

[0079] In some implementations, the task decomposition module 201, during the process of decomposing the initial task to obtain a plurality of subtasks and the task relationships between the subtasks, performs:

[0080] Obtaining a first task structure and a first task function corresponding to the initial task, and then decomposing the initial task according to the first task structure and the first task function to obtain a plurality of first intermediate tasks;

[0081] Using a task satisfaction model to predict satisfaction with the first intermediate task, to obtain a first satisfaction with the first intermediate task;

[0082] Decomposing the first intermediate task according to the first satisfaction level to obtain a plurality of subtasks;

[0083] Determine the corresponding dependency relationships between the plurality of subtasks, and determine the task relationship between the subtasks according to the dependency relationships.

[0084] In some implementations, during the process of decomposing the first intermediate task according to the first satisfaction level to obtain the plurality of subtasks, the task decomposition module 201 performs:

[0085] If the first satisfaction level is less than a preset satisfaction level, a second task structure and a second task function corresponding to the first intermediate task are obtained, and then the first intermediate task is decomposed according to the second task structure and the second task function to obtain a plurality of second intermediate tasks;

[0086] Using the task satisfaction model to predict the satisfaction of the second intermediate task, and obtaining a second satisfaction level corresponding to the second intermediate task;

[0087] Continue to decompose the second intermediate task according to the second satisfaction level until the second satisfaction level corresponding to the second intermediate task is greater than or equal to the preset satisfaction level;

[0088] If the first satisfaction level is greater than or equal to the preset satisfaction level, obtaining a plurality of the subtasks according to the first intermediate task;

[0089] If the second satisfaction level is greater than or equal to the preset satisfaction level, a plurality of subtasks are obtained according to the first intermediate task and the second intermediate task.

[0090] In some implementations, the task clustering module 202 performs, during the process of clustering the subtasks according to the task relationships to obtain a plurality of target tasks:

[0091] Obtaining task information corresponding to the subtasks, and establishing a task execution graph according to the task relationship between the subtasks;

[0092] Divide the subtasks into execution order according to the task execution graph to obtain a corresponding task set under the same execution order;

[0093] Calculating the functional similarity and execution mode similarity between any two related tasks in the task set according to the task information;

[0094] Determining the target similarity between the two related tasks according to the functional similarity and the execution mode similarity;

[0095] The task set is clustered according to the target similarity to obtain a plurality of target tasks.

[0096] In some implementations, the target task includes a first task and a second task, the target load includes a first load corresponding to the first task and a second load corresponding to the second task, and the target processing result includes a first processing result corresponding to the first task and a second processing result corresponding to the second task. In the process of processing the target data corresponding to the target task according to the target load and the target execution order to obtain the target processing result corresponding to the target task, the task execution module 204 executes:

[0097] Determine a first message queue corresponding to a process in which the first load executes the first task, and determine a second message queue corresponding to a process in which the second load executes the second task;

[0098] Exchanging data through the first message queue and the second message queue to obtain first relevant data of the second task required in the process of executing the first task and to obtain second relevant data of the first task required in the process of executing the second task;

[0099] Execute the first task according to the first relevant data to obtain a first processing result corresponding to the first task;

[0100] The second task is executed according to the second related data to obtain a second processing result corresponding to the second task.

[0101] In some implementations, the cloud computing-based multi-task collaborative data processing system 200 further performs:

[0102] Determine a target database, store the target processing result in the target database, set access permissions for the target processing result, and obtain target access permissions corresponding to the target processing result, so that a target user can query data in the target database according to the target access permissions.

[0103] In some implementations, the cloud computing-based multi-task collaborative data processing system 200 further performs:

[0104] Obtaining the task running time and resource utilization rate corresponding to the target task;

[0105] Determine the task execution status corresponding to the target task according to the task running time and the resource utilization rate;

[0106] An exception handling strategy corresponding to the target task is determined according to the task execution status.

[0107] In some implementations, the cloud computing-based multi-task collaborative data processing system 200 may be applied to a terminal device.

[0108] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the multi-task collaborative data processing system 200 based on cloud computing described above can refer to the corresponding process in the aforementioned multi-task collaborative data processing method embodiment based on cloud computing, and will not be repeated here.

[0109] See also Figure 3 , Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.

[0110] like Figure 3As shown, the terminal device 300 includes a processor 301 and a memory 302, and the processor 301 and the memory 302 are connected via a bus 303, such as an I2C (Inter-integrated Circuit) bus.

[0111] Specifically, the processor 301 is used to provide computing and control capabilities to support the operation of the entire terminal device. The processor 301 can be a central processing unit (CPU), and the processor 301 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0112] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0113] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the terminal device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] The processor is used to run the computer program stored in the memory, and implement any one of the multi-task collaborative data processing methods based on cloud computing provided by the embodiments of the present invention when executing the computer program.

[0115] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:

[0116] Obtaining an initial task, and decomposing the initial task to obtain a plurality of subtasks and task relationships between the subtasks;

[0117] Clustering the subtasks according to the task relationships to obtain multiple target tasks;

[0118] Determine a target optimization model according to the plurality of target tasks, and determine a target load and a target execution order corresponding to the target tasks according to the target optimization model;

[0119] The target data corresponding to the target task is processed according to the target load and the target execution order to obtain a target processing result corresponding to the target task.

[0120] In some implementations, the processor 301, during the process of decomposing the initial task to obtain a plurality of subtasks and the task relationships between the subtasks, executes:

[0121] Obtaining a first task structure and a first task function corresponding to the initial task, and then decomposing the initial task according to the first task structure and the first task function to obtain a plurality of first intermediate tasks;

[0122] Using a task satisfaction model to predict satisfaction with the first intermediate task, to obtain a first satisfaction with the first intermediate task;

[0123] Decomposing the first intermediate task according to the first satisfaction level to obtain a plurality of subtasks;

[0124] Determine the corresponding dependency relationships between the plurality of subtasks, and determine the task relationship between the subtasks according to the dependency relationships.

[0125] In some implementations, during the process of decomposing the first intermediate task according to the first satisfaction level to obtain the plurality of subtasks, the processor 301 executes:

[0126] If the first satisfaction level is less than a preset satisfaction level, a second task structure and a second task function corresponding to the first intermediate task are obtained, and then the first intermediate task is decomposed according to the second task structure and the second task function to obtain a plurality of second intermediate tasks;

[0127] Using the task satisfaction model to predict the satisfaction of the second intermediate task, and obtaining a second satisfaction level corresponding to the second intermediate task;

[0128] Continue to decompose the second intermediate task according to the second satisfaction level until the second satisfaction level corresponding to the second intermediate task is greater than or equal to the preset satisfaction level;

[0129] If the first satisfaction level is greater than or equal to the preset satisfaction level, obtaining a plurality of the subtasks according to the first intermediate task;

[0130] If the second satisfaction level is greater than or equal to the preset satisfaction level, a plurality of subtasks are obtained according to the first intermediate task and the second intermediate task.

[0131] In some implementations, during the process of clustering the subtasks according to the task relationships to obtain a plurality of target tasks, the processor 301 executes:

[0132] Obtaining task information corresponding to the subtasks, and establishing a task execution graph according to the task relationship between the subtasks;

[0133] Divide the subtasks into execution order according to the task execution graph to obtain a corresponding task set under the same execution order;

[0134] Calculating the functional similarity and execution mode similarity between any two related tasks in the task set according to the task information;

[0135] Determining the target similarity between the two related tasks according to the functional similarity and the execution mode similarity;

[0136] The task set is clustered according to the target similarity to obtain a plurality of target tasks.

[0137] In some implementations, the target task includes a first task and a second task, the target load includes a first load corresponding to the first task and a second load corresponding to the second task, and the target processing result includes a first processing result corresponding to the first task and a second processing result corresponding to the second task. In the process of processing the target data corresponding to the target task according to the target load and the target execution order to obtain the target processing result corresponding to the target task, the processor 301 executes:

[0138] Determine a first message queue corresponding to a process in which the first load executes the first task, and determine a second message queue corresponding to a process in which the second load executes the second task;

[0139] Exchanging data through the first message queue and the second message queue to obtain first relevant data of the second task required in the process of executing the first task and to obtain second relevant data of the first task required in the process of executing the second task;

[0140] Execute the first task according to the first relevant data to obtain a first processing result corresponding to the first task;

[0141] The second task is executed according to the second related data to obtain a second processing result corresponding to the second task.

[0142] In some implementations, the processor 301 further executes:

[0143] Determine a target database, store the target processing result in the target database, set access permissions for the target processing result, and obtain target access permissions corresponding to the target processing result, so that a target user can query data in the target database according to the target access permissions.

[0144] In some implementations, the processor 301 further executes:

[0145] Obtaining the task running time and resource utilization rate corresponding to the target task;

[0146] Determine the task execution status corresponding to the target task according to the task running time and the resource utilization rate;

[0147] An exception handling strategy corresponding to the target task is determined according to the task execution status.

[0148] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned cloud computing-based multi-task collaborative data processing method embodiment, and will not be repeated here.

[0149] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any multi-task collaborative data processing method based on cloud computing provided in the description of the embodiment of the present invention.

[0150] The storage medium may be an internal storage unit of the terminal device in the aforementioned embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device.

[0151] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0152] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0153] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-task collaborative data processing method based on cloud computing, characterized in that: The method comprises: Obtaining an initial task, and decomposing the initial task to obtain a plurality of subtasks and task relationships between the subtasks; Clustering the subtasks according to the task relationships to obtain multiple target tasks; Determine a target optimization model according to the plurality of target tasks, and determine a target load and a target execution order corresponding to the target tasks according to the target optimization model; The target data corresponding to the target task is processed according to the target load and the target execution order to obtain a target processing result corresponding to the target task.

2. The method according to claim 1, characterized in that The step of decomposing the initial task to obtain a plurality of subtasks and the task relationships between the subtasks includes: Obtaining a first task structure and a first task function corresponding to the initial task, and then decomposing the initial task according to the first task structure and the first task function to obtain a plurality of first intermediate tasks; Using a task satisfaction model to predict satisfaction with the first intermediate task, to obtain a first satisfaction with the first intermediate task; Decomposing the first intermediate task according to the first satisfaction level to obtain a plurality of subtasks; Determine the corresponding dependency relationships between the plurality of subtasks, and determine the task relationship between the subtasks according to the dependency relationships.

3. The method according to claim 2, characterized in that Decomposing the first intermediate task according to the first satisfaction level to obtain a plurality of subtasks includes: If the first satisfaction level is less than a preset satisfaction level, a second task structure and a second task function corresponding to the first intermediate task are obtained, and then the first intermediate task is decomposed according to the second task structure and the second task function to obtain a plurality of second intermediate tasks; Using the task satisfaction model to predict the satisfaction of the second intermediate task, and obtaining a second satisfaction level corresponding to the second intermediate task; Continue to decompose the second intermediate task according to the second satisfaction level until the second satisfaction level corresponding to the second intermediate task is greater than or equal to the preset satisfaction level; If the first satisfaction level is greater than or equal to the preset satisfaction level, obtaining a plurality of the subtasks according to the first intermediate task; If the second satisfaction level is greater than or equal to the preset satisfaction level, a plurality of subtasks are obtained according to the first intermediate task and the second intermediate task.

4. The method according to claim 1, characterized in that: The clustering of the subtasks according to the task relationship to obtain a plurality of target tasks includes: Obtaining task information corresponding to the subtasks, and establishing a task execution graph according to the task relationship between the subtasks; Divide the subtasks into execution order according to the task execution graph to obtain a corresponding task set under the same execution order; Calculating the functional similarity and execution mode similarity between any two related tasks in the task set according to the task information; Determining the target similarity between the two related tasks according to the functional similarity and the execution mode similarity; The task set is clustered according to the target similarity to obtain a plurality of target tasks.

5. The method according to claim 1, characterized in that The target task includes a first task and a second task, the target load includes a first load corresponding to the first task and a second load corresponding to the second task, the target processing result includes a first processing result corresponding to the first task and a second processing result corresponding to the second task, and the processing of the target data corresponding to the target task according to the target load and the target execution order to obtain the target processing result corresponding to the target task includes: Determine a first message queue corresponding to a process in which the first load executes the first task, and determine a second message queue corresponding to a process in which the second load executes the second task; Exchanging data through the first message queue and the second message queue to obtain first relevant data of the second task required in the process of executing the first task and to obtain second relevant data of the first task required in the process of executing the second task; Execute the first task according to the first relevant data to obtain a first processing result corresponding to the first task; The second task is executed according to the second related data to obtain a second processing result corresponding to the second task.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Determine a target database, store the target processing result in the target database, set access permissions for the target processing result, and obtain target access permissions corresponding to the target processing result, so that a target user can query data in the target database according to the target access permissions.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Obtaining the task running time and resource utilization rate corresponding to the target task; Determine the task execution status corresponding to the target task according to the task running time and the resource utilization rate; An exception handling strategy corresponding to the target task is determined according to the task execution status.

8. A multi-task collaborative data processing system based on cloud computing, characterized in that: include: A task decomposition module, used to obtain an initial task, and decompose the initial task to obtain a plurality of subtasks and task relationships between the subtasks; A task clustering module, used for clustering the subtasks according to the task relationships to obtain multiple target tasks; A task optimization module, used to determine a target optimization model according to the plurality of target tasks, and determine a target load and a target execution order corresponding to the target tasks according to the target optimization model; The task execution module is used to process the target data corresponding to the target task according to the target load and the target execution order to obtain the target processing result corresponding to the target task.

9. A terminal device, characterized in that: The terminal device includes a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program and implement the cloud computing-based multi-task collaborative data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer storage medium for computer storage, characterized in that: The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the cloud computing-based multi-task collaborative data processing method described in any one of claims 1 to 7.

Images