Heterogeneous model co-processing system and method based on task dynamic decomposition

Abstract

The invention discloses a heterogeneous model co-processing system and method based on task dynamic decomposition, and relates to the technical field of big data resource services, a commander scheduling module is used for generating a task decomposition strategy according to a task type, a data scale, a priority level, a time limit requirement and a current system resource state, and sending the task decomposition strategy to the commander scheduling module; the heterogeneous model management module is used for maintaining various calculation models, the calculation models comprise a distributed calculation model, a streaming calculation model and a batch processing calculation model, capability evaluation is carried out on the calculation models, processing capability parameters of the calculation models are obtained, and the processing capability parameters of the calculation models are sent to the heterogeneous model management module; the system has the advantages that unified task scheduling and management are achieved through the commander scheduling module, task decomposition strategies are generated according to task characteristics and system resource states, decomposition commands are issued, centralized scheduling control is achieved, and coordination and efficiency of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of big data resource service technology, specifically to a heterogeneous model collaborative processing system and method based on dynamic task decomposition. Background Technology

[0002] With the rapid development of big data technology, big data resource services are widely used in various industries. In practical applications, big data processing tasks are often characterized by large data scale, complex processing logic, and high real-time requirements. In order to meet these needs, the industry usually adopts heterogeneous computing models for collaborative processing, such as distributed computing models, streaming computing models, and batch processing computing models.

[0003] Existing heterogeneous model collaborative processing schemes mainly suffer from the following problems: First, the task decomposition lacks dynamism. Existing solutions typically use predefined static decomposition rules, which cannot be dynamically adjusted according to task characteristics and system resource status, resulting in low resource utilization. Second, there is a lack of a unified scheduling mechanism. In the existing schemes, each computing model often runs independently, lacking a unified scheduling and coordination mechanism, which leads to low task processing efficiency. Third, the model selection is not intelligent enough. When allocating subtasks to the computing model, the existing solutions usually adopt a simple load balancing strategy, without fully considering the matching relationship between task characteristics and model characteristics, which affects the processing effect. Therefore, a new technical solution is needed to achieve dynamic task decomposition, unified scheduling management, and intelligent model selection, thereby improving the efficiency of heterogeneous model collaborative processing. Summary of the Invention

[0004] The purpose of this invention is to provide a heterogeneous model collaborative processing system and method based on task dynamic decomposition.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a heterogeneous model collaborative processing system based on dynamic task decomposition, comprising: The task receiving module is used to receive tasks to be processed and extract the task type, data size, priority level and time limit requirements of the task. The commander scheduling module is used to generate a task decomposition strategy based on the task type, data size, priority level, time limit requirements and current system resource status, and to issue a decomposition command to the task dynamic decomposition module. The task dynamic decomposition module is used to decompose the task to be processed into multiple subtasks according to the decomposition command and the task decomposition strategy. Each subtask includes a subtask identifier, processing content and dependencies. The heterogeneous model management module is used to maintain various computing models, including distributed computing models, streaming computing models, and batch processing computing models, and to evaluate the capabilities of each computing model to obtain the processing capability parameters of each computing model. The collaborative processing module is used to allocate each subtask to the corresponding computing model for collaborative processing according to the processing content of the subtask and the processing capability parameters, and to monitor the processing progress of each subtask. The results aggregation module is used to receive the subtask processing results returned by each computing model, aggregate the subtask processing results according to the dependencies, and generate the final task result.

[0006] Furthermore, the commander dispatch module includes: The resource monitoring unit is used to monitor the resource utilization and available computing power of each computing model in the system in real time. The strategy generation unit is used to calculate the task complexity score based on the task type, data size, resource utilization rate and available computing power, and determine the decomposition granularity and allocation scheme based on the task complexity score to generate the task decomposition strategy. The command issuing unit is used to encapsulate the task decomposition strategy into the decomposition command and issue it to the task dynamic decomposition module.

[0007] Furthermore, the method by which the strategy generation unit calculates the task complexity score is as follows: ; in, Rate the task complexity. For normalized task type weight values, For normalized data size weight values, The priority weight value is the normalization level. The normalization method is to map the original value to the [0,1] interval. The specific mapping relationship is preset in the system configuration.

[0008] Furthermore, the method by which the task dynamic decomposition module decomposes the task to be processed into multiple sub-tasks according to the task decomposition strategy includes: Obtain the decomposition granularity in the task decomposition strategy; The number of subtasks is determined based on the decomposition granularity. The processing content of the task to be processed is divided according to the number of sub-tasks, forming multiple sub-task processing contents; Assign a unique subtask identifier to each subtask; Analyze the sequential relationship and data transfer relationship between the processing contents of each subtask, and establish the aforementioned dependency relationship.

[0009] Furthermore, the method by which the collaborative processing module assigns each subtask to the corresponding computing model for collaborative processing includes: Obtain the processing capability parameters of each computing model, including throughput, response time, and load level; Based on the processing content characteristics of the subtask, the adaptability of each computing model to the subtask is determined. The adaptability is calculated based on the degree of matching between the processing content characteristics of the subtask and the capability characteristics of the computing model, and the value range is [0,1]. Based on the combined adaptability and processing capability parameters, calculate the overall score of each computing model for the sub-task; Subtasks are assigned to the computational model with the highest overall score.

[0010] This invention also provides a method for collaborative processing of heterogeneous models based on dynamic task decomposition, comprising the following steps: S1: Receive the task to be processed, and extract the task type, data size, priority level and time limit requirements of the task; S2: Based on the task type, data size, priority level, time limit requirements, and current system resource status, generate a task decomposition strategy and issue a decomposition command; S3: According to the decomposition command and the task decomposition strategy, the task to be processed is decomposed into multiple subtasks, each subtask containing a subtask identifier, processing content and dependencies; S4: Maintain multiple computing models, including distributed computing models, streaming computing models, and batch processing computing models, and evaluate the capabilities of each computing model to obtain the processing capability parameters of each computing model; S5: Based on the processing content of the subtask and the processing capability parameters, assign each subtask to the corresponding computing model for collaborative processing, and monitor the processing progress of each subtask. S6: Receive the subtask processing results returned by each computing model, summarize the subtask processing results according to the dependency relationship, and generate the final task result.

[0011] Furthermore, the method for generating the task decomposition strategy in step S2 includes: Real-time monitoring of resource utilization and available computing power of each computing model in the system; Calculate the task complexity score based on the task type, data size, resource utilization, and available computing power. The task decomposition strategy is generated by determining the decomposition granularity and allocation scheme based on the task complexity score.

[0012] Further, step S3, which involves decomposing the task to be processed into multiple sub-tasks according to the task decomposition strategy, includes: Obtain the decomposition granularity in the task decomposition strategy; The number of subtasks is determined based on the decomposition granularity (the decomposition granularity represents the amount of data or computational load processed by each subtask, which is determined by the system based on task complexity scoring); The processing content of the task to be processed is divided according to the number of sub-tasks, forming multiple sub-task processing contents; Assign a unique subtask identifier to each subtask; Analyze the sequential relationship and data transfer relationship between the processing contents of each subtask, and establish the aforementioned dependency relationship.

[0013] Furthermore, step S5, which assigns each subtask to its corresponding computational model for collaborative processing, includes: Obtain the processing capability parameters of each computing model, including throughput, response time, and load level; Based on the processing content characteristics of the subtask, the adaptability of each computing model to the subtask is determined. The adaptability is calculated based on the degree of matching between the processing content characteristics of the subtask and the capability characteristics of the computing model, and the value range is [0,1]. Based on the combined adaptability and processing capability parameters, calculate the overall score of each computing model for the sub-task; Subtasks are assigned to the computational model with the highest overall score.

[0014] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows: 1. This invention achieves unified task scheduling and management through the commander scheduling module. It generates task decomposition strategies and issues decomposition commands based on task characteristics and system resource status, thereby realizing centralized scheduling control and improving system coordination and efficiency.

[0015] 2. This invention, through a task dynamic decomposition module, can dynamically decompose a task into multiple subtasks according to a task decomposition strategy and establish dependencies between subtasks. Compared with static decomposition, it can better adapt to different task characteristics and system states, improving the flexibility and rationality of task decomposition.

[0016] 3. This invention maintains multiple computing models and performs capability assessments through a heterogeneous model management module, and intelligently matches them according to the characteristics of sub-tasks and model capabilities through a collaborative processing module, thereby achieving optimized allocation of sub-tasks and computing models, giving full play to the advantages of each computing model and improving processing efficiency.

[0017] 4. By calculating a task complexity score, this invention comprehensively considers factors such as task type, data size, and priority level, which can more accurately assess the complexity of a task, provide a scientific basis for task decomposition and resource allocation, and improve the system's resource utilization rate.

[0018] 5. This invention uses a result aggregation module to summarize the processing results of each subtask according to the dependency relationship, which ensures the correctness and completeness of task processing and guarantees the reliability of the final task result. Attached Figure Description

[0019] Figure 1 A schematic diagram of the structure of a heterogeneous model collaborative processing system based on dynamic task decomposition provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the commander dispatch module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the task dynamic decomposition process provided in an embodiment of the present invention; Figure 4 A flowchart illustrating the heterogeneous model collaborative processing method based on dynamic task decomposition provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the subtask allocation and processing flow provided in an embodiment of the present invention. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not for limiting the present invention. In addition, it should be noted that, for ease of description, only the parts related to the present invention are shown in the drawings, not all the structures.

[0021] Example 1 like Figure 1 As shown, this embodiment provides a heterogeneous model collaborative processing system based on dynamic task decomposition, including: a task receiving module, a commander scheduling module, a task dynamic decomposition module, a heterogeneous model management module, a collaborative processing module, and a result aggregation module.

[0022] The task receiving module receives tasks to be processed and extracts their task type, data size, priority level, and time limit. Specifically, the task receiving module receives tasks submitted by external systems through a standard interface. These tasks can be data analysis tasks, data mining tasks, real-time computing tasks, etc. The task receiving module parses the received tasks and extracts key information such as task type, data size, priority level, and time limit. Among these, the task type identifies the processing characteristics of the task, such as computationally intensive, data-intensive, or real-time; the data size indicates the amount of data the task needs to process; the priority level indicates the importance of the task; and the time limit indicates the time limit for the task to be completed.

[0023] The Commander Scheduling Module generates task decomposition strategies based on task type, data size, priority, time limit, and current system resource status, and issues decomposition commands to the Task Dynamic Decomposition Module. As the core control unit of the system, the Commander Scheduling Module is responsible for unified task scheduling and management. The Commander Scheduling Module receives task information from the Task Receiving Module and obtains the current system resource status, including the resource utilization rate and available computing power of each computing model. Based on this information, the Commander Scheduling Module performs comprehensive analysis and decision-making, generates a decomposition strategy for the task, and issues it to the Task Dynamic Decomposition Module in the form of a decomposition command.

[0024] like Figure 2 As shown, the commander scheduling module includes a resource monitoring unit, a strategy generation unit, and a command issuance unit.

[0025] The resource monitoring unit is used to monitor the resource utilization and available computing power of each computing model in the system in real time. Through interaction with the heterogeneous model management module, the resource monitoring unit obtains the real-time running status of each computing model, including resource utilization information such as CPU utilization, memory utilization, network bandwidth usage, and task queue length, as well as the available computing power of each computing model based on the current status.

[0026] The strategy generation unit calculates the task complexity score based on task type, data size, resource utilization, and available computing power. It then determines the decomposition granularity and allocation scheme based on the task complexity score, generating a task decomposition strategy. Taking into account both task characteristics and system status, the strategy generation unit calculates the task complexity score using the following formula: ; in, The task complexity is scored, with a value ranging from 0 to 1. This is a normalized task type weight value, ranging from 0 to 1. Different task types correspond to different weight values; for example, real-time tasks have a weight value of 0.9, while batch processing tasks have a weight value of 0.5. This is a normalized data size weight value, ranging from 0 to 1. The larger the data size, the larger this value. The normalized priority weight value ranges from 0 to 1, with the value increasing as the priority level increases.

[0027] The command issuing unit is used to encapsulate the task decomposition strategy into a decomposition command and issue it to the task dynamic decomposition module. The command issuing unit encapsulates the task decomposition strategy generated by the strategy generation unit to form a decomposition command containing information such as decomposition granularity, number of subtasks, and allocation scheme, and issues it to the task dynamic decomposition module through the internal communication interface.

[0028] The dynamic task decomposition module is used to decompose a task into multiple subtasks based on decomposition commands and task decomposition strategies. Each subtask contains a subtask identifier, processing content, and dependencies, such as... Figure 3 As shown, the specific workflow of the task dynamic decomposition module is as follows: First, the task dynamic decomposition module receives the decomposition command issued by the commander scheduling module and obtains the decomposition granularity in the task decomposition strategy.

[0029] Then, the number of subtasks is determined based on the decomposition granularity. For example, for a data analysis task, the number of subtasks can be calculated based on the amount of data and the decomposition granularity.

[0030] Next, the processing content of the task to be processed is divided according to the number of subtasks, forming multiple subtask processing contents. For data processing tasks, the data can be divided into multiple data blocks according to certain rules, and each data block corresponds to the processing content of a subtask.

[0031] Next, a unique subtask identifier is assigned to each subtask. The subtask identifier is generated by adding the task ID to the subtask sequence number to ensure global uniqueness.

[0032] Then, the sequential relationship and data transfer relationship between the processing contents of each subtask are analyzed to establish dependencies. For example, if some subtasks can only be executed after other subtasks are completed, then there is a dependency relationship between these subtasks. The task dynamic decomposition module constructs a dependency graph to record the dependencies between each subtask.

[0033] The heterogeneous model management module is used to maintain various computing models, including distributed computing models, streaming computing models, and batch processing computing models. It evaluates the capabilities of each computing model and obtains its processing capability parameters. The heterogeneous model management module manages various computing models deployed in the system, including Hadoop-based distributed computing models, Spark-based streaming computing models, and batch processing computing models based on batch processing engines. The heterogeneous model management module regularly evaluates the capabilities of each computing model by executing benchmark tests to obtain its processing capability parameters, including throughput, response time, and load level. Throughput represents the amount of data that a computing model can process per unit of time, response time represents the average time required for a computing model to complete a task, and load level represents the current load level of the computing model.

[0034] The collaborative processing module is used to allocate each subtask to the corresponding computing model for collaborative processing based on the processing content and processing capacity parameters of the subtask, and to monitor the processing progress of each subtask, such as... Figure 5 As shown, the specific workflow of the collaborative processing module is as follows: First, the collaborative processing module obtains the processing capability parameters of each computing model, including throughput, response time, and load level.

[0035] Then, based on the processing content characteristics of the subtask, the adaptability of each computing model to the subtask is determined. The adaptability is calculated based on the degree of matching between the processing content characteristics of the subtask and the capability characteristics of the computing model, and the value range is [0,1]. The adaptability reflects the degree of matching between the computing model and the characteristics of the subtask. For example, for subtasks that require high throughput, the distributed computing model has a higher adaptability, and for subtasks that require low latency, the streaming computing model has a higher adaptability.

[0036] Next, considering both the adaptability and processing capacity parameters, a comprehensive score for each computational model on the subtask is calculated. The comprehensive score is calculated using a weighted summation method, as shown in the formula: ; in, The score is a comprehensive rating, ranging from 0 to 1. For fit, the value ranges from 0 to 1. To calculate the throughput of the model, This represents the maximum throughput value of all computational models in the system under the current state. To calculate the load level of the model, the value ranges from 0 to 1. The saturation load level is set to 1.

[0037] Then, the subtasks are assigned to the computational model with the highest overall score. The collaborative processing module selects the computational model with the highest overall score and submits the subtasks to that model for processing.

[0038] Meanwhile, the collaborative processing module monitors the processing progress of each subtask in real time, records the start time, current status, completion progress and other information of the subtask, and receives the processing results when the subtask is completed.

[0039] The results aggregation module receives the subtask processing results returned by each computing model, aggregates the subtask processing results according to dependencies, and generates the final task result. The results aggregation module receives the processing results of each subtask from the collaborative processing module, and aggregates the subtask results in the correct order according to the dependency graph established by the task dynamic decomposition module. For subtasks with dependencies, the results aggregation module waits for the preceding subtask to complete before passing its result to the subsequent subtask. When all subtasks are completed, the results aggregation module integrates the processing results of each subtask, generates the final task result, and returns it to the task submitter.

[0040] Example 2 like Figure 4 As shown, this embodiment provides a heterogeneous model collaborative processing method based on dynamic task decomposition, including the following steps: S1: Receive tasks to be processed and extract the task type, data size, priority level and time limit requirements.

[0041] In step S1, the system receives externally submitted tasks through a standard interface. These tasks can be various big data processing tasks, such as data analysis, data mining, and real-time computing. The system parses the received tasks and extracts key attribute information, including task type, data size, priority level, and time limit requirements. This information will be used for subsequent task decomposition and resource scheduling decisions.

[0042] S2: Generate a task decomposition strategy based on the task type, data size, priority level, time limit requirements, and current system resource status, and issue decomposition commands.

[0043] In step S2, the system generates a decomposition strategy for the task based on the extracted task attributes and the current system resource status. Specifically, step S2 includes the following sub-steps: S21: Real-time monitoring of the resource utilization and available computing power of each computing model in the system.

[0044] The system uses a resource monitoring mechanism to obtain real-time information on the resource usage of each computing model, including CPU utilization, memory usage, and network bandwidth, and assesses the available computing capacity of each computing model based on the current status.

[0045] S22: Calculate the task complexity score based on task type, data size, resource utilization, and available computing power.

[0046] The system uses the following formula to calculate the task complexity score.

[0047] The following method is used to calculate each normalized weight value: Weight values ​​for task types The computational complexity is normalized according to the task type; for real-time tasks... The value is set to 0.9 for interactive tasks. The value is set to 0.7 for batch processing tasks. The value is set to 0.5.

[0048] For data size weight value Normalization is performed based on the data size, with a baseline data size set at 100GB. When the task's data size is less than 100GB, it is calculated proportionally. Value, when the data size reaches 100GB or more, The value is set to 1.

[0049] For priority level weight values Normalization is performed based on task priority, with higher priority tasks... The value is set to 1 for medium priority tasks. The value is set to 0.6 for low-priority tasks. The value is set to 0.3.

[0050] S23: Determine the decomposition granularity and allocation scheme based on the task complexity score, generate a task decomposition strategy, and the system will then apply the calculated task complexity score. Determine the granularity of task decomposition, when When the value is between 0.7 and 1, fine-grained decomposition is used, and the number of subtasks is set to 16 to 32. When the value is between 0.4 and 0.7, medium-granularity decomposition is used, and the number of subtasks is set to 8 to 16. When the value is between 0 and 0.4, coarse-grained decomposition is adopted, and the number of subtasks is set to 4 to 8.

[0051] Meanwhile, the system determines a preliminary allocation scheme for subtasks based on the available computing power and task characteristics of each computing model. For computationally intensive subtasks, they are preferentially allocated to distributed computing models; for subtasks with high real-time requirements, they are preferentially allocated to streaming computing models; and for subtasks involving large-scale data processing, they are preferentially allocated to batch processing computing models.

[0052] The generated task decomposition strategy includes information such as decomposition granularity, number of subtasks, and preliminary allocation scheme, and is issued to the task decomposition execution unit in the form of decomposition commands.

[0053] S3: Based on the decomposition command and task decomposition strategy, the task to be processed is decomposed into multiple subtasks. Each subtask contains a subtask identifier, processing content, and dependencies.

[0054] In step S3, the system performs task decomposition operations based on the received decomposition command and task decomposition strategy, such as... Figure 3 As shown, the specific steps include the following: S31: Obtain the decomposition granularity in the task decomposition strategy.

[0055] The system parses the decomposition granularity parameters from the decomposition command.

[0056] S32: Determine the number of subtasks based on the decomposition granularity.

[0057] The system determines the specific number of subtasks based on the granularity of the decomposition and the actual situation of the task.

[0058] S33: Divide the processing content of the task to be processed into multiple sub-tasks based on the number of sub-tasks.

[0059] Based on the characteristics of the task, the system uses an appropriate partitioning method to divide the task processing content into multiple parts. For data processing tasks, the partitioning can be done according to data fragmentation; for calculation tasks, the partitioning can be done according to calculation steps.

[0060] S34: Assign a unique subtask identifier to each subtask.

[0061] The system generates a unique identifier for each subtask, consisting of a task ID and a subtask sequence number, ensuring global uniqueness within the system.

[0062] S35: Analyze the sequential relationship and data transfer relationship between the processing contents of each subtask, and establish dependencies.

[0063] The system analyzes the logical relationships between subtasks, identifies which subtasks have sequential execution constraints, and which subtasks need to transfer data. The system constructs a subtask dependency graph to record the dependencies between subtasks.

[0064] S4: Maintain multiple computing models, including distributed computing models, streaming computing models, and batch processing computing models, and evaluate the capabilities of each computing model to obtain the processing capability parameters of each computing model.

[0065] In step S4, the system manages and maintains multiple heterogeneous computing models, and periodically evaluates the capabilities of each computing model. The evaluation methods include executing standard test tasks, measuring performance indicators such as throughput and response time of each computing model, and evaluating the available computing capabilities of each computing model based on the current load.

[0066] Processing capacity parameters include: Throughput: Represents the amount of data that the computing model can process per unit of time, measured in GB / s.

[0067] Response time: Represents the average time required for the computational model to complete a standard task, measured in seconds.

[0068] Load level: Indicates the current load level of the calculation model, with a value range of 0 to 1, where 0 indicates idle and 1 indicates full load.

[0069] S5: Based on the processing content and processing capacity parameters of the subtasks, assign each subtask to the corresponding computing model for collaborative processing, and monitor the processing progress of each subtask.

[0070] In step S5, the system assigns each decomposed subtask to the appropriate computational model for processing, such as... Figure 5 As shown, the specific steps include: S51: Obtain the processing capability parameters of each computing model, including throughput, response time, and load level.

[0071] The system obtains the processing capability parameters of each computing model from the computing model management unit.

[0072] S52: Based on the processing content characteristics of the subtask, determine the suitability of each computing model for the subtask. The suitability is calculated based on the degree of matching between the processing content characteristics of the subtask and the capability characteristics of the computing model, and the value range is [0,1].

[0073] The system analyzes the processing characteristics of subtasks, including computational complexity, data access patterns, and real-time requirements. Based on the characteristics of each computational model, it evaluates the degree of matching between the computational model and the subtask, obtaining a fitness value. The fitness value ranges from 0 to 1, with a higher value indicating a higher degree of matching.

[0074] For example, for subtasks requiring high throughput processing, the fitness of the distributed computing model is set to 0.9; for subtasks requiring low latency response, the fitness of the streaming computing model is set to 0.95; and for subtasks requiring sequential processing of large amounts of data, the fitness of the batch processing computing model is set to 0.85.

[0075] S53: Calculate the overall score of each computing model for the subtask by combining the overall adaptability and processing capability parameters.

[0076] S54: Assign subtasks to the computational model with the highest overall score.

[0077] The system sorts the comprehensive scores of each computing model, selects the computing model with the highest comprehensive score, and submits the subtask to that computing model for processing.

[0078] S55: Monitor the processing progress of each subtask.

[0079] The system establishes a task monitoring mechanism to track the execution status of each subtask in real time, including start time, current progress, estimated completion time, and other information, and receives the processing results when the subtask is completed.

[0080] S6: Receive the subtask processing results returned by each computing model, summarize the subtask processing results according to the dependencies, and generate the final task result.

[0081] In step S6, the system receives the subtask processing results returned by each computing model. Based on the dependency graph established during task decomposition, the system summarizes the subtask results in the correct order. For subtasks without dependencies, their results can be summarized in parallel. For subtasks with dependencies, the system waits for the preceding subtask to complete before passing its results to the subsequent subtasks for processing.

[0082] Once all subtasks are completed, the system integrates the processing results of each subtask to generate the final result of the task. The integration process adopts different strategies depending on the task type, such as data aggregation, result merging, and statistical aggregation.

[0083] The system formats the final result and returns it to the task submitter, completing the entire task processing flow.

[0084] Through the above steps, this invention realizes a heterogeneous model collaborative processing method based on dynamic task decomposition. It can dynamically adjust the task decomposition strategy according to the task characteristics and system status, intelligently allocate sub-tasks to suitable computing models, realize efficient collaboration of heterogeneous models, and improve the processing efficiency of big data resource services and the utilization rate of system resources.

[0085] Example 3 This embodiment provides a specific application scenario to illustrate the actual application effect of the present invention.

[0086] An internet company needs to perform real-time analysis of massive amounts of user behavior data. The analysis task includes four steps: data cleaning, feature extraction, pattern recognition, and result visualization. The data size is 500GB, the task type is real-time, and the priority level is high.

[0087] The specific process of processing using the system and method of this invention is as follows: 1. The task receiving module receives the analysis task and extracts the task type as real-time, the data size as 500GB, the priority level as high, and the time limit as 30 minutes.

[0088] 2. The Commander's scheduling module detected that three computing models are available in the system: distributed computing model, streaming computing model, and batch processing computing model. The current load levels of each model are 0.5, 0.3, and 0.6, respectively.

[0089] 3. The commander scheduling module calculates the task complexity score: Real-time tasks The value is 0.9; Data size of 500GB The value is 1; High priority corresponds to The value is 1; .

[0090] 4. Due to The value is 0.96, which falls within the range of 0.7 to 1. The system uses fine-grained decomposition to determine the number of subtasks as 24.

[0091] 5. The dynamic task decomposition module breaks down the task into 24 sub-tasks: Subtasks 1-6 are responsible for data cleaning. Based on the total data volume of 500GB and the decomposition granularity, each subtask processes an average of about 83GB of data. Subtasks 7-12 are responsible for feature extraction, which depends on the results of subtasks 1-6; Subtasks 13-18 are responsible for pattern recognition and depend on the results of subtasks 7-12. Subtasks 19-24 are responsible for visualizing the results, which depend on the results of subtasks 13-18.

[0092] 6. The collaborative processing module assigns tasks to each subtask: The data cleaning subtasks (subtasks 1-6) require high throughput and are assigned to the distributed computing model; The feature extraction subtasks (subtasks 7-12) require low latency and are assigned to the streaming computing model; The pattern recognition subtasks (subtasks 13-18) have high computational complexity and are therefore assigned to the distributed computing model. The results visualization subtask (subtasks 19-24) has a small data volume and is assigned to the streaming computing model.

[0093] 7. Each computing model processes its assigned subtasks in parallel, and the collaborative processing module monitors the processing progress in real time.

[0094] 8. The results summary module receives the processing results of each subtask in the order of dependency, and finally completes the processing of the entire task within 28 minutes, meeting the time limit requirement.

[0095] The system and method of this invention reduce the processing time of the task by 40% and improve the system resource utilization rate by 35% compared with the traditional static allocation method, proving the effectiveness of the invention.

[0096] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A heterogeneous model collaborative processing system based on dynamic task decomposition, comprising a collaborative processing system, characterized in that: The collaborative processing system includes: The task receiving module is used to receive tasks to be processed and extract the task type, data size, priority level and time limit requirements of the task. The commander scheduling module is used to generate a task decomposition strategy based on the task type, data size, priority level, time limit requirements and current system resource status, and to issue a decomposition command to the task dynamic decomposition module. The task dynamic decomposition module is used to decompose the task to be processed into multiple subtasks according to the decomposition command and the task decomposition strategy. Each subtask includes a subtask identifier, processing content and dependencies. The heterogeneous model management module is used to maintain various computing models, including distributed computing models, streaming computing models, and batch processing computing models, and to evaluate the capabilities of each computing model to obtain the processing capability parameters of each computing model. The collaborative processing module is used to allocate each subtask to the corresponding computing model for collaborative processing according to the processing content of the subtask and the processing capability parameters, and to monitor the processing progress of each subtask. The results aggregation module is used to receive the subtask processing results returned by each computing model, aggregate the subtask processing results according to the dependencies, and generate the final task result.

2. The heterogeneous model collaborative processing system based on dynamic task decomposition according to claim 1, characterized in that: The commander dispatch module includes: The resource monitoring unit is used to monitor the resource utilization and available computing power of each computing model in the system in real time. The strategy generation unit is used to calculate the task complexity score based on the task type, data size, resource utilization rate and available computing power, and determine the decomposition granularity and allocation scheme based on the task complexity score to generate the task decomposition strategy. The command issuing unit is used to encapsulate the task decomposition strategy into the decomposition command and issue it to the task dynamic decomposition module.

3. The heterogeneous model collaborative processing system based on dynamic task decomposition according to claim 1, characterized in that: The method by which the strategy generation unit calculates the task complexity score is as follows: ; in, Rate the task complexity. For normalized task type weight values, For normalized data size weight values, The priority weight value is the normalization level. The normalization method is to map the original value to the [0,1] interval. The specific mapping relationship is preset in the system configuration.

4. The heterogeneous model collaborative processing system based on dynamic task decomposition according to claim 1, characterized in that: The method by which the task dynamic decomposition module decomposes the task to be processed into multiple sub-tasks according to the task decomposition strategy includes: Obtain the decomposition granularity in the task decomposition strategy; The number of subtasks is determined based on the decomposition granularity. The processing content of the task to be processed is divided according to the number of sub-tasks, forming multiple sub-task processing contents; Assign a unique subtask identifier to each subtask; Analyze the sequential relationship and data transfer relationship between the processing contents of each subtask, and establish the aforementioned dependency relationship.

5. The heterogeneous model collaborative processing system based on dynamic task decomposition according to claim 1, characterized in that: The method by which the collaborative processing module assigns each subtask to the corresponding computing model for collaborative processing includes: Obtain the processing capability parameters of each computing model, including throughput, response time, and load level; Based on the processing content characteristics of the subtask, the adaptability of each computing model to the subtask is determined. The adaptability is calculated based on the degree of matching between the processing content characteristics of the subtask and the capability characteristics of the computing model, and the value range is [0,1]. Based on the combined adaptability and processing capability parameters, calculate the overall score of each computing model for the sub-task; Subtasks are assigned to the computational model with the highest overall score.

6. A heterogeneous model collaborative processing method based on dynamic task decomposition according to any one of claims 1-5, comprising a collaborative processing method, characterized in that: The collaborative processing method includes the following steps: S1: Receive the task to be processed, and extract the task type, data size, priority level and time limit requirements of the task; S2: Based on the task type, data size, priority level, time limit requirements, and current system resource status, generate a task decomposition strategy and issue a decomposition command; S3: According to the decomposition command and the task decomposition strategy, the task to be processed is decomposed into multiple subtasks, each subtask containing a subtask identifier, processing content and dependencies; S4: Maintain multiple computing models, including distributed computing models, streaming computing models, and batch processing computing models, and evaluate the capabilities of each computing model to obtain the processing capability parameters of each computing model; S5: Based on the processing content of the subtask and the processing capability parameters, assign each subtask to the corresponding computing model for collaborative processing, and monitor the processing progress of each subtask. S6: Receive the subtask processing results returned by each computing model, summarize the subtask processing results according to the dependency relationship, and generate the final task result.

7. The heterogeneous model collaborative processing method based on dynamic task decomposition according to claim 6, characterized in that: The method for generating the task decomposition strategy in step S2 includes: Real-time monitoring of resource utilization and available computing power of each computing model in the system; Calculate the task complexity score based on the task type, data size, resource utilization, and available computing power. The task decomposition strategy is generated by determining the decomposition granularity and allocation scheme based on the task complexity score.

8. The heterogeneous model collaborative processing method based on dynamic task decomposition according to claim 6, characterized in that: Step S3, which involves decomposing the task to be processed into multiple sub-tasks according to the task decomposition strategy, includes: Obtain the decomposition granularity in the task decomposition strategy; The number of subtasks is determined based on the decomposition granularity. The processing content of the task to be processed is divided according to the number of sub-tasks, forming multiple sub-task processing contents; Assign a unique subtask identifier to each subtask; Analyze the sequential relationship and data transfer relationship between the processing contents of each subtask, and establish the aforementioned dependency relationship.

9. A heterogeneous model collaborative processing method based on dynamic task decomposition according to claim 6, characterized in that: Step S5, which involves assigning each subtask to a corresponding computational model for collaborative processing, includes: Obtain the processing capability parameters of each computing model, including throughput, response time, and load level; Based on the processing content characteristics of the subtask, the adaptability of each computing model to the subtask is determined. The adaptability is calculated based on the degree of matching between the processing content characteristics of the subtask and the capability characteristics of the computing model, and the value range is [0,1]. Based on the combined adaptability and processing capability parameters, calculate the overall score of each computing model for the sub-task; Subtasks are assigned to the computational model with the highest overall score.

Images