Resource requisition scheduling method for data sharding

Abstract

The present invention relates to a resource requisition scheduling method for data sharding, including: A. determining the maximum possible number of data shards that data can be divided into; B. generating a pre-scheduled resource set; C. determining that each data shard is allocated to a pre-scheduled resource node D. Calculate the final completion time of the task; E. Obtain the completion time and the data fragmentation scale factor; F. Obtain the scale coefficient with the largest value to obtain the actual fragmentation size of the data fragmentation; G. According to the remaining data volume Or the number of remaining shards can be iterated and data sharding is completed; H. According to the pre-segmentation coefficient and the actual and slicing granularity, the final data shard size and corresponding resource scheduling strategy are obtained. The present invention can make full use of the performance of available resource nodes in resource requisition scheduling, adopt a reasonable data fragmentation strategy, and requisition reasonable resource nodes for each data fragmentation to perform data processing, so that the overall task completion time is the shortest, and the task efficiency is effectively improved. execution efficiency.

Description

technical field [0001] The invention relates to a computer data processing method, in particular to a resource requisition and scheduling method for data fragmentation. Background technique [0002] Due to the diversification and resource sharing of the cloud computing system, and its large user base, the number of tasks and data that need to be processed at the same time in the cloud environment is also extremely large, so the resources and tasks in the cloud environment should be rationally and efficiently allocated. Scheduling, ensuring the user's service quality, improving the user's task execution efficiency and resource utilization efficiency, are the key points and difficulties in the research of cloud computing technology. The physical resource nodes in the cloud computing system are heterogeneous, and the status of the nodes will change dynamically with the invocation of the resources. The resources in the system are constantly called and released, and new resources...

AI Technical Summary

Problems solved by technology

Since the existing common method of data sharding is to averagely shard the data to be processed, according to the number of available resources, the data associated with the task is evenly sharded and distributed to each resource node for execution. During execution, due to the performance difference between each resource node, the time for each data fragment to complete data processing will vary greatly, so the overall time for task processing will be longer and the task processing efficiency will be lower.

Images