Task scheduling method and related equipment

Abstract

The invention discloses a task scheduling method and related equipment, which are used for improving the file migration efficiency, reducing the influence of a file migration process on business and guaranteeing the availability of the service of a source end at the same time. The task scheduling method is applied to a master node in a distributed cluster, and comprises the following steps: determining a plurality of slave nodes with normal running states in the distributed cluster; distributing a to-be-migrated file set based on the size of a file in the to-be-migrated file set to obtain a first file subset to be distributed to each slave node; and based on the thread number of each slave node and the first file subset, a migration task is issued to each slave node.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a task scheduling method and related equipment. Background Art

[0002] With the rapid development of information technology, in scenarios such as business system upgrades, switching between old and new systems, and data center relocation, it is often necessary to migrate massive files stored on the source side to the destination side.

[0003] The file migration process will affect the operation of the source service, so it is particularly important to improve the efficiency of file migration to reduce the impact on the source service. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a task scheduling method and related equipment for improving the efficiency of file migration, reducing the impact of the file migration process on the business, and ensuring the availability of the service at the source end.

[0005] In order to achieve the above objectives, the present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a task scheduling method, which is applied to a master node in a distributed cluster, comprising:

[0007] Determine a slave node in the distributed cluster that is in a normal operating state;

[0008] Allocating the set of files to be migrated based on the sizes of the files in the set of files to be migrated to obtain a first subset of files to be allocated to each of the slave nodes;

[0009] Based on the number of threads of each slave node and the first file subset, a migration task is issued to each slave node.

[0010] In a second aspect, an embodiment of the present application provides a task scheduling device, which is applied to a master node in a distributed cluster, including:

[0011] A determination unit, configured to determine a slave node in the distributed cluster whose operating status is normal;

[0012] an allocating unit, configured to allocate the set of files to be migrated based on the sizes of the files in the set of files to be migrated, to obtain a first subset of files to be allocated to each of the slave nodes;

[0013] The sending unit is used to send the migration task to each of the slave nodes based on the number of threads of each of the slave nodes and the first file subset.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0015] processor;

[0016] a memory for storing instructions executable by the processor;

[0017] The processor is configured to execute the instructions to implement the task scheduling method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the task scheduling method described in the first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute part or all of the steps in the task scheduling method described in the first aspect.

[0020] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0021] The master node in the distributed cluster uniformly performs task scheduling, determines multiple slave nodes in the distributed cluster that are in normal operating status, and allocates the file set to be migrated, obtains the first file subset to be allocated to each slave node, and then issues a corresponding migration task to each slave node to migrate the first file subset, so that the migration of the file set to be migrated is completed through multiple slave nodes, the file migration efficiency is improved, and the impact of the file migration process on the service operation of the source end can be reduced.

[0022] In addition, the technical solution proposed in the embodiment of the present application also improves the allocation method of the file set to be migrated. First, the file set to be migrated is allocated based on the size of the files in the file set to be migrated, and the first file subset to be allocated to each slave node is obtained to ensure that the size of the files to be allocated to each slave node is relatively even; then, based on the number of threads of each slave node and the first file subset to be allocated to each slave node, a migration task is issued to each slave node to ensure that the size of the files to be migrated of each thread is relatively small, so that the transmission efficiency of each thread is relatively small, the overall migration time depends on the migration time of the largest file, and the migration efficiency is improved, thereby reducing the impact of the file migration process on the service operation of the source end. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1A schematic diagram of an implementation environment applicable to the data migration method provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of a task scheduling method provided for an embodiment of the present application;

[0026] Figure 3 A flowchart of a task scheduling method provided for another embodiment of the present application;

[0027] Figure 4 A flowchart of a file migration method provided for an embodiment of the present application;

[0028] Figure 5 A structural diagram of a task scheduling device provided for one embodiment of the present application;

[0029] Figure 6 A schematic diagram of the structure of a file migration device provided for one embodiment of the present application;

[0030] Figure 7 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0032] The terms "first", "second", etc. in this specification and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in this specification and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0033] Some concept descriptions:

[0034] Absolute path: The path from the root directory at the top of the tree directory structure to a subdirectory or file.

[0035] Redis: A memory-based, persistent, log-based, key-value database.

[0036] This application studies the file migration process in related technologies and finds that, in the current migration process, a single thread is mainly used for end-to-end serial file transfer, that is, the files in the source end are traversed and transferred according to the file directory of the source end, and the next file is transferred after one file is transferred. However, if the file directory is too deep and the number of files is large, the transmission rate of this method is limited by the processing speed of a single thread, resulting in low migration efficiency, which will have a significant impact on the service operation of the source end.

[0037] In view of this, the embodiment of the present application proposes a distributed file migration solution, which uses a distributed cluster to migrate the massive files to be migrated to the destination. During the migration process, the master node in the distributed cluster uniformly performs task scheduling, and multiple slave nodes in normal operation status respectively create corresponding threads to obtain multiple threads, and then distribute the massive files to multiple threads for migration. Compared with the file transfer solution based on a single-thread serial type, this multi-threaded parallel file transfer solution can improve the file migration efficiency, thereby reducing the impact of the file migration process on the service operation of the source end.

[0038] On this basis, considering that in the multi-threaded parallel transmission mode, the file migration efficiency is mainly affected by the file size transferred by each thread. Specifically, if the massive files are unevenly distributed among multiple threads, such as a thread transferring a large number of large files, the overall migration time depends on the time it takes for the thread to finally complete the migration.

[0039] In view of this, the technical solution proposed in the embodiment of the present application also improves the allocation method of massive files. First, the file set to be migrated is allocated based on the size of the files in the file set to be migrated, and the first file subset to be allocated to each slave node is obtained to ensure that the size of the files to be allocated to each slave node is relatively even; then, based on the number of threads of each slave node and the first file subset to be allocated to each slave node, a migration task is issued to each slave node to ensure that the size of the files to be migrated by each thread is relatively small, so that the transmission efficiency of each thread is relatively small, the overall migration time depends on the migration time of the largest file, and the migration efficiency is improved, thereby reducing the impact of the file migration process on the service operation of the source end.

[0040] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0041] In order to facilitate the understanding of the technical solutions provided by the embodiments of the present application, firstly Figure 1 The implementation environment to which the technical solution provided in the embodiment of the present application is applicable is described. It should be understood that the technical solution provided in the embodiment of the present application is applicable to Figure 1The implementation environment shown is only an exemplary description and should not be understood as a limitation on the implementation environment to which the technical solution is applicable.

[0042] like Figure 1 As shown, an implementation environment applicable to the technical solution provided in the embodiment of the present application includes a source end, a destination end and a distributed cluster.

[0043] The source end can be a device required for business operation, such as an old business system to be upgraded, an original database, etc. The destination end is a device used to synchronize data stored in the source end, such as a new business system after upgrade, a new database, etc.

[0044] A distributed cluster includes multiple nodes, which can be servers with data processing capabilities, such as physical servers or cloud servers that provide cloud computing services. According to different functions, the multiple nodes are divided into master nodes and slave nodes.

[0045] The master node is mainly responsible for monitoring and task scheduling. Monitoring refers to the master node monitoring the task-related information of the source end and the running status of the slave nodes in the distributed cluster. Among them, the task-related information may include the resource usage information of the source end, such as but not limited to the number of processor cores, total resource utilization and transmission resource utilization of the source end. The total resource utilization is used to reflect the total consumption of resources (such as central processing unit, memory, bandwidth, I / O) by the source end during operation. The transmission resource utilization refers to the average amount of resources consumed by a single thread to transfer files. The running status of the slave node indicates whether the slave node is normal. Task scheduling refers to the master node assigning the migration tasks of the file set to be migrated in the source end to multiple slave nodes with normal running status for execution based on the monitored task-related information and the running status of the slave node.

[0046] The slave node is mainly responsible for receiving the migration tasks issued by the master node, establishing multiple threads between the source and destination based on the migration tasks, and using multiple threads to transfer the files indicated by the migration tasks to the destination to achieve data synchronization between the source and destination.

[0047] In actual applications, files can be set on the master node, namely the node information file (also called the host file) and the task file (also called the transfer.conf file). The node information file is used to record the information of all nodes in the distributed cluster, such as but not limited to global information and local information. The global information is shared by all nodes, including but not limited to the communication protocol between nodes, the communication port between nodes, the login account of the node, the login password of the node, etc. The local information is used by each slave node, including but not limited to the identification and operation status of the slave node, etc.

[0048] The task file is used to record the information required to perform the migration task, such as but not limited to the communication protocol between the source and the destination, the communication port, the login user of the slave node, the login password of the slave node, the address of the source, the directory to which the files to be migrated belong in the source, the address of the destination, the storage directory of the files to be migrated in the destination, the verification flag of the file, etc. The verification flag is used to indicate whether to verify the file before migrating it.

[0049] The master node updates the node information file and task file based on the real-time monitoring results, and schedules tasks between slave nodes based on the information recorded in these two files. In order to reduce the pressure on the master node and improve the computing speed of the master node, the master node can store the file set to be migrated in the source end in the memory of the master node before scheduling tasks. If the number of files contained in the file set to be migrated is too large, such as millions or tens of millions, the memory of the master node may be insufficient or occupied too much, resulting in a crash. At this time, the master node can persist the file set to be migrated, such as persisting the file set to be migrated in the key-value database redis.

[0050] After receiving the migration task, the slave node establishes a communication connection with the destination based on the information recorded in the task file, and checks whether there is a corresponding file directory in the destination to store the files migrated as instructed by the migration task; if not, the corresponding file directory is created in the destination. The slave node also establishes a communication connection with the source based on the information recorded in the task file, creates multiple threads, reads the files migrated as instructed by the migration task from the source, and migrates the read files to the corresponding file directory of the destination through multiple threads.

[0051] Based on the above implementation environment, the embodiment of the present application proposes a task scheduling method, which can be executed by a master node in a distributed cluster. In addition, the embodiment of the present application also proposes a file migration method, which can be executed by a slave node in a distributed cluster. The task scheduling method and the file migration method provided by the embodiment of the present application are described in detail below in conjunction with the accompanying drawings.

[0052] See also Figure 2 , is a flow chart of a task scheduling method provided by an embodiment of the present application, the method is applied to a master node in a distributed cluster, and the method comprises the following steps:

[0053] S202, determining a plurality of slave nodes in a distributed cluster whose operating status is normal.

[0054] The master node can determine whether the operating status of each slave node is normal by monitoring the heartbeat signal of each slave node in the distributed cluster.

[0055] S204, allocating the files to be migrated based on the sizes of the files in the file set to be migrated, and obtaining a first file subset to be allocated to each slave node.

[0056] In the embodiment of the present application, a relatively average file size allocated to each slave node may be taken as a goal, and various methods may be used to allocate the file set to be migrated, and the specific method may be selected according to actual needs.

[0057] In one embodiment, in order to ensure that the file size allocated to each slave node is relatively even, the above S204 may include the following steps: determining the file allocation strategy of the file set to be migrated based on the number of files included in the file set to be migrated and the number of multiple slave nodes; allocating the file set to be migrated based on the file allocation strategy and the size of the files in the file set to be migrated, and obtaining the first file subset to be allocated to each slave node.

[0058] As an example, when the number of files in the file set to be migrated is less than or equal to the number of the plurality of slave nodes, the file allocation strategy includes an average allocation strategy. In this case, the manner of allocating the file set to be migrated includes: selecting a first slave node from the plurality of slave nodes, the number of the first slave nodes being equal to the number of files in the file set to be migrated; and based on the number of the first slave nodes, evenly allocating the file set to be migrated to obtain a first file subset to be allocated to each slave node.

[0059] For example, suppose that the file set to be migrated includes file 1 to file 5, a total of 5 files, and the number of slave nodes in the distributed cluster that are in normal operation is 6. First, select 5 slave nodes in normal operation from the distributed cluster as the first slave nodes, and number these first slave nodes to obtain slave nodes 1 to 5. Then, determine that the first file subset to be allocated to slave node 1 includes file 1, determine that the first file subset to be allocated to slave node 2 includes file 2, and so on, until all files in the file set to be migrated are allocated.

[0060] It can be seen that this file allocation method can ensure that each first slave node is allocated files equally, and then use multiple first slave nodes to migrate the allocated files in parallel, which helps to improve the file migration efficiency.

[0061] As another example, when the number of files in the file set to be migrated is greater than the number of the above-mentioned multiple slave nodes, the file allocation strategy includes an average allocation strategy and a remainder allocation strategy. The first file subset includes a first file and a second file. In this case, the manner of allocating the file set to be migrated includes: dividing the file set to be migrated into a second file subset and a third file subset based on the size of the files in the file set to be migrated and the number of multiple slave nodes; allocating the second file subset based on the average allocation strategy to obtain the first file to be allocated to each slave node; performing a remainder operation on the file identifiers of the files in the third file subset to obtain a first remainder operation result, and allocating the third file subset based on the remainder operation result and the remainder allocation strategy to obtain the second file to be allocated to each slave node.

[0062] Specifically, the number of files in the second file subset is equal to the number of the above-mentioned multiple slave nodes. When dividing the file set to be migrated, a random division method can be used to obtain the second file subset and the third file subset. In a more optimal solution, in order to ensure that the size of the files to be allocated to each slave node is relatively even, the file set to be migrated is divided into the second file subset and the third file subset, including: sorting the files in the file set to be migrated according to size to obtain the arrangement sequence number of the files in the file set to be migrated; determining the target arrangement sequence number based on the number of multiple slave nodes; obtaining the files in the file set to be migrated whose arrangement sequence number is less than or equal to the target arrangement sequence number to obtain the second file subset, and obtaining the files in the file set to be migrated whose arrangement sequence number is greater than the target arrangement sequence number to obtain the third file subset.

[0063] More specifically, the target arrangement sequence number is equal to the number of the plurality of slave nodes.

[0064] For example, assuming that the file set to be migrated includes 10 files, the number of slave nodes is 5, and these slave nodes are numbered to obtain slave nodes 1 to 5. First, the files in the file set to be migrated are arranged in order from large to small to obtain the arrangement number of each file. Then, the first 5 files are evenly distributed, that is, the number of the first file to be allocated to each slave node is 1. For example, it is determined that the first file to be allocated to slave node 1 is the file with the arrangement number 1, and the first file to be allocated to slave node 2 is the file with the arrangement number 2, and so on, until the first 5 files are allocated. Then, with 5 as the dividend, the arrangement numbers of the files in the last 5 positions are taken as a remainder operation, and these files are allocated based on the remainder obtained to obtain the second file to be allocated to each slave node. For example, it is determined that the second file to be allocated to slave node 1 includes a file with a remainder of 1, the second file to be allocated to slave node 2 includes a file with a remainder of 2, the second file to be allocated to the slave node includes a file with a remainder of 3, the second file to be allocated to slave node 4 includes a file with a remainder of 4, and the second file to be allocated to slave node 5 includes a file with a remainder of 0. At this point, the last 5 files are allocated; or, it is determined that the second file to be allocated to slave node 5 includes a file with a remainder of 1, the second file to be allocated to slave node 4 includes a file with a remainder of 2, the second file to be allocated to slave node 3 includes a file with a remainder of 3, the second file to be allocated to slave node 2 includes a file with a remainder of 4, and the second file to be allocated to slave node 1 includes a file with a remainder of 0. At this point, the last 5 files are allocated.

[0065] It can be seen that this file allocation method makes the number of files allocated to each slave node relatively evenly, and when there are many large files in the file set to be migrated, it ensures that each slave node can be allocated with large files, thereby improving the parallel speed of large file migration and shortening the overall file migration time. For example, if the file set to be migrated contains 5 large files, if these 5 large files are not evenly distributed to 5 slave nodes, then the overall migration time is the time required for the slave node with the most large files to complete the migration; and through the above file allocation method, these 5 large files can be evenly distributed to 5 slave nodes. When the migration rate of each slave node is equal, the overall migration time is the migration time of the largest file, so the file migration efficiency is significantly improved.

[0066] The embodiment of the present application shows some implementation methods of the above S204. Of course, it should be understood that the above S204 can also be implemented in other ways, and the embodiment of the present application does not limit this.

[0067] S206: Based on the number of threads of each slave node and the first file subset, a migration task is issued to each slave node.

[0068] The files to be migrated are stored in the source end. In one embodiment, the number of threads of each slave node may be a fixed value pre-set in the task file of the master node, and the master node reads the number of threads of each slave node from the task list.

[0069] In another embodiment, it is considered that the number of threads will affect the service operation and file migration efficiency of the source end. For example, if the number of threads is too large, although the number of files migrated in parallel can be increased and the file migration efficiency can be improved, it will also increase the resource utilization of the source end, so that the unused resource rate of the service of the source end is reduced, thereby affecting the service operation of the source end; if the total number of threads is too small, although more resources can be allocated to the service of the source end to ensure the normal operation of the service, it will also reduce the number of files migrated in parallel, thereby reducing the file migration efficiency. In view of this, the task scheduling method provided in the embodiment of the present application also includes a method for determining the number of threads of each slave node, specifically including: determining the total number of threads of the source end based on resource usage information; determining the number of threads of each slave node based on the total number of threads and the number of multiple slave nodes.

[0070] As an example, the resource usage information of the source end includes the number of processor cores of the source end (such as the number of CPIU cores). In this case, the above-mentioned determination of the total number of threads of the source end based on the resource usage information may include: determining the total number of threads of the source end based on the number of processor cores and the number of multiple slave nodes.

[0071] For example, the product of the number of processor cores and the number of slave nodes is determined as the total number of threads on the source side. Assuming that the number of processor cores on the source side is 2 and the distributed cluster contains 5 slave nodes in normal operation, the total number of threads on the source side is 10. In this way, it can be ensured that the file migration task can be executed on each processor core on the source side, and the services running on the source side will not be affected due to too many files being migrated in parallel.

[0072] As another example, the resource usage information of the source end includes the total resource utilization rate and the transmission resource utilization rate of the source end, wherein the total resource utilization rate refers to the total resource consumption rate of the source end during operation, and the transmission resource utilization rate refers to the average amount of resources consumed by a single thread to transmit a file. In this case, in this case, the above-mentioned determination of the total number of threads of the source end based on the resource usage information may include: determining the unused resource rate of the source end based on the total resource utilization rate; determining the total number of threads of the source end based on the unused resource rate and the transmission resource utilization rate.

[0073] For example, the total number of threads on the source side is determined based on the ratio between the unused resource rate and the transmission resource utilization rate. If the total resource utilization rate is 60% and the transmission resource utilization rate is 5%, the unused resource rate is 1-60%=40%, that is, 40% of the resources can be used for file migration tasks, and the total number of threads = 40% / 5% = 8. In this way, the migration capacity of the slave node can be increased while ensuring the service availability of the source side, thereby improving the efficiency of file migration.

[0074] It is worth noting that the above two examples of determining the total number of threads can be used either alone or in combination. For example, initially, the total number of threads on the source side is determined based on the number of processor cores and the number of slave nodes; then, the total number of threads on the source side is dynamically adjusted based on the total resource utilization of the source side and the utilization of migration resources.

[0075] After determining the sum of the total number of threads on the source end, specifically, the number of threads allocated to each slave node can be relatively averaged as a goal, and the number of threads of each slave node can be determined based on the total number of threads and the number of multiple slave nodes.

[0076] For example, if the total number of threads is 10 and the number of slave nodes is 5, the number of threads for each slave node is 2. For another example, if the total number of threads is 9 and the number of slave nodes is 5, the number of threads for 4 slave nodes is 2 and the number of threads for 1 slave node is 1.

[0077] In the above implementation, the total number of threads on the source end is determined based on the resource usage information of the source end, and then the number of threads of each slave node is determined based on the total number of threads and the number of multiple slave nodes. This can better balance the relationship between the service availability of the source end and the file migration efficiency, and improve the file migration efficiency as much as possible while ensuring the service availability, thereby reducing the impact of the file migration process on the service operation.

[0078] Since each thread can execute one migration task, the number of threads of each slave node is equal to the number of migration tasks to be executed by the slave node. Based on the number of threads of each slave node and the first file subset, the migration task is sent to each slave node, which can ensure that the size of the files migrated by each thread is relatively small, so that the transmission efficiency of each thread is relatively small, and the overall migration time depends on the migration time of the largest file, so the migration efficiency is improved, thereby reducing the impact of the file migration process on the service operation of the source end.

[0079] In one embodiment, in the above S206, for each slave node, based on the number of threads of the slave node and the first file subset to be allocated to the slave node, sending the task to be migrated to the slave node may include the following steps:

[0080] S261, allocating threads on the source end based on the number of threads of the slave node to obtain a first thread set to be allocated to the slave node.

[0081] S262: Allocate the first file subset based on the sizes of the files in the first file subset to obtain a file to be allocated to each thread in the first thread set.

[0082] Determining the files to be allocated to each thread based on the sizes of the files in the first file subset can ensure that the sizes of the files migrated by each thread are relatively even, so that the transmission efficiency of each thread has a small difference.

[0083] As an example, when the number of files in the first file subset is less than or equal to the number of threads in the first thread set, the first file subset may be evenly distributed. For example, assuming that the first file subset includes file 1 and file 2, and the first thread set includes thread 1 and thread 2, it can be determined that the files to be allocated to thread 1 include file 1, and the files to be allocated to thread 2 include file 2.

[0084] As another example, the method for determining the files to be allocated to each thread includes the following steps: determining the arrangement sequence numbers of the files in the first file subset based on the size of the files in the first file subset; taking the number of threads in the first thread set as the dividend, performing a modulo operation on the arrangement sequence numbers of the files in the first file subset to obtain a second modulo operation result; allocating the first file subset based on the second modulo operation result to obtain the files to be allocated to each thread in the first thread set. This method is suitable for the case where the number of files in the first file subset is large and the number of threads in the first thread set is small.

[0085] For example, assuming that the first file subset includes 10 files, and the first thread set includes thread 1 and thread 2, the 10 files are sorted in descending order to obtain the arrangement number of each file. Then, with 2 as the dividend, a remainder operation is performed on the arrangement number of each file, and it is determined that the files to be allocated to thread 1 include files with a remainder of 1, and it is determined that the files to be allocated to thread 2 include files with a remainder of 0.

[0086] S263: Send a migration task to a slave node based on the first thread set and the file to be allocated to each thread in the first thread set.

[0087] Specifically, for each thread of the slave node, based on the file to be assigned to the thread, a migration task to be executed by the thread can be created. For example, if the first thread set of the slave node includes two threads, marked as thread 1 and thread 2, the files to be assigned to thread 1 include file 1 and file 3, and the files to be assigned to thread 2 include file 2 and file 4, then migration task 1 is created for thread 1, and migration task 2 is created for thread 2. Migration task 1 is used to migrate file 1 and file 3 from the source end to the destination end through thread 1, and migration task 2 is used to migrate file 2 and file 4 from the source end to the destination end through thread 2.

[0088] The embodiment of the present application shows some implementation methods of the above S206. Of course, it should be understood that the above S206 can also be implemented in other ways, and the embodiment of the present application does not limit this.

[0089] In another embodiment of the present application, in order to avoid slave node abnormalities during the migration process and affect the overall migration efficiency, after the master node sends the migration task to the slave node, it also monitors the execution status of the migration task reported by the slave node, and then eliminates the abnormal slave nodes, and reallocates the files that have not been migrated by the abnormal slave nodes to other normal slave nodes.

[0090] Specifically, see 3, which is a flowchart of a task scheduling method provided in another embodiment of the present application. The method is applied to a master node in a distributed cluster. The method may include the following steps:

[0091] S302, determining a plurality of slave nodes in a distributed cluster whose operating status is normal.

[0092] In the embodiment of the present application, the above S302 and Figure 2 The specific implementation of S202 in the illustrated embodiment is similar and will not be described in detail here.

[0093] S304: Allocate the file set to be migrated based on the sizes of the files in the file set to be migrated to obtain a first file subset to be allocated to each slave node.

[0094] In the embodiment of the present application, the above S304 and Figure 2 The specific implementation of S204 in the illustrated embodiment is similar and will not be repeated here.

[0095] S306: Based on the number of threads of each slave node and the first file subset, a migration task is issued to each slave node.

[0096] In the embodiment of the present application, the above S306 and Figure 2 The specific implementation of S206 in the illustrated embodiment is similar and will not be described in detail here.

[0097] S308: monitor the execution status of the migration task reported by multiple slave nodes.

[0098] S310: Determine a slave node that has not reported an execution status within a specified period as a second slave node, and determine a slave node that has reported an execution status within a specified period as a third slave node.

[0099] After receiving the migration task sent by the master node, the slave node reports the execution status of the migration task to the master node at a preset time interval, such as XX file has been migrated, YY file is being migrated, etc. The master node can count the migration status of each file in the file set to be migrated and the migration progress (such as completion rate, etc.) of the file set to be migrated based on the execution status reported by the slave node.

[0100] The master node also determines whether the slave node is abnormal based on the execution status reported by the slave node. As an example, if a slave node does not report the execution status within a specified period of time, the slave node is determined to be disconnected or abnormal, and then the slave node is determined to be the second slave node, and the slave node is deleted. The deletion method can be to comment out the information of the slave node in the node information file.

[0101] If a slave node reports the execution status within a specified period of time, it is determined that the slave node is normal, and then the slave node is determined as the third slave node.

[0102] S312: Allocate the files that have not been migrated from the second slave node to the third slave node.

[0103] The files that have not been migrated may include but are not limited to files that have not been migrated, files that are being migrated, etc.

[0104] The master node can allocate the files that have not been migrated by the second slave node to the third slave node in various ways. As an example, the above-mentioned Figure 2 In the illustrated embodiment, the incompletely migrated files are allocated to the third slave node in a similar manner to the implementation of S204, which will not be described in detail.

[0105] One or more embodiments of the present application provide a task scheduling method in which a master node in a distributed cluster uniformly performs task scheduling, determines multiple slave nodes in the distributed cluster that are in normal operating status, and allocates the file set to be migrated, obtains the first file subset to be allocated to each slave node, and then issues a corresponding migration task to each slave node to migrate the first file subset, so that the migration of the file set to be migrated is completed through multiple slave nodes, the file migration efficiency is improved, and the impact of the file migration process on the service operation of the source end can be reduced.

[0106] In addition, the technical solution proposed in the embodiment of the present application also improves the allocation method of the file set to be migrated. First, the file set to be migrated is allocated based on the size of the files in the file set to be migrated, and the first file subset to be allocated to each slave node is obtained to ensure that the size of the files to be allocated to each slave node is relatively even; then, based on the number of threads of each slave node and the first file subset to be allocated to each slave node, a migration task is issued to each slave node to ensure that the size of the files to be migrated of each thread is relatively small, so that the transmission efficiency of each thread is relatively small, the overall migration time depends on the migration time of the largest file, and the migration efficiency is improved, thereby reducing the impact of the file migration process on the service operation of the source end.

[0107] See also Figure 4 , is a flowchart of a file migration method provided by an embodiment of the present application, the method is applied to a slave node in a distributed cluster, and the method includes the following steps:

[0108] S402, reporting the running status to the master node in the distributed cluster.

[0109] The master node may determine, based on the received running status, a slave node in the distributed cluster with a normal running status to perform the migration task.

[0110] S404, executing the migration task issued by the master node.

[0111] Among them, the migration task is issued by the master node to the slave node based on the number of threads of the slave node and the first file subset to be allocated to the slave node. The first file subset is obtained by the master node allocating the file set to be migrated based on the size of the files in the file set to be migrated when the running state is normal.

[0112] After receiving the migration task, the slave node establishes communication connections with the source and destination ends respectively according to the source and destination addresses indicated by the migration task, and starts threads based on the number of threads indicated by the migration task. Then, the slave node pulls the file indicated by the migration task from the source end, and migrates the file to the destination end through the thread corresponding to the file.

[0113] In one embodiment, when the master node issues a migration task to the slave node, if it is necessary to perform file verification before migrating a certain file, a verification mark is carried in the file. In this case, the above S404 may include the following steps: traversing the files in the first file subset, if the traversed file carries a verification mark and the file does not exist in the destination, or the file does not carry a verification mark, then the file is migrated from the source to the destination; if the traversed file carries a verification mark and the file already exists in the destination, then the file is skipped, that is, the file is not migrated, but the next file in the first file subset is traversed.

[0114] For example, the slave node receives migration task 1, which indicates that file 1, file 3, and file 5 are migrated from the source end to the destination end through thread 1. The slave node traverses each file that migration task 1 indicates to migrate. When traversing to file 1, file 1 carries a verification mark and file 1 does not exist in the destination end, so file 1 is migrated from the source end to the destination end through thread 1; it continues to traverse file 2. File 2 carries a verification mark and file 2 exists in the destination end, so file 2 is not migrated; it continues to traverse file 3. File 3 does not carry a verification mark, so file 3 is migrated from the source end to the destination end through thread 1. At this point, the slave node has completed migration task 1.

[0115] In the above implementation, when the migration task indicates that the migrated file carries a verification mark, it is first verified whether the file exists in the destination end. If it exists, the migration operation of the file is skipped. This not only does not destroy the integrity of the existing files in the destination end and meets the user's needs to retain the files in the destination end, but also can reduce the number of files that need to be migrated from the node and shorten the overall migration time.

[0116] In addition, if the migration task indicates that the file to be migrated does not carry a verification mark, the file will be directly overwritten and migrated regardless of whether it exists in the destination. In this way, there is no need to perform a large number of procedural verification operations for the migration of massive files, saving resources, reducing the load on the slave node, and improving the overall migration efficiency. This situation can meet the needs of actual application scenarios, such as when the user clearly states that the files in the destination are always based on the source.

[0117] One or more embodiments of the present application provide a file migration method in which a master node in a distributed cluster uniformly performs task scheduling, determines multiple slave nodes in the distributed cluster that are in normal operating status, and allocates a file set to be migrated, thereby obtaining a first file subset to be allocated to each slave node, and then issues a corresponding migration task to each slave node to migrate the first file subset, so that the migration of the file set to be migrated is completed through multiple slave nodes, the file migration efficiency is improved, and the impact of the file migration process on the service operation of the source end can be reduced.

[0118] In addition, the technical solution proposed in the embodiment of the present application also improves the allocation method of the file set to be migrated. First, the file set to be migrated is allocated based on the size of the files in the file set to be migrated, and the first file subset to be allocated to each slave node is obtained to ensure that the size of the files to be allocated to each slave node is relatively even; then, based on the number of threads of each slave node and the first file subset to be allocated to each slave node, a migration task is issued to each slave node to ensure that the size of the files to be migrated of each thread is relatively small, so that the transmission efficiency of each thread is relatively small, the overall migration time depends on the migration time of the largest file, and the migration efficiency is improved, thereby reducing the impact of the file migration process on the service operation of the source end.

[0119] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0120] Based on the same inventive concept, the present application also provides a task scheduling device, which can be applied to the master node in a distributed cluster. Figure 5 , is a structural diagram of a task scheduling device provided by an embodiment of the present application. In a software implementation, the task scheduling device 500 may include: a determination unit 510, an allocation unit 520 and a sending unit 530.

[0121] The determining unit 510 is configured to determine a slave node in the distributed cluster whose operating status is normal.

[0122] The allocating unit 520 is configured to allocate the file set to be migrated based on the size of the files in the file set to be migrated, so as to obtain a first file subset to be allocated to each of the slave nodes.

[0123] The sending unit 530 is used to send the migration task to each of the slave nodes based on the number of threads of each of the slave nodes and the first file subset.

[0124] In another embodiment, the determining unit is further configured to:

[0125] Determine the total number of threads of the source end based on the resource usage information of the source end;

[0126] The number of threads for each slave node is determined based on the total number of threads and the number of the plurality of slave nodes.

[0127] In another embodiment, the resource usage information includes the number of processor cores of the source end;

[0128] When the determining unit determines the total number of threads of the source end based on the resource usage information of the source end, the determining unit performs the following steps:

[0129] Based on the number of processor cores and the number of the plurality of slave nodes, the total number of threads on the source end is determined.

[0130] In another embodiment, the resource usage information includes a total resource utilization rate and a transmission resource utilization rate of the source end;

[0131] When the determining unit determines the total number of threads of the source end based on the resource usage information, the determining unit performs the following steps:

[0132] Based on the total resource utilization, determining an unused resource rate of the source end;

[0133] Based on the unused resource rate and the transmission resource utilization rate, the total number of threads on the source end is determined.

[0134] In another embodiment, the allocation unit is used to:

[0135] Determine a file allocation strategy for the file set to be migrated based on the number of files in the file set to be migrated and the number of the plurality of slave nodes;

[0136] The file set to be migrated is allocated based on the file allocation strategy and the sizes of the files in the file set to be migrated, so as to obtain a first file subset to be allocated to each of the slave nodes.

[0137] In another embodiment, when the number of files in the to-be-migrated file set is greater than the number of the plurality of slave nodes, the file allocation strategy includes an average allocation strategy and a remainder allocation strategy; the first file subset includes a first file and a second file;

[0138] When the allocation unit allocates the set of files to be migrated based on the file allocation strategy and the sizes of the files in the set of files to be migrated to obtain a first subset of files to be allocated to each of the slave nodes, the allocation unit performs the following steps:

[0139] Based on the size of the files in the to-be-migrated file set and the number of the plurality of slave nodes, dividing the to-be-migrated file set into a second file subset and a third file subset;

[0140] Allocate the second file subset based on the average allocation strategy to obtain a first file to be allocated to each slave node;

[0141] A modulo operation is performed on the file identifiers of the files in the third file subset to obtain a first modulo operation result, and the third file subset is allocated based on the modulo operation result and the modulo allocation strategy to obtain a second file to be allocated to each of the slave nodes.

[0142] In another embodiment, when the allocation unit divides the to-be-migrated file set into the second file subset and the third file subset based on the size of the files in the to-be-migrated file set and the number of the plurality of slave nodes, the allocation unit performs the following steps:

[0143] Sorting the files in the to-be-migrated file set according to their sizes to obtain arrangement sequence numbers of the files in the to-be-migrated file set;

[0144] Determining a target arrangement sequence number based on the number of the plurality of slave nodes;

[0145] The files whose sequence numbers in the to-be-migrated file set are less than or equal to the target sequence number are obtained to obtain a second file subset, and the files whose sequence numbers in the to-be-migrated file set are greater than the target sequence number are obtained to obtain a third file subset.

[0146] In another embodiment, the sending unit is used to:

[0147] Allocate the threads of the source end based on the number of threads of the slave node to obtain a first thread set to be allocated to the slave node;

[0148] Allocating the first file subset based on the size of the files in the first file subset to obtain a file to be allocated to each thread in the first thread set;

[0149] Based on the first thread set and the file to be allocated to each thread in the first thread set, a migration task is issued to the slave node.

[0150] In another embodiment, the task scheduling device further includes:

[0151] A monitoring unit, used for monitoring the execution status of the migration tasks reported by the multiple slave nodes;

[0152] The determining unit is further configured to determine a slave node that has not reported an execution status within a specified period as a second slave node, and to determine a slave node that has reported an execution status within the specified period as a third slave node;

[0153] The allocation unit is further used for allocating the files that have not been migrated completely from the second slave node to the third slave node.

[0154] Obviously, the task scheduling device in the embodiment of this specification can be used as the above Figure 2The execution body of the task scheduling method shown in FIG. Figure 2 The functions realized are as follows. Since the principles are the same, they will not be described here.

[0155] Based on the same inventive concept, the present application also provides a file migration device, which can be applied to slave nodes in a distributed cluster. Figure 6 , is a structural diagram of a file migration device provided by an embodiment of the present application. In a software implementation, the file migration device 600 may include: a reporting unit 610 and an execution unit 620.

[0156] The reporting unit 610 is used to report the running status to the master node in the distributed cluster.

[0157] The execution unit 620 is used to execute the migration task issued by the master node.

[0158] Among them, the migration task is issued by the master node to the slave node based on the number of threads of the slave node and the first file subset to be allocated to the slave node. The first file subset is obtained by the master node allocating the file set to be migrated based on the size of the files in the file set to be migrated when the running state is normal.

[0159] In another embodiment, the execution unit is used to:

[0160] Traversing the files in the first file subset;

[0161] If the traversed file carries a verification mark and the file does not exist in the destination end, or the file does not carry the verification mark, the file is migrated from the source end to the destination end;

[0162] If the traversed file carries the verification identifier and the file already exists in the destination end, continue to traverse the next file in the first file subset.

[0163] Obviously, the file migration device in the embodiment of this specification can be used as the above Figure 4 The execution subject of the file migration method shown in FIG. Figure 2 The functions realized are as follows. Since the principles are the same, they will not be described here.

[0164] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 7At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0165] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0166] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0167] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a task scheduling device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0168] Determine the slave nodes in the distributed cluster that are running normally;

[0169] Allocating the set of files to be migrated based on the sizes of the files in the set of files to be migrated to obtain a first subset of files to be allocated to each of the slave nodes;

[0170] Based on the number of threads of each slave node and the first file subset, a migration task is issued to each slave node.

[0171] Alternatively, the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a file migration device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0172] Report the running status to the master node in the distributed cluster;

[0173] Execute the migration task issued by the master node;

[0174] Among them, the migration task is issued by the master node to the slave node based on the number of threads of the slave node and the first file subset to be allocated to the slave node. The first file subset is obtained by the master node allocating the file set to be migrated based on the size of the files in the file set to be migrated when the running state is normal.

[0175] The above application Figure 2 The method performed by the task scheduling device disclosed in the embodiment shown, or the above-mentioned method of the present application Figure 4 The method performed by the file migration device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in software form. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0176] The electronic device may also perform Figure 2 The method and the task scheduling device are implemented in Figure 2 , Figure 3 Alternatively, the electronic device may also perform the functions of the embodiments shown. Figure 4 Method, and realize the file migration device in Figure 4The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0177] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0178] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 2 The method of the embodiment shown is specifically used to perform the following operations:

[0179] Determine the slave nodes in the distributed cluster that are running normally;

[0180] Allocating the set of files to be migrated based on the sizes of the files in the set of files to be migrated to obtain a first subset of files to be allocated to each of the slave nodes;

[0181] Based on the number of threads of each slave node and the first file subset, a migration task is issued to each slave node.

[0182] Alternatively, when the instruction is executed by a portable electronic device including a plurality of application programs, the portable electronic device can execute Figure 3 The method of the embodiment shown is specifically used to perform the following operations:

[0183] Report the running status to the master node in the distributed cluster;

[0184] Execute the migration task issued by the master node;

[0185] Among them, the migration task is issued by the master node to the slave node based on the number of threads of the slave node and the first file subset to be allocated to the slave node. The first file subset is obtained by the master node allocating the file set to be migrated based on the size of the files in the file set to be migrated when the running state is normal.

[0186] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps in the task scheduling method provided by the embodiment of the present application or the file migration method provided by the embodiment of the present application.

[0187] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0188] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0189] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0190] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0191] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A task scheduling method, characterized in that: Applied to a master node in a distributed cluster, the method comprises: Determine a plurality of slave nodes in the distributed cluster that are in normal operating status; Allocating the file set to be migrated based on the size of the files in the file set to be migrated to obtain a first file subset to be allocated to each of the slave nodes; Based on the number of threads of each slave node and the first file subset, a migration task is issued to each slave node.

2. The method according to claim 1, characterized in that: The file set to be migrated is stored in the source end, and the method further includes: Determining the total number of threads of the source end based on the resource usage information of the source end; The number of threads of each of the slave nodes is determined based on the total number of threads and the number of the plurality of slave nodes.

3. The method according to claim 2, characterized in that The resource usage information includes the number of processor cores of the source end; The determining the total number of threads of the source end based on the resource usage information of the source end includes: determining the total number of threads of the source end based on the number of processor cores and the number of the plurality of slave nodes.

4. The method according to claim 2, characterized in that: The resource usage information includes the total resource utilization rate and transmission resource utilization rate of the source end; The determining the total number of threads of the source end based on the resource usage information of the source end includes: Based on the total resource utilization, determining an unused resource rate of the source end; Based on the unused resource rate and the transmission resource utilization rate, the total number of threads on the source end is determined.

5. The method according to claim 1, characterized in that The allocating the file set to be migrated based on the size of the files in the file set to be migrated to obtain a first file subset to be allocated to each slave node includes: Determine a file allocation strategy for the file set to be migrated based on the number of files in the file set to be migrated and the number of the plurality of slave nodes; The file set to be migrated is allocated based on the file allocation strategy and the sizes of the files in the file set to be migrated, so as to obtain a first file subset to be allocated to each of the slave nodes.

6. The method according to claim 5, characterized in that In the case that the number of files in the to-be-migrated file set is greater than the number of the plurality of slave nodes, the file allocation strategy includes an average allocation strategy and a remainder allocation strategy; the first file subset includes a first file and a second file; The allocating the file set to be migrated based on the file allocation strategy and the size of the files in the file set to be migrated to obtain a first file subset to be allocated to each slave node includes: Based on the size of the files in the to-be-migrated file set and the number of the plurality of slave nodes, dividing the to-be-migrated file set into a second file subset and a third file subset; Allocate the second file subset based on the average allocation strategy to obtain a first file to be allocated to each slave node; A modulo operation is performed on the file identifiers of the files in the third file subset to obtain a first modulo operation result, and the third file subset is allocated based on the modulo operation result and the modulo allocation strategy to obtain a second file to be allocated to each of the slave nodes.

7. The method according to claim 6, characterized in that The step of dividing the file set to be migrated into a second file subset and a third file subset based on the size of the files in the file set to be migrated and the number of the plurality of slave nodes comprises: Sorting the files in the to-be-migrated file set according to their sizes to obtain arrangement sequence numbers of the files in the to-be-migrated file set; Determining a target arrangement sequence number based on the number of the plurality of slave nodes; The files whose sequence numbers in the to-be-migrated file set are less than or equal to the target sequence number are obtained to obtain a second file subset, and the files whose sequence numbers in the to-be-migrated file set are greater than the target sequence number are obtained to obtain a third file subset.

8. The method according to claim 1, characterized in that For each of the slave nodes, based on the number of threads of the slave node and the first file subset to be allocated to the slave node, sending a migration task to the slave node includes: Allocate the threads of the source end based on the number of threads of the slave node to obtain a first thread set to be allocated to the slave node; Allocating the first file subset based on the size of the files in the first file subset to obtain a file to be allocated to each thread in the first thread set; Based on the first thread set and the file to be allocated to each thread in the first thread set, a migration task is issued to the slave node.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Monitoring the execution status of the migration tasks reported by the multiple slave nodes; Determine a slave node that has not reported an execution status within a specified period as a second slave node, and determine a slave node that has reported an execution status within the specified period as a third slave node; Allocate the files that have not been migrated by the second slave node to the third slave node.

10. A task scheduling device, characterized in that: Applicable to the master node in a distributed cluster, including: A determination unit, used to determine a plurality of slave nodes in the distributed cluster whose operating status is normal; An allocating unit, configured to allocate the file set to be migrated based on the size of the files in the file set to be migrated, to obtain a first file subset to be allocated to each of the slave nodes; The sending unit is used to send the migration task to each of the slave nodes based on the number of threads of each of the slave nodes and the first file subset to be allocated to each of the slave nodes.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the task scheduling method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the task scheduling method as claimed in any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the task scheduling method according to any one of claims 1 to 10.

Images