Distributed computing method and device and related equipment

Abstract

The invention discloses a distributed computing method and device and related equipment, and the method comprises the steps: determining a plurality of idle computing units which are GPUs in computing nodes; determining a target computing unit cluster and a backup computing unit cluster from the plurality of computing units; controlling the computing units in the target computing unit cluster to execute the task; and in the process of executing the task, when the first computing unit in the target computing unit cluster is not idle, controlling the first computing unit to store related data in the process of executing the task in the shared memory, and controlling the second computing unit in the backup computing unit cluster to continuously execute the task according to the related data in the shared memory. When the computing unit which is executing the task is not idle, the computing unit does not need to be temporarily allocated to replace the execution of the task, and the task is quickly migrated to the backup computing unit to continue to be executed through the shared memory, so that the efficiency of performing distributed computing by using idle computing resources is improved, and the stability and continuity of task execution are guaranteed.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a distributed computing method, apparatus, and related equipment. Background Art

[0002] At present, with the rapid development of computer technology and the Internet, users' demand for computing resources is gradually expanding. Especially with the increasing popularity of artificial intelligence (AI) technology, users' demand for graphics processing units (GPUs) is increasing day by day. Therefore, users can improve the execution efficiency of tasks through distributed computing, that is, users assign tasks to multiple GPUs in the network for processing according to computing needs.

[0003] In actual applications, without increasing GPU hardware investment, users can rent multiple GPUs from GPU rental companies on the network within a certain period of time to perform tasks. However, the multiple rented GPUs are often not fully used during the rental period, and there is usually a lot of waste of GPU resources. In addition, during non-peak periods such as nighttime, there are also a large number of idle GPU resources in servers, personal computers and other devices. Traditional distributed computing is not effective in integrating these idle GPU resources, and problems are prone to occur when using these idle GPU resources to perform tasks. Summary of the invention

[0004] The embodiment of the present application provides a distributed computing method to improve the efficiency of using idle computing resources for distributed computing and ensure the stability and continuity of task execution. In addition, the embodiment of the present application also provides a corresponding distributed computing device, computing equipment, computer-readable storage medium and computer program product.

[0005] In a first aspect, an embodiment of the present application provides a distributed computing method, which is applied to a service platform, and the service platform includes a shared memory. The method includes: determining multiple idle computing units, wherein the computing units are graphics processing units (GPUs) in computing nodes, and the computing nodes are terminal devices or servers; determining a target computing unit cluster and a backup computing unit cluster, wherein the target computing unit cluster and the backup computing unit cluster are both sets of some computing units in the multiple computing units, and the target computing unit cluster and the backup computing unit cluster do not include the same computing unit; controlling the computing units in the target computing unit cluster to perform tasks; in the process of executing the task, when the first computing unit is not idle, controlling the first computing unit to store relevant data in the process of executing the task in the shared memory, and controlling the second computing unit to continue to perform the task according to the relevant data in the shared memory, wherein the first computing unit is a computing unit in the target computing unit cluster, and the second computing unit is a computing unit in the backup computing unit cluster.

[0006] In a possible implementation, the method further includes: sending an environment image corresponding to the task to a target cluster, the environment image being used for computing nodes in the target cluster to load a computing environment for the task based on the environment image, the computing nodes in the target cluster being computing nodes to which computing units in the target computing unit cluster belong. Controlling the computing units in the target computing unit cluster to execute the task includes: controlling the computing units in the target computing unit cluster to execute the task in the computing environment.

[0007] In a possible implementation, the method further includes: sending the environment image to a backup cluster, the environment image is also used for computing nodes in the backup cluster to load the computing environment of the task based on the environment image, and the computing nodes in the backup cluster are computing nodes to which computing units in the backup computing unit cluster belong. Controlling the second computing unit to continue to execute the task according to the relevant data in the shared memory includes: controlling the second computing unit to continue to execute the task according to the relevant data in the shared memory in the computing environment of the computing node to which the second computing unit belongs.

[0008] In a possible implementation, the plurality of computing units constitute a computing resource pool. The determining of the plurality of idle computing units comprises: obtaining idle computing resource information of a plurality of computing nodes, the idle computing resource information comprising the number, serial number and device model of the computing units; and determining the computing resource pool according to the idle computing resource information.

[0009] In a possible implementation, the idle computing resource information is specified by the computing node.

[0010] In one possible implementation, determining the target computing unit cluster and the backup computing unit cluster includes: obtaining task requirement information, the task requirement information including the number of computing units required to perform the task, the number of computing units to serve as backups, and a required device model, where the required device model is a device model of the computing unit that performs the task; and determining the target computing unit cluster and the backup computing unit cluster based on the task requirement information.

[0011] In one possible implementation, the method further includes: obtaining task requirement modification information, the task requirement modification information including increasing or decreasing the number of computing units required to perform the task, increasing or decreasing the number of computing units used as backups and the required device models; and adjusting the target computing unit cluster and the backup computing unit cluster according to the task requirement modification information.

[0012] In a possible implementation, the method further includes: when the first computing unit is not idle, moving the first computing unit out of the computing resource pool; or, when the third computing unit is not idle, moving the third computing unit out of the computing resource pool, the third computing unit being a computing unit in the computing resource pool that does not belong to the target computing unit cluster and the backup computing unit cluster.

[0013] In a second aspect, the present application provides a distributed computing device, which includes: a determination module, the determination module is used to determine multiple idle computing units, the computing unit is a graphics processing unit GPU in a computing node, and the computing node is a terminal device or a server; determine a target computing unit cluster and a backup computing unit cluster, the target computing unit cluster and the backup computing unit cluster are both sets composed of some computing units in the multiple computing units, and the target computing unit cluster and the backup computing unit cluster do not include the same computing unit; a control module, the control module is used to control the computing units in the target computing unit cluster to perform tasks; in the process of executing the task, when the first computing unit is not idle, control the first computing unit to store relevant data in the process of executing the task in the shared memory, and control the second computing unit to continue to execute the task according to the relevant data in the shared memory, the first computing unit is a computing unit in the target computing unit cluster, and the second computing unit is a computing unit in the backup computing unit cluster.

[0014] In a possible implementation, the device further includes: a sending module, the sending module is used to send the environment image corresponding to the task to the target cluster, the environment image is used for the computing nodes in the target cluster to load the computing environment of the task based on the environment image, and the computing nodes in the target cluster are computing nodes to which the computing units in the target computing unit cluster belong. The control module is specifically used to control the computing units in the target computing unit cluster to execute the task in the computing environment.

[0015] In a possible implementation, the sending module is further used to send the environment image to the backup cluster, and the environment image is also used for the computing nodes in the backup cluster to load the computing environment of the task based on the environment image, and the computing nodes in the backup cluster are computing nodes to which the computing units in the backup computing unit cluster belong. The control module is specifically used to control the second computing unit to continue to execute the task in the computing environment of the computing node to which the second computing unit belongs according to the relevant data in the shared memory.

[0016] In a possible implementation, the plurality of computing units constitute a computing resource pool. The determination module is specifically configured to: obtain idle computing resource information of a plurality of computing nodes, the idle computing resource information including the number, serial number and device model of the computing units; and determine the computing resource pool according to the idle computing resource information.

[0017] In a possible implementation, the idle computing resource information is specified by the computing node.

[0018] In one possible implementation, the determination module is specifically used to: obtain task requirement information, the task requirement information including the number of computing units required to perform the task, the number of computing units serving as backups, and the required device model, where the required device model is the device model of the computing unit that performs the task; determine the target computing unit cluster and the backup computing unit cluster based on the task requirement information.

[0019] In one possible embodiment, the device also includes: an adjustment module, which is used to obtain task requirement modification information, the task requirement modification information including increasing or decreasing the number of computing units required to perform the task, increasing or decreasing the number of computing units used as backups and the required equipment model; adjusting the target computing unit cluster and the backup computing unit cluster according to the task requirement modification information.

[0020] In a possible implementation, the device further includes: a removal module, which is used to move the first computing unit out of the computing resource pool when the first computing unit is not idle; or to move the third computing unit out of the computing resource pool when the third computing unit is not idle, wherein the third computing unit is a computing unit in the computing resource pool that does not belong to the target computing unit cluster and the backup computing unit cluster.

[0021] In a third aspect, an embodiment of the present application further provides a computing device, which may include a processor and a memory:

[0022] The memory is used to store computer programs;

[0023] The processor is used to execute the method described in the first aspect and any one of the embodiments of the first aspect according to the computer program.

[0024] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method described in the first aspect and any one of the implementation methods of the first aspect.

[0025] In a fifth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the method described in the first aspect and any one of the implementations of the first aspect.

[0026] In the above implementation of the embodiment of the present application, multiple idle computing units are determined, and the computing unit is a graphics processor GPU in a computing node, and the computing node is a terminal device or a server. Secondly, a target computing unit cluster and a backup computing unit cluster are determined, and the target computing unit cluster and the backup computing unit cluster are both sets of some computing units in multiple computing units, and the target computing unit cluster and the backup computing unit cluster do not include the same computing unit. Subsequently, the computing units in the target computing unit cluster are controlled to perform tasks, and in the process of performing tasks, when the first computing unit is not idle, the first computing unit is controlled to store the relevant data in the process of performing tasks in a shared memory, and the second computing unit is controlled to continue to perform tasks according to the relevant data in the shared memory, wherein the first computing unit is a computing unit in the target computing unit cluster, and the second computing unit is a computing unit in the backup computing unit cluster. In this way, multiple idle computing units are determined, and the corresponding computing units are allocated according to the actual needs of the user's tasks, that is, the target computing unit cluster and the backup computing unit cluster are determined for the user to perform tasks, so that idle computing resources can be fully utilized, and can be uniformly managed and scheduled, effectively avoiding the situation where computing resources are wasted. Moreover, by determining the target computing unit cluster and the backup computing unit cluster in advance and using shared memory to realize the rapid sharing of relevant data between computing units, when the computing unit that is executing the task is not idle, there is no need to temporarily allocate computing units to take over the task. At the same time, the task can be quickly migrated to the backup computing unit through shared memory for continued execution, which greatly reduces the time consumption of task migration of idle computing resources, effectively improves the overall efficiency of using idle computing resources for distributed computing, and ensures the stability and continuity of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0028] Figure 1 This is a schematic diagram of the architecture of a distributed computing system in an embodiment of the present application;

[0029] Figure 2 A schematic diagram of a distributed computing method in an embodiment of the present application;

[0030] Figure 3 A schematic diagram of the architecture of a distributed computing software in an embodiment of the present application;

[0031] Figure 4A schematic diagram of the structure of a distributed computing device in an embodiment of the present application;

[0032] Figure 5 A schematic diagram of the hardware structure of a computing device in an embodiment of the present application. DETAILED DESCRIPTION

[0033] See also Figure 1 , is a schematic diagram of a distributed computing architecture provided by this application. Figure 1 As shown, the distributed computing system 10 includes a processor 100, a shared memory 200, and at least one computing node. Figure 1 The example in which computing nodes 301 and 302 are included is taken. In actual application scenarios, the distributed computing system 10 may include any number of computing nodes.

[0034] In addition, a computing node includes at least one computing unit, e.g. Figure 1 In the example, computing node 301 includes computing unit 401 and computing unit 402, and computing node 302 includes computing unit 403, computing unit 404, and computing unit 405. In actual application scenarios, a computing node may include any number of computing units.

[0035] The processor 100 and the shared memory 200 may be connected via a bus, which may be, for example, a peripheral component interconnect express (PCIE) bus, etc. The processor 100 and the computing node 301 or the computing node 302 may be connected via a network, and the shared memory 200 and the computing node 301 or the computing node 302 may also be connected via a network, which may be, for example, a local area network (LAN), a wide area network (WAN), etc. in terms of coverage, and may be, for example, a wired network or a wireless network, etc. in terms of connection mode.

[0036] The processor 100 refers to a device with data processing and control capabilities, such as a central processing unit (CPU), a microprocessor (MPU), etc., or an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc., without limitation.

[0037] The shared memory 200 refers to a device with data storage capability, and may be, for example, a semiconductor memory, a magnetic memory, an optical memory, etc., but is not limited thereto.

[0038] The computing unit 401 (other computing units are similar) refers to a device with computing capabilities, such as a GPU, etc., but this is not limited.

[0039] The computing node 301 (similar to the computing node 302 ) refers to a device equipped with a computing unit, such as a terminal device, a server, etc., but is not limited to this.

[0040] Generally, the working states of different computing units in the distributed computing system 10 may be different. For example, some computing units may be idle, while other computing units may not be idle. Figure 1 In the distributed computing system 10, one or more computing units in an idle state may constitute a computing resource pool 40. Figure 1 As shown, computing resource pool 40 may include computing unit 401 in computing node 301 , and computing units 403 and 404 in computing node 302 (assuming that these computing units are in an idle state).

[0041] The computing resource pool 40 is a virtualized resource pool composed of one or more physical computing units. In actual application, the computing resource pool 40 includes a record of information of one or more computing units. For example, the record can be a table in a database, etc., which is not limited to this.

[0042] In actual application scenarios, users' demand for computing resources is gradually increasing, especially when it comes to the use of AI technology. Users often need more computing resources to improve the computing efficiency of tasks. For example, in the training of deep learning models, GPUs that are good at parallel computing are usually used to accelerate the processing of large amounts of training data, and multiple GPUs working together can significantly accelerate the training process. Therefore, for users, enterprises and other computing resource users, tasks can be assigned to multiple GPUs in the network for parallel processing, that is, using distributed computing methods to further improve the efficiency of task execution.

[0043] However, since directly purchasing a large number of GPUs for distributed computing will cause great economic pressure on computing resource users, short-term, irregular or unstable computing needs will result in low cost-effectiveness of GPU hardware investment. Therefore, computing resource users often rent multiple GPUs for a certain period of time from GPU leasing companies in the network to perform tasks. However, the rented GPU resources are often not fully utilized. This is because computing resource users usually do not always run tasks at full load during the rental period, and the computing requirements of tasks may also change dynamically, resulting in a window period when the rented GPU resources are idle. In addition, there are also a large number of GPU resources in servers, personal computers and other devices. These GPU resources are also in an idle state and are not fully utilized during non-peak periods such as at night.

[0044] Generally, these idle GPU resources can be integrated to meet the computing needs of computing resource users, so that GPU resources are not wasted in large quantities. However, traditional distributed computing is not very effective in integrating idle resources, and problems are prone to occur when using these idle GPU resources to perform tasks. This is because idle GPU resources may be reclaimed by the device to which the GPU resources belong at any time during the execution of tasks and will no longer be idle, which may cause the ongoing tasks to be interrupted at any time, and the migration of tasks to the newly allocated GPU resources will cause a large delay, greatly affecting the overall efficiency of distributed computing.

[0045] Based on this, in the distributed computing system 10 provided in the present application, the processor 100 will determine a plurality of idle computing units, for example, a plurality of computing units can constitute a computing resource pool, wherein the computing unit is a GPU in a computing node, and the computing node is a terminal device or a server. Furthermore, the processor 100 determines a target computing unit cluster and a backup computing unit cluster, wherein the target computing unit cluster and the backup computing unit cluster are both sets of some computing units in a plurality of computing units, and the target computing unit cluster and the backup computing unit cluster do not include the same computing unit, for example, the processor 100 can use 5 of the plurality of computing units as the target computing unit cluster, and use 3 of the plurality of computing units as the backup computing unit cluster. The processor 100 will control the computing units in the target computing unit cluster to perform tasks, and in the process of performing tasks, when the first computing unit in the target computing unit cluster is not idle, the processor 100 will control the first computing unit to store the relevant data in the process of performing tasks in a shared memory, and control the second computing unit in the backup computing unit cluster to continue to perform tasks according to the relevant data in the shared memory.

[0046] In this way, the processor 100 determines a plurality of computing units, and allocates corresponding computing units to computing resource users according to actual computing needs, that is, determines a target computing unit cluster and a backup computing unit cluster for computing resource users to perform tasks. Thus, the processor 100 can fully integrate and utilize idle computing resources, and perform unified management, allocation and scheduling, effectively avoiding the situation where computing resources are wasted. In addition, the processor 100 also determines the target computing unit cluster and the backup computing unit cluster in advance, and uses a shared memory to realize the rapid sharing of relevant data between computing units, so that when the computing unit that is executing a task in the target computing unit cluster is recovered by the computing node to which it belongs and is not idle, the processor 100 does not need to temporarily allocate a computing unit to take over the execution of the task, and can quickly migrate the task to the computing unit of the backup computing unit cluster through the shared memory to continue to execute, which greatly reduces the time-consuming task migration when the idle computing resources cannot continue to execute the task, effectively improves the overall efficiency of integrating idle computing resources for distributed computing, and ensures the stability and continuity of task execution.

[0047] Understandably, Figure 1 The architecture of the distributed computing system shown is only an example of an architecture provided by the embodiment of the present application. In other possible distributed computing systems, more number of computing nodes, more number of computing units, more number of processors or more number of shared memories may also be included.

[0048] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, various non-limiting implementation methods in the embodiments of the present application are exemplarily described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0049] See also Figure 2 , Figure 2 A schematic diagram of a distributed computing method in an embodiment of the present application is shown, which can be applied to Figure 1 The distributed computing system 10 shown in FIG. 1 may be applied to other applicable distributed computing systems. Figure 1 The distributed computing system 10 shown in the figure is used as an example to illustrate the distributed computing. Figure 2 As shown, the method may specifically include the following steps.

[0050] S201: The processor 100 determines a plurality of idle computing units, where the computing units are GPUs in computing nodes, and the computing nodes are terminal devices or servers.

[0051] In terminal devices or servers, there are a large number of idle GPU resources. The processor 100 can determine these idle GPU resources and perform unified management, task allocation and scheduling to avoid wasting idle GPU resources. Among them, the terminal device or server, as a computing node of distributed computing, can communicate with the processor 100 through the network to transmit tasks and other task-related data; the idle GPU resources, as a computing unit of distributed computing, are responsible for actually completing the tasks assigned by the processor 100.

[0052] Specifically, the multiple computing units determined by the processor 100 can constitute a computing resource pool 40. The computing resource pool 40 is a virtualized resource pool composed of one or more physical computing units, wherein the computing resource pool 40 includes information of multiple computing units, and the processor 100 performs task allocation and migration according to the computing resource pool 40. Among them, the computing resource pool 40 can be a table in a database, which records the relevant information of each computing unit and the corresponding relationship between each computing unit and the computing node to which it belongs. In addition, the computing resource pool 40 can also record other contents such as the Internet Protocol (IP) address and the Media Access Control Address (MAC) of the computing node to which each computing unit belongs, which is not limited.

[0053] As an implementation example, the processor 100 may obtain idle computing resource information corresponding to a plurality of computing nodes, and determine a computing resource pool 40 according to the idle computing resource information, wherein the idle computing resource information includes the number, serial number, and device model of computing units.

[0054] Specifically, there are several computing units in the computing node, and some of these computing units may be idle. The processor 100 can obtain the number of idle computing units in each computing node and add these computing units to the computing resource pool 40. In order to facilitate the management of these idle computing units, the processor 100 can also obtain the identification of these computing units to distinguish different computing units, wherein the identification of the computing unit can be the device identification number of the computing unit, or it can be a combination of the device identification number of the computing node to which it belongs and the serial number of the computing unit in the computing node to which it belongs, and there is no limitation on this. In addition, the processor 100 can also obtain the device models of these computing units, wherein the device model can indicate the computing power of the corresponding computing unit. For example, the processor 100 can determine the number of cores, floating-point operation performance and other information of the computing unit through the device model.

[0055] Furthermore, there are multiple implementations of the processor 100 obtaining the idle computing resource information of the computing node as follows, which are described by taking the processor 100 obtaining the idle computing resource information of the computing node 302 as an example.

[0056] In the first implementation, for a computing node that autonomously joins the distributed computing system 10, it can be considered that all computing units in the computing node are in an idle state, that is, they can all be uniformly managed and assigned tasks by the processor 100. The processor 100 can automatically obtain the number of all computing units in the computing node and the corresponding identifiers and corresponding device models. For example, the processor 100 can obtain the number of all computing units in the computing node 302, and the respective numbers and device models of the three computing units, namely, computing unit 403, computing node 404, and computing unit 405.

[0057] In the second implementation, only some computing units in the computing node are in an idle state, and the processor 100 can automatically obtain the occupancy rate of each computing unit and determine whether the computing unit is in an idle state. Specifically, the processor 100 can regard the computing units whose occupancy rate is lower than the occupancy rate threshold as computing units in an idle state, and determine the number of computing units in an idle state and the corresponding numbers and corresponding device models. For example, for the computing units in the computing node 302, the occupancy rate of the computing unit 405 is higher than the occupancy rate threshold, while the occupancy rates of the computing units 403 and 404 are both lower than the occupancy rate threshold. Therefore, the processor 100 determines that the computing units 403 and 404 in the computing node 302 are in an idle state, and then the processor 100 can determine the number of computing units in an idle state, as well as the respective identifiers and respective device models of the two computing units 403 and 404.

[0058] In the third implementation, even if there are a large number of idle computing units in the computing node, the computing node may not allow all the idle computing units to be uniformly managed and assigned tasks by the processor 100. For example, the computing node itself also has computing needs, and some computing units are only in a short idle state and do not need to be added to the computing resource pool 40. At this time, the idle computing resource information obtained by the processor 100 can be information specified by the computing node, that is, the idle computing resource information is the number of idle computing units specified by the computing node and the corresponding identifier and corresponding device model. For example, the computing node 302 only specifies that the computing unit 403 is in an idle state, and the idle computing resource information obtained by the processor 100 includes the number of idle computing units, as well as the identifier and device model of the computing unit 403.

[0059] It is worth noting that the above three implementation methods are only used as some exemplary explanations. In actual application, the processor 100 can also obtain the idle computing resource information of the computing node based on other methods, which is not limited to this.

[0060] S202: The processor 100 determines a target computing unit cluster and a backup computing unit cluster, wherein the target computing unit cluster and the backup computing unit cluster are both sets consisting of some computing units in a plurality of computing units, and the target computing unit cluster and the backup computing unit cluster do not include the same computing unit.

[0061] Based on the determined multiple computing units, the processor 100 can determine that some of the multiple computing units are used to perform the task, that is, the target computing unit cluster, and can also determine that another part of the multiple computing units are used for the backup of the task execution, that is, the backup computing unit cluster, wherein the target computing unit cluster and the backup computing unit cluster do not contain the same computing unit, and the number of computing units contained in the target computing unit cluster and the backup computing unit cluster can be equal or unequal. In addition, the target computing unit cluster and the backup computing unit cluster can contain only one computing unit or a larger number of computing units, and there is no limitation on this.

[0062] When actually executing a task, multiple computing units constitute a computing resource pool 40. When the computing units in the target computing unit cluster are not idle, the processor 100 can control the computing units in the backup computing unit cluster to take over the execution of the task without temporarily reallocating new computing units from the determined computing resource pool 40, thereby ensuring the continuity and stability of task execution.

[0063] As an implementation example, the processor 100 can obtain the task requirement information of the computing resource user, and determine the target computing unit cluster and the backup computing unit cluster according to the task requirement information. Specifically, the processor 100 can obtain the number of computing units required for the actual execution of the task specified by the computing resource user, that is, the number of computing units contained in the target computing unit cluster; it can also obtain the number of computing units specified by the computing resource user as a backup for task execution, that is, the number of backup computing unit clusters. In addition, the processor 100 can also obtain the device model of the computing unit required for the execution of the task specified by the computing resource user, that is, the required device model, wherein the required device model is one or more of the multiple device models contained in the computing resource pool 40. For example, the computing resource user can specify that 5 computing units are required for the actual execution of the task, and 3 computing units are required as a backup for the execution of the task, and specify that these computing units are all of a certain specified device model. The processor 100 assigns a matching target computing unit cluster and a backup computing unit cluster to the task according to the task requirement information specified by the computing resource user.

[0064] Further, the processor 100 can obtain the task requirement modification information of the computing resource user, and adjust the target computing unit cluster and the backup computing unit cluster according to the task requirement information. Specifically, the computing resource user can timely adjust the number of computing units or device models required based on the dynamically changing task requirements. The processor 100 can obtain the number of computing units specified by the computing resource user to increase or decrease the number of computing units that actually execute the task, that is, adjust the number of the target computing unit cluster; it can also obtain the number of computing units specified by the computing resource user to increase or decrease as the backup for task execution, that is, adjust the number of backup computing unit clusters. In addition, the processor 100 can also obtain the device model of the computing unit required for the execution of the task adjusted by the computing resource user, that is, the adjusted required device model. For example, as the task requirements change, the computing resource user can add 2 computing units to execute the task, and at the same time add 2 computing units as the backup for executing the task, thereby improving the efficiency of task execution. The computing resource user can also adjust the required device model so that the computing power of the adjusted computing unit is stronger. The processor 100 adjusts the target computing unit cluster and the backup computing unit cluster for the task accordingly according to the task requirement modification information specified by the computing resource user obtained.

[0065] Furthermore, in order to enable the task to be quickly deployed to the computing unit and start execution, the processor 100 can also send the environment image corresponding to the task to the target cluster, wherein the target cluster is the computing node to which the computing unit in the target computing unit cluster belongs. Among them, the environment image can be provided by the computing resource user, or the computing resource user can select an environment image that matches the task from common basic environment images. When the computing node in the target cluster receives the environment image, it can load the computing environment required for task execution based on the environment image, so that the computing unit can start executing the assigned task in the computing environment. In addition, the computing resource user can also make other settings for the computing environment through the processor 100. For example, the processor 100 can modify some parameters in the computing environment so that the task can be executed smoothly.

[0066] In this way, the processor 100 can uniformly manage the environment images of the tasks, so that different tasks can be quickly deployed on different computing nodes, and then the execution of the tasks can be started quickly.

[0067] In addition, in order to enable the task being executed to be quickly migrated to the computing unit in the backup computing unit cluster for succession execution, the processor 100 can also send the environment image corresponding to the task to the backup cluster, where the backup cluster is the computing node to which the computing unit in the backup computing unit cluster belongs. When the computing node in the backup cluster receives the environment image, it can load the computing environment required for task execution based on the environment image so that the computing unit can quickly take over the execution of the task in the computing environment. In addition, other settings of the computing environment by the computing resource user will be synchronized with the target cluster and the backup cluster, so that after the task is migrated, the computing environment for executing the task will not change.

[0068] In this way, the processor 100 sends the environment image corresponding to the task to the target cluster and the backup cluster at the same time, so that when the task being executed needs to be migrated, there is no need to wait for the computing environment required for the task to be loaded, thereby ensuring the continuity and stability of task execution.

[0069] It should be noted that when the task execution is completed or the computing resource user does not need to execute the task, the processor 100 will release the computing units in the target computing unit cluster and the backup computing unit cluster to execute other tasks.

[0070] S203: The processor 100 controls the computing units in the target computing unit cluster to execute the task.

[0071] After determining the target computing unit cluster, the processor 100 can control the computing units in the target computing unit cluster to start executing the task. Specifically, the processor 100 can send control commands to the computing units in the target computing unit cluster, so that each computing unit executes the task in the configured computing environment, and the processor 100 can obtain the execution status of the task in real time to ensure that the task is executed smoothly.

[0072] As an implementation example, the relevant data of the task is stored in the shared memory 200, and the processor 100 can send a control command to the computing unit in the target computing unit cluster, so that each computing unit starts to execute the task based on the relevant data stored in the shared memory 200, wherein the relevant data of the task includes the initial data for executing the task, and also includes the newly generated data during the subsequent task execution. For example, when performing a model training task, the relevant data of the task includes a data set, a model file, parameter data during the training process, a log file, and intermediate results, and these relevant data are stored together in the shared memory 200 and managed by the processor 100. In addition, the processor 100 can also periodically control the computing units in the target computing unit cluster to update the relevant data of the task in the shared memory 200, and the processor 100 can also periodically obtain the progress of the task execution.

[0073] In actual application, the processor 100 can obtain the command of the computing resource user, and uniformly control the computing units in the target computing unit cluster based on the obtained command. For example, the processor 100 can adjust the parameters of each computing unit to perform a task based on the command of the computing resource user to modify the task parameters, and the processor 100 can also obtain the evaluation index of each computing unit to perform a task based on the command of the computing resource user to view the evaluation index such as the loss value.

[0074] In this way, the processor 100 serves as a unified entrance for executing tasks, eliminating the need for computing resource users to directly access each computing unit, thereby achieving unified management of tasks and unified management of the computing resource pool 40.

[0075] S204: The processor 100 determines whether there is a non-idle computing unit in the target computing unit cluster during the task execution process. If yes, step S205 is executed; if not, step S203 is continued.

[0076] During the task execution process, the computing units in the target computing unit cluster may be unable to continue to execute the tasks assigned by the processor 100 at any time because they are not idle. The processor 100 will continue to monitor the execution of tasks by each computing unit. When a computing unit that is not idle appears, the processor 100 will quickly migrate the task to the computing unit in the backup computing unit cluster for continued execution.

[0077] Specifically, the processor 100 can obtain the idle computing resource information of each computing node in the target cluster, and can determine whether the computing unit is idle according to the idle resource information and adjust the computing resource pool 40 accordingly. For the third computing unit, the third computing unit is a computing unit in the computing resource pool 40 that does not belong to the target computing unit cluster and the backup computing unit cluster. When the third computing unit is not idle, the processor 100 will directly move the third computing unit out of the computing resource pool 40; for the first computing unit, the first computing unit is a computing unit that is not idle in the target computing unit cluster, and the processor 100 will execute step S205 at this time.

[0078] S205: The processor 100 controls the first computing unit to store relevant data in the process of executing the task in the shared memory 200, and controls the second computing unit to continue to execute the task according to the relevant data in the shared memory 200, wherein the first computing unit is a non-idle computing unit in the target computing unit cluster, and the second computing unit is a computing unit in the backup computing unit cluster.

[0079] When the first computing unit that is executing a task cannot continue to execute the task because it is not idle, the processor 100 will first store the relevant data in the process of the first computing unit executing the task in the shared memory 200. For example, when the first computing unit is performing a model training task, the processor 100 will control the first computing unit to save the relevant data of the task, such as the data set, model file, parameter data in the training process, log file, intermediate results, etc., in the shared memory 200. In actual application, the processor 100 can regularly update the relevant data of the task, so that when the first computing unit is not idle, only the relevant data that has not been updated needs to be saved in the shared memory 200, without repeatedly saving the data set, updated log file and other data, further reducing the time consumption of task migration.

[0080] After all relevant data during the execution of the task by the first computing unit is stored in the shared memory 200 , the processor 100 can move the first computing unit out of the computing resource pool 40 and the target computing unit cluster.

[0081] Furthermore, the processor 100 may control the second computing unit to continue to execute the task according to the relevant data in the shared memory 200. Specifically, the computing node to which the second computing unit belongs may load the computing environment required for task execution based on the environment image in step S202, so that the second computing unit can continue to execute the task in the loaded computing environment according to the relevant data stored in the shared memory 200 by the first computing unit.

[0082] When the second computing unit takes over the first computing unit to continue to execute the task, the processor 100 can add the second computing unit to the target computing unit cluster and allocate new computing units to the backup computing unit cluster according to the number of backup computing unit clusters specified in the task requirement information. The newly allocated computing unit can also refer to the method in step S202 to load the computing environment required for task execution, which will not be repeated here.

[0083] In this way, through the shared memory 200, the processor 100 can quickly migrate the task to the second computing unit for continued execution when the first computing unit cannot continue to execute the task, which greatly reduces the time consumption of idle computing resource task migration and ensures the stability and continuity of task execution.

[0084] In this embodiment, the processor 100 can adopt the steps described in the above steps S201 to S205 to continuously allocate target computing unit clusters and backup computing unit clusters for different tasks, and continuously control the execution of tasks and the migration of computing units to ensure that tasks can be executed stably and continuously when using idle computing resources for distributed computing.

[0085] In addition, the present application also provides a distributed computing software, see Figure 3 , Figure 3 A schematic diagram of the architecture of a distributed computing software in an embodiment of the present application is shown. Figure 3 The distributed computing software shown includes:

[0086] The entry layer is the entry point for computing resource users to obtain computing resources, publish task content, access task status and other services. The entry layer includes mobile users and PC (personal computer) users.

[0087] The application layer is used to provide distributed computing services to computing resource users, where the application layer includes user access control panels and public components.

[0088] The user access control panel is used to provide a control interface for distributed computing services to computing resource users. In actual application, the user access control panel is a web user interface (Website User Interface, WUI).

[0089] Common components are used to provide multiple distributed computing services to computing resource users. Common components include a custom scheduling system, computing node data collector, Nvidia Device Plugins, and a distributed image manager.

[0090] A custom scheduling system is used to schedule GPUs to provide computing services for users.

[0091] The computing node data collector is used to collect data information of each computing node, including idle resource information and other task-related data information.

[0092] Nvidia Device Plugins is a Kubernetes component that automatically detects and exposes Nvidia GPUs on compute nodes and adds them to the compute resource pool.

[0093] The distributed image manager is used to manage the environment image of the task and provide the environment image for each computing node, so that the task can be successfully executed in the computing environment loaded by the computing node.

[0094] The K3s scheduling layer is used to schedule idle GPU resources for computing resource users to provide computing services. K3s is a lightweight Kubernetes. The K3s scheduling layer includes K3s Server and K3s Agent.

[0095] K3s Server is responsible for managing and coordinating the execution of tasks by each computing node. K3s Server includes APIServer (Application Programming Interface Service), Kube Proxy (Kubernetes Proxy), Flannel, Etcd, Scheduler (Scheduler) and Controller Mannager (Controller Manager).

[0096] API Server is used to process interaction requests between upper-level computing resource users and computing nodes.

[0097] Kube Proxy is a network proxy in Kubernetes, which is used to expose computing services to upper-level computing resource users and provide network traffic forwarding.

[0098] Flannel is a network plug-in in Kubernetes that is used to create a virtual network in Kubernetes so that computing nodes can communicate with each other.

[0099] Etcd is used to store and manage the status configuration of computing resource pools.

[0100] Scheduler is used to schedule the tasks of computing resource users to appropriate computing nodes.

[0101] Controller Manager is used to maintain and control the status of computing resources in the computing resource pool.

[0102] K3s Agent is an agent program on each computing node, which is used to actually execute tasks and pass idle resource information and other task-related data content to the upper layer. K3s Agent includes Docker (application container engine), Ctr, Kubelet and Docker Compose.

[0103] Docker is used to create and manage containers so that each computing node can load the computing environment required for the task.

[0104] Ctr is a command-line tool for Containerd. It is also used to create and manage containers so that each computing node can load the computing environment required for the task. Containerd and Docker are both carriers for task execution.

[0105] Kubelet is used to communicate with the API Server and maintain and monitor the running status of each task.

[0106] Docker Compose, for running and managing multiple Docker containers.

[0107] The GPU device layer is a computing resource pool used to provide actual computing services to computing resource users. The GPU device layer may include servers in an Internet Data Center (IDC) and terminal devices in various enterprise work areas. Each server and terminal device contains multiple GPU resources.

[0108] The public storage and migration component is used to provide migration services between various computing resources and shared storage services between various computing resources for tasks. The public storage and migration component includes public cloud storage and private cloud storage, where both public cloud storage and private cloud storage can provide shared storage services for various computing resources. Specifically, users of distributed computing software can choose to store task-related data in public cloud storage or private cloud storage. In addition, the common storage and migration component also includes a cross-region backup controller, which is used to provide backup services for K3s scheduling.

[0109] In actual application, the distributed computing software can determine multiple computing units through the computing node data collector and Nvidia DevicePlugins, and the multiple computing units constitute the computing resource pool of the GPU device layer; then the distributed computing software can allocate the target computing unit cluster and the backup computing unit cluster for the task through the custom scheduling system, and configure the computing environment for each computing node in the target computing unit cluster and the backup computing unit cluster through the distributed mirror manager; then, the distributed computing software executes the tasks of the computing resource user through K3s Server and K3s Agent, and can respond to other commands of the computing resource user about the task; finally, when the first computing unit in the target computing unit cluster is not idle, the first computing unit can store the relevant data in the process of executing the task in the private cloud storage or the public cloud storage, so that the second computing unit in the backup computing unit cluster can continue to execute the task according to the relevant data in the private cloud storage or the public cloud storage.

[0110] In addition, the present application embodiment also provides a distributed computing device. Figure 4 , Figure 4 A schematic diagram of the structure of a distributed computing device in an embodiment of the present application is shown. Figure 4 The distributed computing device 400 shown includes:

[0111] A determination module 401 is used to determine a plurality of idle computing units, wherein the computing unit is a graphics processing unit GPU in a computing node, and the computing node is a terminal device or a server; determine a target computing unit cluster and a backup computing unit cluster, wherein the target computing unit cluster and the backup computing unit cluster are both sets consisting of some computing units in the plurality of computing units, and the target computing unit cluster and the backup computing unit cluster do not include the same computing unit;

[0112] A control module 402, wherein the control module 402 is used to control the computing units in the target computing unit cluster to perform tasks; in the process of performing the task, when the first computing unit is not idle, the first computing unit is controlled to store relevant data in the process of performing the task in the shared memory, and the second computing unit is controlled to continue to perform the task according to the relevant data in the shared memory, the first computing unit is a computing unit in the target computing unit cluster, and the second computing unit is a computing unit in the backup computing unit cluster.

[0113] In a possible implementation, the device further includes: a sending module 403, the sending module 403 is used to send the environment image corresponding to the task to the target cluster, the environment image is used for the computing nodes in the target cluster to load the computing environment of the task based on the environment image, and the computing nodes in the target cluster are the computing nodes to which the computing units in the target computing unit cluster belong. The control module 402 is specifically used to: control the computing units in the target computing unit cluster to execute the task in the computing environment.

[0114] In a possible implementation, the sending module 403 is further used to send the environment image to the backup cluster, and the environment image is also used for the computing nodes in the backup cluster to load the computing environment of the task based on the environment image, and the computing nodes in the backup cluster are computing nodes to which the computing units in the backup computing unit cluster belong. The control module 402 is specifically used to control the second computing unit to continue to execute the task in the computing environment of the computing node to which the second computing unit belongs according to the relevant data in the shared memory.

[0115] In a possible implementation, the plurality of computing units constitute a computing resource pool. The determination module 401 is specifically used to: obtain idle computing resource information of a plurality of computing nodes, the idle computing resource information including the number, serial number and device model of the computing units; and determine the computing resource pool according to the idle computing resource information.

[0116] In a possible implementation, the idle computing resource information is specified by the computing node.

[0117] In one possible implementation, the determination module 401 is specifically used to: obtain task requirement information, the task requirement information including the number of computing units required to perform the task, the number of computing units serving as backups, and the required device model, where the required device model is the device model of the computing unit that performs the task; determine the target computing unit cluster and the backup computing unit cluster based on the task requirement information.

[0118] In one possible embodiment, the device also includes: an adjustment module 404, which is used to obtain task requirement modification information, and the task requirement modification information includes increasing or decreasing the number of computing units required to perform the task, increasing or decreasing the number of computing units used as backups and the required equipment model; adjusting the target computing unit cluster and the backup computing unit cluster according to the task requirement modification information.

[0119] In a possible implementation, the device also includes: a removal module 405, which is used to move the first computing unit out of the computing resource pool when the first computing unit is not idle; or, when the third computing unit is not idle, move the third computing unit out of the computing resource pool, the third computing unit being a computing unit in the computing resource pool that does not belong to the target computing unit cluster and the backup computing unit cluster.

[0120] It should be noted that the information interaction, execution process, etc. between the modules and units of the above-mentioned device are based on the same concept as the method embodiment in the embodiments of the present application, and the technical effects they bring are the same as those of the method embodiment in the embodiments of the present application. For specific contents, please refer to the description in the method embodiment shown in the aforementioned embodiments of the present application, and will not be repeated here.

[0121] In addition, the present application embodiment also provides a computing device. Figure 5 , Figure 5 FIG. 1 is a schematic diagram showing the hardware structure of a computing device in an embodiment of the present application. Figure 5 As shown, computing device 500 may include a processor 501 and a memory 502 .

[0122] Wherein, the memory 502 is used to store computer programs;

[0123] The processor 501 is used to execute the distributed computing method described in the above method embodiment according to the computer program.

[0124] In addition, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the distributed computing method described in the above method embodiment.

[0125] The "first" in the names of the "first computing unit" and the like mentioned in the embodiments of the present application is only used as a name identifier and does not represent the first in order. The same rule applies to "second", "third", etc.

[0126] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0127] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, in which the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0128] The above description is merely an exemplary embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A distributed computing method, characterized in that: The method is applied to a service platform, the service platform includes a shared memory, and the method includes: Determine a plurality of idle computing units, where the computing units are graphics processing units (GPUs) in computing nodes, and the computing nodes are terminal devices or servers; Determine a target computing unit cluster and a backup computing unit cluster, wherein the target computing unit cluster and the backup computing unit cluster are both sets consisting of some computing units in the plurality of computing units, and the target computing unit cluster and the backup computing unit cluster do not include the same computing unit; Controlling computing units in the target computing unit cluster to execute tasks; In the process of executing the task, when the first computing unit is not idle, the first computing unit is controlled to store relevant data in the process of executing the task in the shared memory, and the second computing unit is controlled to continue to execute the task according to the relevant data in the shared memory. The first computing unit is a computing unit in the target computing unit cluster, and the second computing unit is a computing unit in the backup computing unit cluster.

2. The method according to claim 1, characterized in that The method further comprises: Sending an environment image corresponding to the task to a target cluster, wherein the environment image is used for computing nodes in the target cluster to load a computing environment for the task based on the environment image, and the computing nodes in the target cluster are computing nodes to which computing units in the target computing unit cluster belong; The controlling the computing units in the target computing unit cluster to execute tasks includes: The computing units in the target computing unit cluster are controlled to execute the task in the computing environment.

3. The method according to claim 2, characterized in that The method further comprises: Sending the environment image to a backup cluster, the environment image is also used for computing nodes in the backup cluster to load the computing environment of the task based on the environment image, the computing nodes in the backup cluster are computing nodes to which computing units in the backup computing unit cluster belong; The controlling the second computing unit to continue to execute the task according to the relevant data in the shared memory comprises: The second computing unit is controlled to continue to execute the task according to the relevant data in the shared memory in the computing environment of the computing node to which the second computing unit belongs.

4. The method according to claim 1, characterized in that: The multiple computing units constitute a computing resource pool; The step of determining the idle computing units comprises: Obtaining idle computing resource information of multiple computing nodes, wherein the idle computing resource information includes the number, serial number and device model of computing units; The computing resource pool is determined according to the idle computing resource information.

5. The method according to claim 4, characterized in that The idle computing resource information is specified by the computing node.

6. The method according to claim 4, characterized in that The step of determining the target computing unit cluster and the backup computing unit cluster includes: Acquire task requirement information, where the task requirement information includes the number of computing units required to perform the task, the number of computing units used as backups, and a required device model, where the required device model is a device model of a computing unit that performs the task; The target computing unit cluster and the backup computing unit cluster are determined according to the task requirement information.

7. The method according to claim 6, characterized in that The method further comprises: Acquire task requirement modification information, wherein the task requirement modification information includes increasing or decreasing the number of computing units required to perform the task, increasing or decreasing the number of computing units used as backups, and the required device model; The target computing unit cluster and the backup computing unit cluster are adjusted according to the task requirement modification information.

8. The method according to claim 4, characterized in that The method further comprises: When the first computing unit is not idle, moving the first computing unit out of the computing resource pool; Alternatively, when the third computing unit is not idle, the third computing unit is moved out of the computing resource pool, and the third computing unit is a computing unit in the computing resource pool that does not belong to the target computing unit cluster and the backup computing unit cluster.

9. A distributed computing device, characterized in that: The device comprises: A determination module, the determination module is used to determine multiple idle computing units, the computing unit is a graphics processing unit GPU in a computing node, and the computing node is a terminal device or a server; determine a target computing unit cluster and a backup computing unit cluster, the target computing unit cluster and the backup computing unit cluster are both sets composed of some computing units in the multiple computing units, and the target computing unit cluster and the backup computing unit cluster do not include the same computing unit; A control module, wherein the control module is used to control the computing units in the target computing unit cluster to perform tasks; in the process of performing the task, when the first computing unit is not idle, the first computing unit is controlled to store relevant data in the process of performing the task in the shared memory, and the second computing unit is controlled to continue to perform the task according to the relevant data in the shared memory, the first computing unit is a computing unit in the target computing unit cluster, and the second computing unit is a computing unit in the backup computing unit cluster.

10. A computing device, characterized in that: The computing device comprises a processor and a memory: The memory is used to store computer programs; The processor is configured to execute the method of any one of claims 1 to 8 according to the computer program.

Images