Display card distribution method and device for distributed computing cluster and computing equipment

Abstract

The invention discloses a graphics card distribution method and device for a distributed computing cluster and computing equipment. The method comprises the steps that an execution process is created for physical nodes in the distributed computing cluster; creating a thread for the execution process; wherein the threads are in one-to-one correspondence with the graphics cards on the physical nodes; and calling an external program through the thread, wherein the external program processes the partition data corresponding to the task scheduled to the execution process by calling a display card corresponding to the thread. One execution process is created for each physical node, the threads in one-to-one correspondence with the graphics cards are created, and each thread calls the external program to execute the task, so that all the graphics cards of the physical node can be fully utilized, the utilization rate of the graphics cards in the distributed computing cluster is effectively increased, and the task utilization rate is increased. According to the method, reasonable and effective distribution of graphics card resources can be achieved when the complex cluster calls the external program, meanwhile, the whole process is highly automatic, the workload of a user is reduced, and high expandability is achieved.

Description

Technical Field

[0001] The present application relates to the field of distributed computing technology, and in particular to a method, apparatus, computing device, computer storage medium and computer program product for allocating graphics cards in a distributed computing cluster. Background Art

[0002] In the scenario where a distributed computing cluster calls an external independent program, the external independent program needs to call the graphics card in the physical node to process data, so it is necessary to configure the allocation of graphics card resources. In the prior art, the allocation of graphics card resources is usually achieved through manual configuration or based on a static allocation strategy.

[0003] The inventors of this application have found that the existing technology has at least the following deficiencies: the manual configuration method is complicated to operate and has high labor costs; the method based on static allocation strategy usually requires pre-setting parameters, such as the number of graphics cards required for each task, and cannot adapt to the situation when the dynamic task load changes or the graphics card resources of each physical node in the cluster are inconsistent. In short, it is difficult to achieve accurate graphics card resource allocation with the existing technology, and graphics card resources cannot be fully utilized. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide a method, apparatus, computing device, computer storage medium and computer program product that overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the present application, a method for allocating graphics cards in a distributed computing cluster is provided, comprising:

[0006] Create execution processes for physical nodes in a distributed computing cluster;

[0007] Create threads for executing processes; threads correspond one-to-one to graphics cards on physical nodes;

[0008] The external program is called through the thread, and the external program processes the partition data corresponding to the task scheduled to the execution process through the graphics card corresponding to the calling thread.

[0009] Optionally, before calling the external program through the thread, the method further includes:

[0010] When it is detected that a graphics card is in an idle state, it is determined whether there is any unprocessed data in the partition data;

[0011] Calling an external program through a thread, wherein the external program processes the partition data corresponding to the task scheduled to the execution process through the graphics card corresponding to the calling thread further includes:

[0012] If there is still unprocessed data, the thread corresponding to the graphics card in the idle state calls the external program, so that the external program calls the graphics card in the idle state to process the unprocessed data.

[0013] Optionally, the method further comprises:

[0014] Record the index of each graphics card on the physical node in the specified list;

[0015] Before calling the external program through the thread, the method further includes:

[0016] Polls the specified list to determine which graphics cards are idle.

[0017] Optionally, calling an external program through a thread, and the external program processing the partition data corresponding to the task scheduled to the execution process through a graphics card corresponding to the calling thread further includes:

[0018] The partition data, external program and graphics card index corresponding to the task are passed to the wrapper function of the thread so that the thread executes the wrapper function.

[0019] Optionally, creating a thread for executing the process further includes:

[0020] Create a corresponding number of threads according to the number of graphics cards in the physical node;

[0021] The method also includes: establishing a one-to-one correspondence between threads and graphics cards.

[0022] Optionally, the method further comprises:

[0023] Partition the data to be processed to obtain data of each partition;

[0024] Each task is obtained according to the data division of each partition, and each task is scheduled.

[0025] According to another aspect of the present application, a graphics card allocation device for a distributed computing cluster is provided, comprising:

[0026] A first creation module is used to create an execution process for a physical node in a distributed computing cluster;

[0027] A second creation module is used to create threads for executing processes; wherein threads correspond one to one with graphics cards on physical nodes;

[0028] The processing module is used to call an external program through a thread, and the external program processes the partition data corresponding to the task scheduled to the execution process through the graphics card corresponding to the calling thread.

[0029] Optionally, the device further comprises:

[0030] A judgment module, used for judging whether there is any unprocessed data in the partition data when it is detected that a graphics card is in an idle state before calling an external program through a thread;

[0031] The processing module is further used for: if there is unprocessed data, calling an external program through the thread corresponding to the graphics card in the idle state, so that the external program calls the graphics card in the idle state to process the unprocessed data.

[0032] Optionally, the device further comprises:

[0033] A recording module is used to record the index of each graphics card on the physical node in a specified list;

[0034] The polling module is used to poll the specified list to determine the graphics cards that are in an idle state before calling an external program through a thread.

[0035] Optionally, the processing module is further used to: pass the partition data corresponding to the task, the external program and the index of the graphics card to the wrapper function of the thread, so that the thread executes the wrapper function.

[0036] Optionally, the second creation module is further used for:

[0037] Create a corresponding number of threads according to the number of graphics cards in the physical node; establish a one-to-one correspondence between threads and graphics cards.

[0038] Optionally, the device further comprises:

[0039] A partitioning module is used to partition the data to be processed to obtain data of each partition;

[0040] The scheduling module is used to obtain various tasks according to the data of each partition and schedule each task.

[0041] According to another aspect of the present application, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0042] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned graphics card allocation method for the distributed computing cluster.

[0043] According to another aspect of the present application, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned graphics card allocation method for a distributed computing cluster.

[0044] According to another aspect of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the above-mentioned method for allocating graphics cards in a distributed computing cluster.

[0045] According to the graphics card allocation method, device, computing equipment, computer storage medium and computer program product of the distributed computing cluster provided by the embodiment of the present application, first, an execution process is created for the physical nodes in the distributed computing cluster; secondly, a thread is created for the execution process; wherein the thread corresponds one-to-one to the graphics card on the physical node; then the external program is called through the thread, and the external program processes the partition data corresponding to the task scheduled to the execution process by calling the graphics card corresponding to the thread. By creating an execution process for each physical node and creating a thread corresponding one-to-one to the graphics card, each thread calls the external program to execute the task, thereby ensuring that all graphics cards of the physical node can be fully utilized, effectively improving the utilization rate of the graphics card in the distributed computing cluster, and being able to realize the reasonable and effective allocation of graphics card resources when a complex cluster calls an external program. At the same time, the whole process is highly automated, reducing the user's workload, and has high scalability.

[0046] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0048] Figure 1 A flowchart of a method for allocating graphics cards in a distributed computing cluster provided by an embodiment of the present application is shown;

[0049] Figure 2 A schematic diagram showing a system architecture provided by another embodiment of the present application is shown;

[0050] Figure 3 A functional structure diagram of a graphics card allocation device for a distributed computing cluster provided in an embodiment of the present application is shown;

[0051] Figure 4 A schematic diagram of the structure of a computing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0053] First, the terms involved in one or more embodiments of the present application are explained.

[0054] Apache Spark framework: is an open source distributed computing framework for large-scale data processing. It provides high-performance data processing capabilities and supports multiple data processing modes, including batch processing, stream processing, machine learning, and graph processing.

[0055] RDD: Resilient Distributed Dataset is the most basic data abstraction in Apache Spark. It is an immutable, distributed data set. RDD provides a powerful abstraction to process large-scale data sets while having fault tolerance and parallel computing capabilities.

[0056] Partition data: It is the logical division unit of data and the basic unit of data processing. An RDD consists of multiple partition data, and each partition data can be calculated independently.

[0057] Executor: Executor is a working node in the Apache Spark runtime environment, responsible for executing tasks and managing memory. Each Executor is a JVM process that can run multiple tasks.

[0058] Task: The smallest data processing unit in Apache Spark, which is executed in parallel by Executor on each node. Apache Spark divides partitioned data into several tasks, each of which contains one or more partitioned data. Task scheduling creates and assigns tasks based on the number and dependencies of data partitions.

[0059] Figure 1 A flowchart of a method for allocating graphics cards in a distributed computing cluster provided by an embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps:

[0060] Step S110, creating an execution process for a physical node in the distributed computing cluster.

[0061] The distributed computing cluster contains multiple physical nodes, each of which has multiple graphics cards (GPUs). For each physical node, a corresponding execution process (executor) is created.

[0062] Step S120, creating a thread for the execution process.

[0063] A corresponding thread pool is created for each execution process. The number of threads contained in the thread pool is consistent with the number of graphics cards on the physical node, and the threads correspond to the graphics cards on the physical node one by one.

[0064] Step S130, calling an external program through a thread, and the external program processes the partition data corresponding to the task scheduled to the execution process through the graphics card corresponding to the calling thread.

[0065] Among them, a task is the smallest data processing unit in Apache Spark. Each task is used to process one or more partitioned data. The threads in the execution process are used to execute the tasks scheduled to the execution process. The process of executing a task is the process in which the thread calls an external program to process the partitioned data corresponding to the task. The external program specifically calls the corresponding graphics card resources to process the partitioned data.

[0066] In specific implementation, the data to be processed is partitioned to obtain partition data. The data to be processed is the data that needs to be calculated by a distributed computing cluster, usually large-scale data. Then, each task is obtained according to the partition data, and each task is scheduled to distribute all tasks to various execution processes.

[0067] In actual distributed computing clusters, different physical nodes often have different numbers of graphics cards due to different purchase batches and different machine configurations. In the method based on static allocation strategy, each physical node is required to have the same number of graphics cards, otherwise it is difficult to achieve accurate graphics card allocation. Therefore, the operation is performed according to the configuration with the minimum number of graphics cards, resulting in the inability to fully utilize the graphics card resources in the cluster, causing a waste of graphics card resources. In addition, this method cannot meet computing needs under high load conditions, and the scope of application of this method is also small.

[0068] According to the graphics card allocation method of the distributed computing cluster provided in the embodiment of the present application, by creating an execution process for each physical node and creating a thread corresponding to the graphics card one by one, each thread calls an external program to execute a task, thereby ensuring that all graphics cards of the physical node can be fully utilized, effectively improving the utilization rate of graphics cards in the distributed computing cluster, and being able to achieve reasonable and effective allocation of graphics card resources when complex clusters call external programs. At the same time, the whole process is highly automated, and the user does not need to fill in any parameters, which reduces the user's workload and has high scalability. The method of the embodiment of the present application is particularly suitable for scenarios that require large-scale parallel computing, such as for processing large-scale geophysical data, thereby providing an efficient solution for data processing in fields such as seismic exploration.

[0069] In an optional manner, the graphics card is configured to process the next piece of data after processing the current data. Specifically, when it is detected that a graphics card is in an idle state, it is determined whether there is any unprocessed data in the partition data, and the unprocessed data is determined by polling the partition data. Then, the external program is called through the thread, and the external program processes the partition data corresponding to the task scheduled to the execution process through the graphics card corresponding to the calling thread. Further comprising: if there is any unprocessed data, the external program is called through the thread corresponding to the graphics card in the idle state, so that the external program calls the graphics card in the idle state to process the unprocessed data.

[0070] For example, there are 4 graphics cards on a physical node. After starting data processing, a piece of data is first allocated to each graphics card. When a graphics card completes processing of the current data, it is in an idle state. At this time, the next piece of data is allocated to the graphics card, and the graphics card is used to process the next piece of data until the partition data corresponding to all tasks scheduled to the execution process are processed, indicating that the tasks scheduled to the execution process are completed. When all tasks corresponding to the execution process are completed, it indicates that the processing of the data to be processed has been completed.

[0071] If there is no unprocessed data, it means that the task scheduled to the execution process has been completed, and the execution process feeds back a notification of task completion to the master node of the distributed computing cluster.

[0072] In an optional manner, the index of each graphics card on the physical node is recorded in a specified list, the specified list is polled to determine the graphics card in the idle state, and the state of the corresponding graphics card is queried through the index to determine the graphics card in the idle state.

[0073] In an optional manner, a specific implementation method of creating threads for executing a process is: creating a corresponding number of threads according to the number of graphics cards in a physical node, and then establishing a one-to-one correspondence between threads and graphics cards. That is, firstly creating a corresponding number of threads according to the number of graphics cards in a physical node, and binding the threads to the graphics cards one-to-one.

[0074] In an optional manner, the specific implementation method of a thread executing a task is: passing the partition data, external program and graphics card index corresponding to the task to the thread's wrapper function, so that the thread can execute the wrapper function. By executing the wrapper function, the thread can call the corresponding external program and pass the index of the corresponding graphics card to the external program, so that the external program can call the corresponding graphics card through the index to process the corresponding partition data.

[0075] Figure 2 A schematic diagram of a system architecture provided by another embodiment of the present application is shown, the system includes a master node 21 and multiple physical nodes, an execution process is created in each physical node, a first execution process 22 is created for the first physical node, the first task 1 is scheduled to be executed by the first execution process 22, the first execution process 22 has a first thread pool 23 containing multiple threads, each thread in the first thread pool 23 corresponds to a first graphics card 231. Similarly, a second execution process 24 is created for the second physical node, the second task 2 is scheduled to be executed by the second execution process 24, the second execution process 24 has a second thread pool 25 containing multiple threads, each thread in the second thread pool 25 corresponds to a second graphics card 251. The execution process in each physical node will feed back the execution result of the task to the master node 21. The structures corresponding to other physical nodes in the distributed computing cluster are similar and will not be repeated here.

[0076] Another embodiment of the present application provides a method for allocating graphics cards in a distributed computing cluster. The method is applied to an Apache Spark distributed computing cluster. The method includes the following steps:

[0077] In step S310, the master node reads the seismic data as data to be processed, creates a resilient distributed data set, and expands the resilient distributed data set so as to process the data of each partition within the mappartition.

[0078] Step S320, create an Executor on each physical node, specifically through Yarn, and the Executor executes the RDD.mappartiton method to convert the data in each partition to be processed locally.

[0079] The processing steps of tasks within a partition are as follows:

[0080] Step S3301, for each Executor, obtain the number and index of graphics cards of the physical node where it is located, and the number of graphics cards is recorded as NGpuS;

[0081] Specifically, the number of graphics cards is obtained in the mappartition method of the RDD created by Apache Spark, so as to ensure that the different and real number of graphics cards on each physical node can be obtained;

[0082] Step S3302, create a thread pool, the thread pool has NGpuS threads;

[0083] Step S3303, passing the partition data, external program, and graphics card index corresponding to the task to the wrapper function of the thread to start calculation;

[0084] Step S3304: poll all data and graphics card index lists of the partition in the thread pool to determine the idle graphics card and the next data to be processed. After the graphics card processes one data, it processes the next data until all data of the partition is processed.

[0085] When an Executor processes all partitioned data, the tasks scheduled to the Executor are considered completed. When all Executors complete the scheduled tasks, the processing of the seismic data is considered completed.

[0086] In summary, according to the graphics card allocation method for a distributed computing cluster provided in this embodiment, a method for dynamically allocating graphics card resources when calling an external program that requires a graphics card in a Spark distributed computing environment is provided. By utilizing Spark's distributed computing capabilities, the repartitioning of RDD is no longer considered. Instead, the number of graphics cards in the physical node is obtained in the mappartition method of RDD, and NGpuS threads are created for each Executor, thereby ensuring that all graphics cards in the physical node where the Executor is located are fully utilized. This achieves reasonable and effective allocation of graphics card resources when a complex cluster calls external programs, and realizes intelligent allocation of graphics card resources under real-time task loads. This solves the challenge of graphics card resource allocation when calling external programs in a traditional distributed computing environment, especially the graphics card allocation problem when there are multiple graphics cards on each physical node in the cluster and the number of graphics cards is inconsistent, and can achieve efficient data processing and task scheduling.

[0087] Figure 3 The functional structure diagram of the graphics card allocation device of the distributed computing cluster provided by the embodiment of the present application is shown. Figure 3 As shown, the device comprises:

[0088] A first creation module 41, used to create an execution process for a physical node in a distributed computing cluster;

[0089] A second creation module 42 is used to create threads for executing processes; wherein threads correspond one to one with graphics cards on physical nodes;

[0090] The processing module 43 is used to call an external program through a thread, and the external program processes the partition data corresponding to the task scheduled to the execution process through the graphics card corresponding to the calling thread.

[0091] In an optional manner, the device further comprises:

[0092] A judgment module, used for judging whether there is any unprocessed data in the partition data when it is detected that a graphics card is in an idle state before calling an external program through a thread;

[0093] The processing module 43 is further used for: if there is unprocessed data, calling an external program through the thread corresponding to the graphics card in the idle state, so that the external program calls the graphics card in the idle state to process the unprocessed data.

[0094] In an optional manner, the device further comprises:

[0095] A recording module is used to record the index of each graphics card on the physical node in a specified list;

[0096] The polling module is used to poll the specified list to determine the graphics cards that are in an idle state before calling an external program through a thread.

[0097] In an optional manner, the processing module 43 is further used to: pass the partition data corresponding to the task, the external program and the index of the graphics card to the wrapper function of the thread, so that the thread executes the wrapper function.

[0098] In an optional manner, the second creation module 42 is further used for:

[0099] Create a corresponding number of threads according to the number of graphics cards in the physical node; establish a one-to-one correspondence between threads and graphics cards.

[0100] In an optional manner, the device further comprises:

[0101] A partitioning module is used to partition the data to be processed to obtain data of each partition;

[0102] The scheduling module is used to obtain various tasks according to the data of each partition and schedule each task.

[0103] To summarize, according to the graphics card allocation device for a distributed computing cluster provided in this embodiment, an execution process is created for each physical node, and threads corresponding to graphics cards are created one by one. Each thread calls an external program to execute a task, thereby ensuring that all graphics cards of the physical node can be fully utilized. This effectively improves the utilization rate of graphics cards in the distributed computing cluster, and can achieve reasonable and effective allocation of graphics card resources when a complex cluster calls an external program. At the same time, the entire process is highly automated, and the user does not need to fill in any parameters, which reduces the user's workload and has high scalability.

[0104] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program, which enables a processor to perform operations corresponding to the graphics card allocation method for a distributed computing cluster in any of the above method embodiments.

[0105] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, which can enable a processor to perform operations corresponding to the graphics card allocation method for a distributed computing cluster in any of the above method embodiments.

[0106] Figure 4 A schematic diagram of the structure of an embodiment of a computing device of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0107] like Figure 4 As shown, the computing device may include: a processor (processor) 502 , a communications interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0108] The processor 502, the communication interface 504, and the memory 506 communicate with each other via the communication bus 508. The communication interface 504 is used to communicate with other devices such as a client or other server network elements. The processor 502 is used to execute the program 510, which can specifically execute the relevant steps in the above-mentioned embodiment of the graphics card allocation method for a distributed computing cluster of computing devices.

[0109] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0110] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0111] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0112] The program 510 can be specifically used to enable the processor 502 to execute the graphics card allocation method of the distributed computing cluster in any of the above-mentioned method embodiments. The specific implementation of each step in the program 510 can refer to the corresponding descriptions in the corresponding steps and units in the graphics card allocation embodiment of the distributed computing cluster, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the aforementioned method embodiment, which will not be repeated here.

[0113] The algorithm or display provided here are not inherently related to any specific computer, virtual system or other equipment. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application.

[0114] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0115] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present application.

[0116] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0117] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0118] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0119] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A method for allocating graphics cards in a distributed computing cluster, characterized in that: include: Create execution processes for physical nodes in a distributed computing cluster; Creating a thread for the execution process; wherein the thread corresponds one-to-one to the graphics card on the physical node; An external program is called through the thread, and the external program processes the partition data corresponding to the task scheduled to the execution process by calling the graphics card corresponding to the thread.

2. The method according to claim 1, characterized in that Before calling the external program through the thread, the method further comprises: When it is detected that a graphics card is in an idle state, determining whether there is any unprocessed data in the partition data; The calling of an external program through the thread, wherein the external program processes the partition data corresponding to the task scheduled to the execution process by calling the graphics card corresponding to the thread, further comprises: If there is still unprocessed data, the thread corresponding to the graphics card in the idle state calls an external program, so that the external program calls the graphics card in the idle state to process the unprocessed data.

3. The method according to claim 2, characterized in that The method further comprises: Record the index of each graphics card on the physical node in a specified list; Before calling the external program through the thread, the method further comprises: The specified list is polled to determine graphics cards that are in an idle state.

4. The method according to any one of claims 1 to 3, characterized in that The calling of an external program through the thread, wherein the external program processes the partition data corresponding to the task scheduled to the execution process by calling the graphics card corresponding to the thread, further comprises: The partition data, the external program and the index of the graphics card corresponding to the task are passed to the wrapper function of the thread, so that the thread executes the wrapper function.

5. The method according to claim 1, characterized in that The creating a thread for the execution process further comprises: Creating a corresponding number of threads according to the number of graphics cards in the physical node; The method further includes: establishing a one-to-one correspondence between threads and graphics cards.

6. The method according to claim 1, characterized in that The method further comprises: Partition the data to be processed to obtain data of each partition; Various tasks are obtained according to the partition data, and each task is scheduled.

7. A graphics card allocation device for a distributed computing cluster, characterized in that: include: A first creation module is used to create an execution process for a physical node in a distributed computing cluster; A second creation module, configured to create threads for the execution process; wherein the threads correspond one-to-one to the graphics cards on the physical nodes; The processing module is used to call an external program through the thread, and the external program processes the partition data corresponding to the task scheduled to the execution process by calling the graphics card corresponding to the thread.

8. A computing device, characterized in that include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the graphics card allocation method for a distributed computing cluster according to any one of claims 1-6.

9. A computer storage medium, characterized in that The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the graphics card allocation method for a distributed computing cluster according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the method for allocating graphics cards of a distributed computing cluster as described in any one of claims 1 to 6.

Images