Multi-accelerator card scheduling method and system for model acceleration training

Abstract

The invention relates to the field of artificial intelligence, and discloses a multi-acceleration card scheduling method and system for model acceleration training, and the method comprises the steps: obtaining a first demand computing power according to the data processing speed of a model training demand and a set computing power use efficiency, according to the data processing speed of the current model training and the data processing speed corresponding to the first demand computing power, obtaining the first computing power use efficiency of the current model; according to the first computing power use efficiency of the current model and the data processing speed of the model training demand, obtaining a second demand computing power, and forming a second accelerator card sequence; and completing accelerator card scheduling according to the second accelerator card sequence. According to the method, the computing power use efficiency is evaluated in real time, and data support is provided for optimal configuration of computing power resources.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a multi-acceleration card scheduling method and system for model acceleration training. Background Art

[0002] In the field of deep learning, as the complexity of models continues to increase, the demand for computing resources is also growing. In order to accelerate the model training process, multiple accelerator cards (such as GPUs, FPGAs, etc.) are usually used for parallel computing. However, how to efficiently and dynamically schedule these accelerator card resources to maximize the efficiency of computing power while ensuring the stability and reliability of the training process is a major challenge currently faced.

[0003] Traditional accelerator card scheduling methods are often based on static configuration, that is, before training begins, a certain number of accelerator cards are manually allocated based on the expected demand of the model. This method has several obvious problems:

[0004] Waste of computing power: Since the actual computing power requirements during model training may vary with the training stage, static configuration often leads to excess computing power resources in some stages and insufficient computing power in other stages, resulting in waste of computing power.

[0005] Inflexible scheduling: Once the accelerator card is allocated, it is difficult to dynamically adjust according to real-time needs during training, which lacks flexibility.

[0006] Low fault handling efficiency: When an accelerator card fails, traditional scheduling methods usually require manual intervention to troubleshoot and repair the fault, affecting training efficiency.

[0007] The efficiency of computing power usage is difficult to evaluate: There is a lack of effective mechanisms to evaluate and adjust the efficiency of computing power usage in real time, which leads to increased training costs. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multi-acceleration card scheduling method for model acceleration training, comprising the following steps:

[0009] Step 1: According to the data processing speed required for model training and the set computing power utilization efficiency, a first required computing power is obtained; the accelerator card scheduling module generates a computing power container according to the first required computing power; the first computing power container schedules the corresponding accelerator card according to the first required computing power, and generates a first accelerator card sequence; the accelerator card control module performs a linkage test based on the first accelerator card sequence and the target model to obtain the data processing speed of the current model training; according to the data processing speed of the current model training and the data processing speed corresponding to the first required computing power, the first computing power utilization efficiency of the current model is obtained;

[0010] Step 2: According to the utilization efficiency of the first computing power of the current model and the data processing speed required for model training, a second required computing power is obtained; and according to the computing power difference between the second required computing power and the first required computing power, an accelerator card corresponding to the computing power difference is scheduled to form a second accelerator card sequence with the first accelerator card sequence;

[0011] Step 3: The accelerator card control module performs a linkage test according to the second accelerator card sequence and the target model to obtain the data processing speed of the current model training, and obtains the second computing power usage efficiency of the current model training according to the data processing speed of the current model training and the second required computing power. If the first computing power usage efficiency is consistent with the second computing power usage efficiency, proceed to step 7; otherwise, proceed to step 4.

[0012] Step 4: judging the type of accelerator card failure according to the ratio of the total power of the accelerator cards in the first accelerator card sequence to the total power of the accelerator cards in the second accelerator card sequence; if it is a hardware failure, proceeding to step 5; if it is a software failure, proceeding to step 6;

[0013] Step 5: Obtain the power of each accelerator card in the second accelerator card sequence, and obtain the difference between the power of each accelerator card and the design power. The accelerator card with a difference greater than a set difference threshold is a faulty accelerator card. Replace the accelerator card according to the faulty accelerator card information, and return to step 3.

[0014] Step 6: According to the second accelerator card sequence and the first computing power usage efficiency, the target data processing rate of each accelerator card is obtained respectively, and the data processing rate difference between the data processing rate of each accelerator card and the corresponding target data processing rate is obtained respectively. If the data processing rate difference is greater than the set data processing rate difference threshold, it is a faulty accelerator card. According to the faulty accelerator card information, the faulty accelerator is repaired by software, and the process returns to step 3;

[0015] Step seven: completing the accelerator card scheduling according to the second accelerator card sequence.

[0016] Furthermore, the first required computing power is obtained according to the data processing speed required for model training, including:

[0017] The corresponding computing power is obtained according to the data processing speed required for model training, and the first required computing power is obtained according to the ratio of the corresponding computing power to the set computing power utilization efficiency.

[0018] Furthermore, obtaining the first computing power utilization efficiency of the current model according to the data processing speed of the current model training and the first required computing power includes:

[0019] According to the ratio of the data processing speed of the current model training to the data processing speed corresponding to the first required computing power, the first computing power utilization efficiency of the current model is obtained using the following formula:

[0020]

[0021] Furthermore, the second required computing power is obtained according to the first required computing power usage efficiency of the current model and the set computing power usage efficiency, including:

[0022]

[0023] The second required computing power is obtained according to the data processing speed corresponding to the second required computing power.

[0024] Furthermore, the second computing power utilization efficiency of the current model training is obtained according to the data processing speed of the current model training and the second required computing power, including:

[0025]

[0026] Furthermore, judging the type of accelerator card failure according to the ratio of the total power of accelerator cards in the first accelerator card sequence to the total power of accelerator cards in the second accelerator card sequence includes:

[0027] If the ratio of the total power of the accelerator cards in the first accelerator card sequence to the total power of the accelerator cards in the second accelerator card sequence and the ratio of the number of accelerator cards in the first accelerator card sequence to the number of accelerator cards in the second accelerator card sequence are consistent, it is a software failure; otherwise, it is a hardware failure.

[0028] Furthermore, obtaining the target data processing rate of each accelerator card according to the second accelerator card sequence and the first computing power utilization efficiency includes:

[0029] According to the data processing speed corresponding to the computing power of each accelerator card in the second accelerator card sequence and the first computing power utilization efficiency, the target data processing rate of each accelerator card is obtained.

[0030] A multi-accelerator card scheduling system for model accelerated training, applying the multi-accelerator card scheduling method for model accelerated training, comprises a cloud data server, an accelerator card scheduling module, an accelerator card control module, a data acquisition module, a communication module and a data processing module; the accelerator card scheduling module, the accelerator card control module, the data acquisition module and the communication module are respectively connected to the data processing module; the cloud data server is communicatively connected to the communication module.

[0031] The beneficial effects of the present invention are: by dynamically adjusting the allocation of accelerator cards, it is ensured that computing power resources always match the actual needs of model training, thereby avoiding waste of computing power. The computing power utilization efficiency is evaluated in real time, providing data support for the optimal configuration of computing power resources.

[0032] The system can dynamically adjust the number and configuration of accelerator cards according to the real-time needs of the model training process, which improves the flexibility of the system. It supports adding or removing accelerator cards at any time during the training process, which meets the needs of model training of different scales.

[0033] Automatically detect hardware and software faults of accelerator cards, reducing the need for manual intervention and improving troubleshooting efficiency. Through power ratio analysis and data processing rate comparison, faulty accelerator cards can be quickly located, shortening troubleshooting time. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a multi-acceleration card scheduling method for model acceleration training is shown;

[0035] Figure 2 This is a schematic diagram of the principle of a multi-acceleration card scheduling system for model acceleration training. DETAILED DESCRIPTION

[0036] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0037] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0038] like Figure 1 As shown in FIG. 1 , a multi-acceleration card scheduling method for model acceleration training includes the following steps:

[0039] Step 1: According to the data processing speed required for model training and the set computing power utilization efficiency, a first required computing power is obtained; the accelerator card scheduling module generates a computing power container according to the first required computing power; the first computing power container schedules the corresponding accelerator card according to the first required computing power, and generates a first accelerator card sequence; the accelerator card control module performs a linkage test based on the first accelerator card sequence and the target model to obtain the data processing speed of the current model training; according to the data processing speed of the current model training and the data processing speed corresponding to the first required computing power, the first computing power utilization efficiency of the current model is obtained;

[0040] Step 2: According to the utilization efficiency of the first computing power of the current model and the data processing speed required for model training, a second required computing power is obtained; and according to the computing power difference between the second required computing power and the first required computing power, an accelerator card corresponding to the computing power difference is scheduled to form a second accelerator card sequence with the first accelerator card sequence;

[0041] Step 3: The accelerator card control module performs a linkage test according to the second accelerator card sequence and the target model to obtain the data processing speed of the current model training, and obtains the second computing power usage efficiency of the current model training according to the data processing speed of the current model training and the second required computing power. If the first computing power usage efficiency is consistent with the second computing power usage efficiency, proceed to step 7; otherwise, proceed to step 4.

[0042] Step 4: judging the type of accelerator card failure according to the ratio of the total power of the accelerator cards in the first accelerator card sequence to the total power of the accelerator cards in the second accelerator card sequence; if it is a hardware failure, proceeding to step 5; if it is a software failure, proceeding to step 6;

[0043] Step 5: Obtain the power of each accelerator card in the second accelerator card sequence, and obtain the difference between the power of each accelerator card and the design power. The accelerator card with a difference greater than a set difference threshold is a faulty accelerator card. Replace the accelerator card according to the faulty accelerator card information, and return to step 3.

[0044] Step 6: According to the second accelerator card sequence and the first computing power usage efficiency, the target data processing rate of each accelerator card is obtained respectively, and the data processing rate difference between the data processing rate of each accelerator card and the corresponding target data processing rate is obtained respectively. If the data processing rate difference is greater than the set data processing rate difference threshold, it is a faulty accelerator card. According to the faulty accelerator card information, the faulty accelerator is repaired by software, and the process returns to step 3;

[0045] Step seven: completing the accelerator card scheduling according to the second accelerator card sequence.

[0046] The first required computing power is obtained according to the data processing speed required for model training, including:

[0047] The corresponding computing power is obtained according to the data processing speed required for model training, and the first required computing power is obtained according to the ratio of the corresponding computing power to the set computing power utilization efficiency.

[0048] The obtaining of the first computing power utilization efficiency of the current model according to the data processing speed of the current model training and the first required computing power includes:

[0049] According to the ratio of the data processing speed of the current model training to the data processing speed corresponding to the first required computing power, the first computing power utilization efficiency of the current model is obtained using the following formula:

[0050]

[0051] The obtaining of the second required computing power according to the first required computing power utilization efficiency of the current model and the set computing power utilization efficiency includes:

[0052]

[0053] The second required computing power is obtained according to the data processing speed corresponding to the second required computing power.

[0054] The method of obtaining the second computing power utilization efficiency of the current model training according to the data processing speed of the current model training and the second required computing power includes:

[0055]

[0056] The determining the type of accelerator card failure according to the ratio of the total power of the accelerator cards in the first accelerator card sequence to the total power of the accelerator cards in the second accelerator card sequence includes:

[0057] If the ratio of the total power of the accelerator cards in the first accelerator card sequence to the total power of the accelerator cards in the second accelerator card sequence and the ratio of the number of accelerator cards in the first accelerator card sequence to the number of accelerator cards in the second accelerator card sequence are consistent, it is a software failure; otherwise, it is a hardware failure.

[0058] The step of obtaining the target data processing rate of each accelerator card according to the second accelerator card sequence and the first computing power utilization efficiency includes:

[0059] According to the data processing speed corresponding to the computing power of each accelerator card in the second accelerator card sequence and the first computing power utilization efficiency, the target data processing rate of each accelerator card is obtained.

[0060] The method of completing the accelerator card scheduling according to the second accelerator card sequence includes:

[0061] like Figure 2 As shown, a multi-acceleration card scheduling system for model acceleration training applies the multi-acceleration card scheduling method for model acceleration training, including a cloud data server, an accelerator card scheduling module, an accelerator card control module, a data acquisition module, a communication module and a data processing module; the accelerator card scheduling module, the accelerator card control module, the data acquisition module, and the communication module are respectively connected to the data processing module; the cloud data server is communicatively connected to the communication module.

[0062] Specifically, the present invention provides a multi-acceleration card scheduling method for model acceleration training, the method comprising the following steps:

[0063] Step 1: Initial computing power allocation and efficiency evaluation

[0064] Calculate the first required computing power:

[0065] Calculate the required computing power based on the data processing speed required for model training.

[0066] Combined with the set computing power utilization efficiency (such as the expected computing power utilization rate), the first required computing power is obtained through the formula "first required computing power = required computing power / set computing power utilization efficiency".

[0067] Generate computing power container and accelerator card sequence:

[0068] The accelerator card scheduling module generates a computing power container according to the first required computing power, and the container is responsible for scheduling a corresponding number of accelerator cards. The scheduled accelerator cards form a first accelerator card sequence.

[0069] Linkage test and efficiency calculation: The accelerator card control module performs linkage test based on the first accelerator card sequence and the target model to measure the actual data processing speed of the current model training.

[0070] According to the actual data processing speed and the data processing speed corresponding to the first required computing power, the first computing power utilization efficiency of the current model is calculated. The formula is: first required computing power utilization efficiency = actual data processing speed / data processing speed corresponding to the first required computing power.

[0071] Step 2: Adjustment and redistribution of computing power

[0072] Calculate the second required computing power:

[0073] Recalculate the second required computing power based on the first computing power utilization efficiency of the current model and the data processing speed required for model training.

[0074] Specifically, the actual computing power achieved (i.e., the computing power corresponding to the actual data processing speed) is first inferred based on the first computing power utilization efficiency, and then the data processing speed corresponding to the second required computing power is calculated based on the set computing power utilization efficiency and the data processing speed required for model training, thereby obtaining the second required computing power.

[0075] Adjust the accelerator card sequence: according to the difference between the second required computing power and the first required computing power, dispatch a corresponding number of accelerator cards and merge them with the first accelerator card sequence to form a second accelerator card sequence.

[0076] Step 3: Efficiency evaluation again

[0077] Linkage test and efficiency calculation: The accelerator card control module performs linkage test based on the second accelerator card sequence and the target model to measure the actual data processing speed of the current model training.

[0078] According to the actual data processing speed and the second required computing power, the second computing power utilization efficiency of the current model training is calculated. The formula is: second computing power utilization efficiency = actual data processing speed / data processing speed corresponding to the second required computing power.

[0079] Determine efficiency consistency: If the first computing power usage efficiency is consistent with the second computing power usage efficiency, it means that the computing power adjustment is effective and proceed to step 7; otherwise, proceed to step 4 for troubleshooting.

[0080] Step 4: Determine the fault type

[0081] Power ratio analysis: Determine the fault type by comparing the total power ratio of the first accelerator card sequence and the second accelerator card sequence, as well as the ratio of the number of accelerator cards.

[0082] If the power ratio is consistent with the quantity ratio, it may be a software failure; otherwise, it may be a hardware failure.

[0083] Step 5: Hardware troubleshooting

[0084] Identify the faulty accelerator card: obtain the power of each accelerator card in the second accelerator card sequence, compare it with the design power, and regard the accelerator card whose difference is greater than the set threshold as a faulty accelerator card.

[0085] Replace the accelerator card: Replace the faulty accelerator card based on the faulty accelerator card information, and then return to step 3 for re-evaluation.

[0086] Step 6: Software troubleshooting

[0087] Identify faulty accelerator cards: Calculate the target data processing rate of each accelerator card based on the second accelerator card sequence and the first computing power usage efficiency. Compare the actual data processing rate of each accelerator card with the target data processing rate. Accelerator cards with a difference greater than a set threshold are considered faulty accelerator cards.

[0088] Software repair: Perform software repair on the faulty accelerator card based on the faulty accelerator card information, and then return to step 3 for re-evaluation.

[0089] Step 7: Complete accelerator card scheduling

[0090] According to the second accelerator card sequence, the final scheduling of the accelerator card is completed to ensure that the model training is carried out under the optimal computing power configuration.

[0091] Specific:

[0092] According to the data processing speed required for model training, the first required computing power is obtained:

[0093] In specific implementation, the data processing speed required for model training can be estimated through historical data or empirical formulas, and then the required computing power can be calculated.

[0094] The computing power utilization efficiency is usually set based on the actual system conditions and performance targets, such as 90%, 95%, etc.

[0095] The calculation formula for the first computing power utilization efficiency is:

[0096] First computing power utilization efficiency = actual data processing speed / (first required computing power * data processing speed corresponding to the inverse of the set computing power utilization efficiency).

[0097] The calculation of the second required computing power:

[0098] First, the actual computing power achieved is inferred based on the first computing power utilization efficiency, that is, the actual data processing speed is divided by the first computing power utilization efficiency.

[0099] Then, based on the set computing power utilization efficiency and the data processing speed required for model training, the data processing speed corresponding to the second required computing power is obtained through adjustment and calculation, and then the second required computing power is deduced.

[0100] Logic for fault type judgment:

[0101] If the total power ratio is consistent with the quantity ratio, but the computing power utilization efficiency is still not up to standard, it is likely to be a software problem, such as driver incompatibility, inefficient algorithm implementation, etc.

[0102] If the total power ratio is inconsistent with the quantity ratio and there is an obvious deviation, it is likely to be a hardware problem, such as a damaged accelerator card or poor heat dissipation.

[0103] Detailed steps for hardware troubleshooting:

[0104] Get the real-time power data of each accelerator card and compare it with the design power.

[0105] For accelerator cards with abnormally low power, further hardware tests, such as temperature detection and signal integrity testing, are performed to confirm the fault.

[0106] After replacing the faulty acceleration card, re-perform the linkage test to ensure that the system returns to normal.

[0107] Detailed steps for software troubleshooting:

[0108] According to the second acceleration card sequence and the first computing power usage efficiency, a data processing rate that each acceleration card should have is calculated.

[0109] For accelerator cards whose actual data processing rate is significantly lower than the target rate, check their software configuration, such as driver version, algorithm parameters, etc.

[0110] After repairing the software problem, re-perform the linkage test to verify the repair effect.

[0111] The present invention also provides a multi-acceleration card scheduling system for model acceleration training, which applies the above-mentioned multi-acceleration card scheduling method for model acceleration training, including a cloud data server, an acceleration card scheduling module, an acceleration card control module, a data acquisition module, a communication module and a data processing module.

[0112] Cloud data server: responsible for storing model training data, historical computing power usage data, etc., providing data support for the system.

[0113] Accelerator card scheduling module: generates computing power containers according to computing power requirements, is responsible for scheduling accelerator card resources, and forms an accelerator card sequence.

[0114] Accelerator card control module: responsible for communicating with the accelerator card, performing linkage tests, and obtaining the status information of the accelerator card.

[0115] Data acquisition module: responsible for collecting real-time power, temperature and other data of the accelerator card to provide a basis for troubleshooting.

[0116] Communication module: responsible for data communication between the cloud data server and other modules of the system.

[0117] Data processing module: responsible for processing the collected data, calculating the computing power utilization efficiency, determining the fault type, and issuing corresponding control instructions.

[0118] Example 1: Multi-accelerator card scheduling in deep learning model training

[0119] An AI research company is developing a complex deep learning model for image recognition tasks. In order to improve training efficiency, they decided to adopt the multi-accelerator scheduling method provided by the present invention and use multiple GPU accelerator cards to accelerate the model training process.

[0120] Specific steps:

[0121] Initial computing power allocation and efficiency evaluation:

[0122] Calculate the first required computing power:

[0123] Based on historical data and empirical formulas, it is estimated that the data processing speed required for model training is 1000GFLOPS (100 billion floating-point operations per second).

[0124] Set the computing power utilization efficiency to 90%.

[0125] According to the formula "first required computing power = required computing power / set computing power utilization efficiency", the first required computing power is calculated to be 1111GFLOPS (1000GFLOPS / 0.9).

[0126] Generate computing power container and accelerator card sequence:

[0127] The accelerator card scheduling module generates a computing power container based on the first required computing power, and schedules four GPU accelerator cards with equivalent performance to meet the demand, forming the first accelerator card sequence.

[0128] Linkage test and efficiency calculation:

[0129] The accelerator card control module was tested in conjunction with the target model, and the actual data processing speed was measured to be 950 GFLOPS.

[0130] According to the formula, the first computing power utilization efficiency is calculated to be 85.5% (data processing speed corresponding to 950GFLOPS / 1111GFLOPS, that is, the ratio of actual to theoretical).

[0131] Hashrate adjustment and redistribution:

[0132] Calculate the second required computing power: Based on the first computing power utilization efficiency of 85.5%, it can be deduced that the actual computing power achieved is close to the theoretical value, but does not reach the set efficiency.

[0133] Combining the set 90% computing power utilization efficiency and the 1000GFLOPS required for model training, it is calculated that the data processing speed corresponding to the second required computing power needs to be adjusted to a more efficient state, thereby obtaining the new second required computing power.

[0134] Adjust the accelerator card sequence: Based on the difference between the second required computing power and the first required computing power, it is decided to add another GPU accelerator card to improve the overall computing power. The second accelerator card sequence is formed, with a total of 5 GPU accelerator cards.

[0135] Efficiency evaluation again:

[0136] Linkage test and efficiency calculation: A new linkage test was conducted to measure the actual data processing speed to 1080 GFLOPS. The efficiency of the second computing power was calculated to be 97.2% (1080 GFLOPS / the second required computing power corresponding to the adjusted data processing speed).

[0137] Judging efficiency consistency:

[0138] Since the second computing power utilization efficiency of 97.2% is close to the set 90% and the system is stable, the computing power adjustment is considered effective.

[0139] Complete accelerator card scheduling: According to the second accelerator card sequence, complete the final scheduling of the accelerator card to ensure that the model training is carried out under the optimal computing power configuration.

[0140] Results: Through the multi-acceleration card scheduling method of the present invention, the training efficiency of the deep learning model was successfully improved, the expected computing power utilization efficiency was achieved, and the training time was shortened.

[0141] Embodiment 2: Multi-accelerator card scheduling optimization of natural language processing model A natural language processing (NLP) company is training a large language model. In order to improve the training speed and efficiency, it decides to adopt the multi-accelerator card scheduling system of the present invention.

[0142] Specific steps:

[0143] Initial computing power allocation and efficiency evaluation:

[0144] Calculation of the first required computing power: By analyzing the model structure and historical data, it is estimated that the data processing speed required for model training is 1500 TFLOPS (trillion floating-point operations per second).

[0145] Set the computing power utilization efficiency to 95%.

[0146] According to the formula, the first required computing power is calculated to be 1579TFLOPS (1500TFLOPS / 0.95).

[0147] Generate a computing power container and an accelerator card sequence: The accelerator card scheduling module generates a computing power container and schedules eight high-performance GPU accelerator cards to meet the demand, forming the first accelerator card sequence.

[0148] Linkage test and efficiency calculation: The acceleration card control module and the target model were linked to test, and the actual data processing speed was measured to be 1400TFLOPS. The first computing power utilization efficiency was calculated to be 88.6% (data processing speed corresponding to 1400TFLOPS / 1579TFLOPS).

[0149] Hashrate adjustment and redistribution:

[0150] Calculate the second required computing power: Based on the first computing power utilization efficiency of 88.6%, it can be inferred that the actual computing power does not reach the set efficiency.

[0151] Combining the set 95% computing power utilization efficiency and the 1500TFLOPS required for model training, the second required computing power is recalculated.

[0152] Adjust the accelerator card sequence: It was decided to add two more GPU accelerator cards to optimize the computing power distribution. The second accelerator card sequence was formed, with a total of 10 GPU accelerator cards.

[0153] Efficiency evaluation again:

[0154] Linkage test and efficiency calculation: A new linkage test was conducted to measure the actual data processing speed to 1550TFLOPS. The efficiency of the second computing power was calculated to be 98.1% (1550TFLOPS / the second required computing power corresponding to the adjusted data processing speed).

[0155] Judging efficiency consistency: The second computing power utilization efficiency of 98.1% is close to the set 95%, and the system is stable, so the computing power adjustment is considered effective.

[0156] Troubleshooting (if it occurs):

[0157] Fault type judgment: During a certain adjustment, it was found that the computing power utilization efficiency did not improve significantly, so a power ratio analysis was performed. The total power ratio was inconsistent with the quantity ratio, and it was judged that there might be a hardware failure.

[0158] Hardware fault handling: The real-time power of each accelerator card was obtained through the data acquisition module, and it was found that the power of one accelerator card was abnormally low. After replacing the faulty accelerator card and retesting, the system returned to normal.

[0159] Complete accelerator card scheduling: According to the final second accelerator card sequence, complete the accelerator card scheduling to ensure that model training is carried out under the optimal computing power configuration.

Claims

1. A multi-accelerator card scheduling method for model acceleration training, characterized in that: The steps include: Step 1: According to the data processing speed required for model training and the set computing power utilization efficiency, a first required computing power is obtained; the accelerator card scheduling module generates a computing power container according to the first required computing power; the first computing power container schedules the corresponding accelerator card according to the first required computing power, and generates a first accelerator card sequence; the accelerator card control module performs a linkage test based on the first accelerator card sequence and the target model to obtain the data processing speed of the current model training; according to the data processing speed of the current model training and the data processing speed corresponding to the first required computing power, the first computing power utilization efficiency of the current model is obtained; Step 2: According to the utilization efficiency of the first computing power of the current model and the data processing speed required for model training, a second required computing power is obtained; and according to the computing power difference between the second required computing power and the first required computing power, an accelerator card corresponding to the computing power difference is scheduled to form a second accelerator card sequence with the first accelerator card sequence; Step 3: The accelerator card control module performs a linkage test according to the second accelerator card sequence and the target model to obtain the data processing speed of the current model training, and obtains the second computing power usage efficiency of the current model training according to the data processing speed of the current model training and the second required computing power. If the first computing power usage efficiency is consistent with the second computing power usage efficiency, proceed to step 7; otherwise, proceed to step 4. Step 4: judging the type of accelerator card failure according to the ratio of the total power of the accelerator cards in the first accelerator card sequence to the total power of the accelerator cards in the second accelerator card sequence; if it is a hardware failure, proceeding to step 5; if it is a software failure, proceeding to step 6; Step 5: Obtain the power of each accelerator card in the second accelerator card sequence, and obtain the difference between the power of each accelerator card and the design power. The accelerator card with a difference greater than a set difference threshold is a faulty accelerator card. Replace the accelerator card according to the faulty accelerator card information, and return to step 3. Step 6: According to the second accelerator card sequence and the first computing power usage efficiency, the target data processing rate of each accelerator card is obtained respectively, and the data processing rate difference between the data processing rate of each accelerator card and the corresponding target data processing rate is obtained respectively. If the data processing rate difference is greater than the set data processing rate difference threshold, it is a faulty accelerator card. According to the faulty accelerator card information, the faulty accelerator is repaired by software, and the process returns to step 3; Step seven: completing the accelerator card scheduling according to the second accelerator card sequence.

2. The multi-acceleration card scheduling method for model acceleration training according to claim 1, characterized in that: The first required computing power is obtained according to the data processing speed required for model training, including: The corresponding computing power is obtained according to the data processing speed required for model training, and the first required computing power is obtained according to the ratio of the corresponding computing power to the set computing power utilization efficiency.

3. The multi-acceleration card scheduling method for model acceleration training according to claim 2, characterized in that: The obtaining of the first computing power utilization efficiency of the current model according to the data processing speed of the current model training and the first required computing power includes: According to the ratio of the data processing speed of the current model training to the data processing speed corresponding to the first required computing power, the first computing power utilization efficiency of the current model is obtained using the following formula:

4. The multi-acceleration card scheduling method for model acceleration training according to claim 3 is characterized in that: The obtaining of the second required computing power according to the first required computing power utilization efficiency of the current model and the set computing power utilization efficiency includes: The second required computing power is obtained according to the data processing speed corresponding to the second required computing power.

5. The multi-acceleration card scheduling method for model acceleration training according to claim 4, characterized in that: The method of obtaining the second computing power utilization efficiency of the current model training according to the data processing speed of the current model training and the second required computing power includes:

6. The multi-acceleration card scheduling method for model acceleration training according to claim 5, characterized in that: The determining the type of accelerator card failure according to the ratio of the total power of the accelerator cards in the first accelerator card sequence to the total power of the accelerator cards in the second accelerator card sequence includes: If the ratio of the total power of the accelerator cards in the first accelerator card sequence to the total power of the accelerator cards in the second accelerator card sequence and the ratio of the number of accelerator cards in the first accelerator card sequence to the number of accelerator cards in the second accelerator card sequence are consistent, it is a software failure; otherwise, it is a hardware failure.

7. The multi-acceleration card scheduling method for model acceleration training according to claim 6, characterized in that: The step of obtaining the target data processing rate of each accelerator card according to the second accelerator card sequence and the first computing power utilization efficiency includes: According to the data processing speed corresponding to the computing power of each accelerator card in the second accelerator card sequence and the first computing power utilization efficiency, the target data processing rate of each accelerator card is obtained.

8. A multi-accelerator card scheduling system for model acceleration training, characterized in that: The multi-acceleration card scheduling method for model acceleration training according to any one of claims 1 to 7 includes a cloud data server, an accelerator card scheduling module, an accelerator card control module, a data acquisition module, a communication module and a data processing module; the accelerator card scheduling module, the accelerator card control module, the data acquisition module and the communication module are respectively connected to the data processing module; the cloud data server is communicatively connected to the communication module.

Images