A method and system for optimizing performance results of deep learning model training

Abstract

The application discloses a kind of performance result optimization method and system of deep learning model training, method includes: in response to the end of deep learning model training, the performance result index of deep learning model training is acquired and the debugging parameter of deep learning model training is acquired;Search to debugging parameter and from the performance result index corresponding to debugging parameter in search result, the performance result index corresponding to current search is compared with the performance result index corresponding to last search, wherein the performance result index corresponding to last search includes the performance result index of deep learning model training;In response to the comparison result meets preset condition, stop searching to debugging parameter and determine the final parameter of debugging parameter;Based on each final parameter determined, deep learning model training is carried out again to obtain the performance result after optimization.Through the scheme of the application, performance evaluation cycle is shortened, and performance evaluation efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a performance result optimization method and system for deep learning model training. BACKGROUND

[0002] With the development of deep learning models from research to application deployment stage, various AI software and hardware framework for deep learning application model emerge in an endless stream. In the application deployment process, a large number of deep learning models and data sets are processed, and the performance evaluation of various AI application software framework and AI hardware architecture is becoming the focus of the industry. AI performance evaluation tools are thus produced to evaluate the benchmark performance indicators (such as Samples / s / FLOPS, etc.) of each software and hardware platform for deep learning models.

[0003] The current AI performance evaluation tools are mainly used to monitor the key performance indicators in the execution process of deep learning model training / inference in real time, and to summarize and calculate in the final execution stage, so as to comprehensively evaluate the key performance indicators and overall performance in the execution process of deep learning model.

[0004] The existing AI performance evaluation tool tf_cnn_benchmark can be used for training and benchmarking of various deep learning models. Users only need to define the hyperparameters related to training (such as learning_rate, batch_size, weight_decay) and other configuration parameters (such as num_threade, workers), and the framework can perform model training iteration and performance evaluation. The evaluation results related to software and hardware performance are output during the training process, such as Samples / s (the number of samples processed per second, used to measure the performance of model training software), FLOPS (floating point of per second, the number of floating point operations per second, used to measure the performance of hardware), etc.

[0005] The existing AI performance evaluation tools generally only evaluate the test based on fixed parameters after the user defines the hyperparameters and configuration parameters. Each evaluation result only outputs a set of performance data. If the evaluation result of this round is not ideal, the user needs to manually adjust the parameters for a new round of evaluation. Manual parameter adjustment is time-consuming and laborious, resulting in long evaluation period and low efficiency. SUMMARY

[0006] Therefore, the application provides a performance result optimization method and system for deep learning model training. In the base performance evaluation process of the deep learning model, automatic performance-related parameter analysis and parameter tuning are performed according to the last round of evaluation results, so that the deep learning model can exert the optimal performance result in the evaluation process, and the problem that the existing performance evaluation tool needs to manually adjust parameters for a new round of evaluation if the evaluation result of the current round is not ideal, thereby causing a long evaluation period and low efficiency is solved.

[0007] To achieve the above object, in one aspect, the application provides a performance result optimization method for deep learning model training, which specifically comprises the following steps:

[0008] In response to the end of the deep learning model training, the performance result indicators of the deep learning model training are obtained, and the debugging parameters of the deep learning model training are obtained;

[0009] The debugging parameters are searched, and the performance result indicators corresponding to the debugging parameters are obtained from the search results. The performance result indicators corresponding to the current search are compared with the performance result indicators corresponding to the last search, wherein the performance result indicators corresponding to the last search include the performance result indicators of the deep learning model training;

[0010] In response to the comparison result meeting a preset condition, the search of the debugging parameters is stopped, and the final parameters of the debugging parameters are determined;

[0011] The deep learning model training is performed again based on the determined final parameters to obtain an optimized performance result.

[0012] In some embodiments, the performance result indicators of the deep learning model training are obtained, comprising:

[0013] The epoch number, the number of samples processed per second, the average GPU memory occupancy rate, and the average GPU usage rate of the deep learning model training are obtained;

[0014] The debugging parameters of the deep learning model training are obtained, comprising:

[0015] The learning rate, the batch size, and the number of parallel threads of the deep learning model training are obtained.

[0016] In some embodiments, the number of samples processed per second, the average GPU memory occupancy rate, and the average GPU usage rate of the deep learning model training are obtained, comprising:

[0017] The number of samples and the training time of the deep model training are obtained, and the number of samples processed per second is obtained based on the number of samples and the training time;

[0018] The GPU memory occupancy rate and the GPU usage rate are obtained at intervals of a preset period, and the average GPU memory occupancy rate and the average GPU usage rate are calculated based on the number of times of obtaining the GPU memory occupancy rate and the GPU usage rate.

[0019] In some embodiments, the search is performed on the debugging parameter, and a performance result index corresponding to the debugging parameter is obtained from the search result. The performance result index corresponding to the current search is compared with the performance result index corresponding to the last search, including:

[0020] The learning rate is searched, and the epoch number of the current search is obtained from the search result. The epoch number of the current search is compared with the epoch number of the last search.

[0021] The batch size is searched, and the number of samples processed per second of the current batch size search is obtained from the search result. The number of samples processed per second of the current batch size search is compared with the number of samples processed per second of the last batch size search.

[0022] The number of parallel threads is searched, and the number of samples processed per second of the current thread number search is obtained from the search result. The number of samples processed per second of the current thread number search is compared with the number of samples processed per second of the last thread number search.

[0023] In some embodiments, the search is performed on the debugging parameter, including:

[0024] The debugging parameter is adjusted, and the deep learning model is trained based on the adjusted debugging parameter.

[0025] In some embodiments, the search is performed on the learning rate, including: adjusting the learning rate, and training the deep learning model based on the adjusted learning rate.

[0026] In some embodiments, the search is performed on the batch size, including: adjusting the batch size, and training the deep learning model based on the adjusted batch size.

[0027] In some embodiments, the search is performed on the number of parallel threads, including: adjusting the number of parallel threads, and training the deep learning model based on the adjusted number of parallel threads.

[0028] In some embodiments, the method further includes:

[0029] In response to the comparison result not meeting a preset condition, the search step of searching the debugging parameter is returned to obtain a final parameter of the debugging parameter.

[0030] Still another aspect of the embodiment of the present application also provides a performance result optimization system for deep learning model training, comprising:

[0031] an acquisition module configured to acquire a performance result index of the deep learning model training and acquire a debugging parameter of the deep learning model training in response to the end of the deep learning model training;

[0032] a search module configured to search the debugging parameter and acquire a performance result index corresponding to the debugging parameter from a search result, and compare a performance result index corresponding to a current search with a performance result index corresponding to a last search, wherein the performance result index corresponding to the last search comprises the performance result index of the deep learning model training;

[0033] a determination module configured to stop searching the debugging parameter and determine a final parameter of the debugging parameter in response to a comparison result meeting a preset condition;

[0034] a training module configured to re-perform the deep learning model training based on the determined final parameters of the debugging parameters to obtain an optimized performance result.

[0035] The present application has the following beneficial technical effects: by searching each debugging parameter, the final parameter of each debugging parameter is obtained, and the deep learning model is trained based on the final parameters of the debugging parameters to obtain the best performance training result, thereby shortening the performance evaluation period and improving the performance evaluation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other embodiments according to these drawings without creative labor.

[0037] Figure 1 a block diagram of an embodiment of the performance result optimization method for deep learning model training provided by the present application;

[0038] Figure 2 a schematic diagram of an embodiment of the performance result optimization system for deep learning model training provided by the present application. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further specifically describe the embodiments of the present application with reference to the drawings.

[0040] It should be noted that all the expressions of "first" and "second" in the embodiments of the present application are used to distinguish two same name non-same entities or non-same parameters. It can be seen that "first" and "second" are only for the convenience of description, and should not be understood as a limitation of the embodiments of the present application. The subsequent embodiments will not be described one by one.

[0041] Based on the above purpose, the first aspect of the embodiments of the present application proposes an embodiment of a performance result optimization method of deep learning model training. As shown in the figure, it includes the following steps: Figure 1

[0042] Step S101, in response to the end of deep learning model training, obtaining the performance result index of the deep learning model training and obtaining the debugging parameter of the deep learning model training;

[0043] Step S103, searching the debugging parameter and obtaining the performance result index corresponding to the debugging parameter from the search result, comparing the performance result index corresponding to the current search with the performance result index corresponding to the last search, wherein the performance result index corresponding to the last search includes the performance result index of the deep learning model training;

[0044] Step S105, in response to the comparison result meeting the preset condition, stopping searching the debugging parameter and determining the final parameter of the debugging parameter;

[0045] Step S107, re-performing the deep learning model training based on the determined each final parameter to obtain the optimized performance result.

[0046] Based on the AI performance benchmark evaluation tool, based on the AI training software, the following complete training process is supported:

[0047] The deep learning model training process is as follows: obtaining a training data set and preprocessing the data set; obtaining the calculation graph of the training model, reading the preprocessed data set, and performing forward and backward calculation on the model based on the calculation graph; outputting the calculation result value of the model such as loss, Accuracy, etc.

[0048] The parameters related to the model training process include the following two parts:

[0049] I. The parameters related to performance debugging include batch size batchsize, learning rate learning_rate, weight decay weight_decay, momentum Momentum, and the number of parallel threads num_threads, which can be used for performance optimization of deep learning model;

[0050] ​II. Performance evaluation-related parameters include, for example, samples / s, FLOPS, epochs (1 epoch is equivalent to training using all samples in the training set), iterations, etc., which can be used to evaluate the performance of a deep learning model.

[0051] After the deep learning model training is completed, any one or a combination of multiple of the performance debugging-related parameters, such as batch size, learning rate, weight decay, momentum, and the number of parallel threads, are obtained. The obtained debugging parameters are sequentially searched at least once, and the performance result indicators corresponding to the debugging parameters are obtained from the search results. The performance result indicators corresponding to the current search are compared with the performance result indicators corresponding to the last search. In the first search, the performance result indicators corresponding to the last search correspond to the performance result indicators of the deep learning model training. It is determined whether the change in the performance result indicators meets a preset condition. If so, the search is stopped, the final parameters of the current debugging parameters are determined, the search of the next debugging parameter is continued, and the final parameters of all debugging parameters are determined. The deep learning model is retrained based on the final parameters of all debugging parameters to obtain the optimal performance evaluation result after optimization of the debugging parameters.

[0052] The embodiment sequentially searches each debugging parameter to obtain the final parameters of each debugging parameter, trains the deep learning model based on the final parameters of each debugging parameter to obtain the best performance training result, and has a short evaluation period and high evaluation efficiency.

[0053] In some embodiments, the performance result indicators of the deep learning model training are obtained, including:

[0054] The number of epochs, the number of samples processed per second, the average GPU memory occupancy rate, and the average GPU usage rate of the deep learning model training are obtained.

[0055] The debugging parameters of the deep learning model training are obtained, including:

[0056] The learning rate, batch size, and number of parallel threads of the deep learning model training are obtained.

[0057] In some embodiments, the number of samples processed per second, the average GPU memory occupancy rate, and the average GPU usage rate of the deep learning model training are obtained, including:

[0058] The number of samples and the training time of the deep model training are obtained, and the number of samples processed per second is obtained based on the number of samples and the training time.

[0059] The GPU memory occupancy rate and the GPU usage rate are obtained at intervals of a preset period, and the average GPU memory occupancy rate and the average GPU usage rate are calculated based on the number of times of obtaining.

[0060] Specifically, the obtaining process of the number of samples processed per second, the average GPU memory occupancy rate, and the average GPU usage rate is described.

[0061] Suppose that 1000 samples are to be trained by a deep learning model, and the training process is an iteration process of the 1000 samples, and the epochs for completing the training are 10 times.

[0062] The time t2 at the end of the training is obtained, and the time t spent by the current training is obtained as t = t2-t1. The number of samples processed per second is obtained by dividing the number of samples processed in the current training by the time t spent by the current training.

[0063] The GPU memory occupancy rate and the GPU usage rate are obtained every 1 second during the training by using the interface function of NVML, and the average GPU memory occupancy rate and the average GPU usage rate are obtained by taking the average of the values.

[0064] The performance result index under the current parameters is saved for comparison with the performance data under other parameters.

[0065] In some embodiments, the debugging parameters are searched, and the performance result index corresponding to the debugging parameters is obtained from the search results. The performance result index corresponding to the current search is compared with the performance result index corresponding to the last search, including:

[0066] The learning rate is searched, and the epoch number of the current search is obtained from the search results, and the epoch number of the current search is compared with the epoch number of the last search;

[0067] The batch size is searched, and the number of samples processed per second of the current batch size search is obtained from the search results, and the number of samples processed per second of the current batch size search is compared with the number of samples processed per second of the last batch size search;

[0068] The number of parallel threads is searched, and the number of samples processed per second of the current thread number search is obtained from the search results, and the number of samples processed per second of the current thread number search is compared with the number of samples processed per second of the last thread number search.

[0069] In some embodiments, the debugging parameters are searched, including:

[0070] adjusting the debugging parameter, and training the deep learning model based on the adjusted debugging parameter.

[0071] In some embodiments, searching the learning rate comprises: adjusting the learning rate, and training the deep learning model based on the adjusted learning rate.

[0072] In some embodiments, searching the batch size comprises: adjusting the batch size, and training the deep learning model based on the adjusted batch size.

[0073] Specifically, adjusting the batch size and training the deep learning model based on the adjusted batch size comprises:

[0074] obtaining an average GPU memory occupancy rate of a last search and an average GPU usage rate of the last search, and determining whether the average GPU memory occupancy rate of the last search and the average GPU usage rate of the last search are both not greater than a preset percentage;

[0075] in response to the average GPU memory occupancy rate of the last search and the average GPU usage rate of the last search both not being greater than the preset percentage, increasing the batch size;

[0076] training the deep learning model based on the increased batch size.

[0077] In some embodiments, searching the number of parallel threads comprises: adjusting the number of parallel threads, and training the deep learning model based on the adjusted number of parallel threads.

[0078] Specifically, in some embodiments, adjusting the number of parallel threads and training the deep learning model based on the adjusted number of parallel threads comprises:

[0079] obtaining a number of parallel threads of a last search, and determining whether the number of parallel threads is less than a maximum number of threads;

[0080] in response to the number of parallel threads being less than the maximum number of threads, adding 1 to the number of parallel threads;

[0081] training the deep learning model based on the number of parallel threads after the addition.

[0082] In some embodiments, the method further comprises:

[0083] in response to the comparison result not meeting the preset condition, returning to the step of searching the debugging parameter to obtain a final parameter of the debugging parameter.

[0084] Specifically, the determination conditions of the debugging parameters are different. The determination conditions of the final parameters of the learning rate, the batch size, and the number of parallel threads are described below.

[0085] For the learning rate, the current search epoch number is compared with the last search epoch number. If the current search epoch number is less than the last search epoch number, the current learning rate is increased, and the deep learning model is trained based on the increased learning rate. After the training is completed, the change of the epoch index is judged until the current search epoch number is greater than the last search epoch number. If the current search epoch number is greater than the last search epoch number, the average value of the current learning rate and the last learning rate is calculated. It is judged whether the average value is within the preset error. If the average value is within the preset error, the last learning rate is determined as the final learning rate, and the search for the learning rate is stopped, and the search for the number of parallel threads is continued.

[0086] For example, the learning rate learning_rate is increased by 50% for the first time. If the subsequent result num_epochs is less than the last time, the current learning rate is recorded, and the learning rate is continued to be increased by 50%. If num_epochs is greater than the last time, the average value of the current learning rate and the last learning rate is calculated. If the value is within 1% of the last value, the last learning rate is selected as the final learning rate, and the search for the learning rate is stopped.

[0087] For the batch size, if the average GPU memory occupancy rate of the last search and the average GPU usage of the current search are both not greater than the preset percentage, the samples processed per second of the current batch size search is compared with the samples processed per second of the last search. If the samples processed per second of the current batch size search is less than the samples processed per second of the last search, the last batch size is determined as the final batch size. If the samples processed per second of the current batch size search is greater than the samples processed per second of the last search, the batch size is increased, and the deep learning model is trained based on the increased batch size. The change of the batch size is continued to be compared until the samples processed per second of the current batch size search is less than the samples processed per second of the last search, and the search for the batch size is stopped, and the search for the number of parallel threads is continued.

[0088] For example, if the GPU utilization and GPU memory usage do not reach 80% in the first instance, and the current sample size is greater than the previous sample size, then the batch size is increased by the formula: new batch size = old batch size + 8 * 1. If the current sample size is less than the previous sample size, then the search stops and the previous batch size is selected as the final batch size.

[0089] Regarding the number of parallel threads, if the number of parallel threads is not greater than the maximum number of threads, the process compares the number of samples processed per second in the current search with the number of samples processed per second in the previous search. If the number of samples processed per second in the current search is greater than the number of samples processed per second in the previous search, the number of threads in the current search is incremented by 1, and the deep learning model is retrained until the number of samples processed per second in the current search is less than the number of samples processed per second in the previous search. Then, the number of threads in the previous search is determined as the final number of threads, and the search for the number of parallel threads is stopped.

[0090] For example, if the current Samples / s is larger than the previous one, and num_thread is less than the maximum number of threads, then num_threads is incremented by 1. If the current Samples / s is smaller than the previous one, then the previous num_thread is selected as the final num_thread and the search stops.

[0091] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 2 As shown, embodiments of the present invention also provide a performance result optimization system for deep learning model training, comprising:

[0092] The acquisition module 110 is configured to acquire the performance result indicators of the deep learning model training and the debugging parameters of the deep learning model training in response to the end of deep learning model training.

[0093] Search module 120 is configured to search for the debugging parameters and obtain the performance result indicators corresponding to the debugging parameters from the search results, and compare the performance result indicators corresponding to the current search with the performance result indicators corresponding to the previous search, wherein the performance result indicators corresponding to the previous search include the performance result indicators of the deep learning model training.

[0094] The determination module 130 is configured to stop searching for the debugging parameters and determine the final parameters of the debugging parameters in response to the comparison result meeting a preset condition.

[0095] The training module 140 is configured to retrain the deep learning model based on the determined final parameters to obtain an optimized performance result.

[0096] The embodiments of the present application can also include corresponding computer devices. The computer device includes a memory, at least one processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to perform any of the above methods.

[0097] The memory is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules of the performance result optimization method of the deep learning model training in the embodiments of the present application. The processor executes the non-volatile software programs, instructions and modules stored in the memory to perform various functional applications and data processing of the device, i.e. to implement the performance result optimization method of the deep learning model training of the above method embodiments.

[0098] The memory can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged with respect to the processor, which can be connected to the local module through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0099] Finally, it should be noted that those skilled in the art can understand that all or part of the above-mentioned method embodiments can be implemented by a computer program to instruct related hardware to complete, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The storage medium of the program can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. The above-mentioned computer program embodiments can achieve the same or similar effects as the corresponding any of the above-mentioned method embodiments.

[0100] Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present embodiments.

[0101] The above are exemplary embodiments disclosed by the present embodiments, but it should be noted that various changes and modifications can be made without departing from the scope of the present embodiments disclosed by the claims. The functions, steps and / or actions of the method claims described herein need not be performed in any particular order. Furthermore, although elements of the present embodiments disclosed can be described or claimed in individual forms, unless expressly limited to the singular, these elements can also be understood to be implemented in a plurality of forms. The use of "including," "comprising," or "having" and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.

[0102] It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0103] The above-mentioned embodiment number of the present embodiments is only for description, not representing the advantages and disadvantages of the embodiments.

[0104] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed to relevant hardware by a program, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0105] Those of ordinary skill in the art will understand that the above discussion of various embodiments is merely exemplary and is not intended to suggest that the scope of the present embodiments disclosed (including the claims) is limited to these examples; under the concept of the present embodiments, the above embodiments or technical features in different embodiments can also be combined, and there are many other changes of the above aspects of the present embodiments. In order to be brief, they are not provided in detail. Therefore, any omission, modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiments should be included in the protection scope of the present embodiments.

Claims

1. A method for optimizing the performance results of deep learning model training, characterized in that, The method comprises the following steps: in response to the end of the deep learning model training, obtaining a performance result index of the deep learning model training and a plurality of debugging parameters of the deep learning model training, wherein the performance result index comprises an epoch number, a sample number processed per second, an average GPU memory occupancy rate, and an average GPU usage rate; the plurality of debugging parameters comprises a learning rate, a batch size, and a number of parallel threads; searching each debugging parameter in sequence; for the learning rate, comparing the current search epoch number with the last search epoch number, if the current search epoch number is less than the last search epoch number, increasing the current learning rate and training the deep learning model based on the increased learning rate, judging the epoch index change after the training is completed, until the current search epoch number is greater than the last search epoch number; if the current search epoch number is greater than the last search epoch number, calculating the average value of the current learning rate and the last learning rate, judging whether the average value is within a preset error, if the average value is within the preset error, determining the last learning rate as the final learning rate, and stopping the search of the learning rate and continuing the search of the batch size; for the batch size, if the average GPU memory occupancy rate of the last search and the average GPU usage rate of the current search are both not greater than a preset percentage, comparing the sample number processed per second of the current batch size search with the sample number processed per second of the last search; if the sample number processed per second of the current batch size search is less than the sample number processed per second of the last search, determining the last batch size as the final batch size, if the sample number processed per second of the current batch size search is greater than the sample number processed per second of the last search, increasing the batch size and training the deep learning model based on the increased batch size, and continuing to compare the batch size change, until the sample number processed per second of the current batch size search is less than the sample number processed per second of the last search, and stopping the search of the batch size and continuing the search of the number of parallel threads; for the number of parallel threads, in the case that the number of parallel threads is not greater than the maximum number of threads, comparing the sample number processed per second of the current thread number search with the sample number processed per second of the last search, if the sample number processed per second of the current search is greater than the sample number processed per second of the last search, retraining the deep learning model after adding 1 to the current thread number, until the sample number processed per second of the current search is less than the sample number processed per second of the last search, determining the thread number of the last search as the final thread number, and stopping the search of the number of parallel threads; retraining the deep learning model based on the determined final parameters to obtain an optimized performance result.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining the epoch number, the sample number processed per second, the average GPU memory occupancy rate, and the average GPU usage rate of the deep learning model training.

3. The method of claim 2, wherein, obtaining the sample number processed per second, the average GPU memory occupancy rate, and the average GPU usage rate of the deep learning model training comprises the following steps: obtaining a sample number and a training time of the deep learning model training, and obtaining the sample number per second based on the sample number and the training time; obtaining GPU memory occupancy and GPU usage at intervals of a preset period, and calculating the average GPU memory occupancy and the average GPU usage based on the number of times of obtaining. 4.A system for optimizing performance results of deep learning model training, characterized in that, comprising: an obtaining module configured to, in response to the end of the deep learning model training, obtain a performance result indicator of the deep learning model training and obtain a debugging parameter of the deep learning model training, wherein the performance result indicator comprises an epoch number and a sample number per second; and the debugging parameter comprises a learning rate, a batch size, and a number of parallel threads; a searching module configured to search the debugging parameter in sequence; wherein for the learning rate, comparing the current search epoch number with the last search epoch number, if the current search epoch number is less than the last search epoch number, increasing the current learning rate and training the deep learning model based on the increased learning rate, judging the change of the epoch number after the training ends, until the current search epoch number is greater than the last search epoch number; if the current search epoch number is greater than the last search epoch number, calculating the average value of the current learning rate and the last learning rate, judging whether the average value is within a preset error, if the average value is within the preset error, determining the last learning rate as the final learning rate, and stopping the search of the learning rate and continuing the search of the batch size; for the batch size, if the average GPU memory occupancy of the last search and the average GPU usage of the current search are both not greater than a preset percentage, comparing the sample number per second of the current batch size search with the sample number per second of the last search; if the sample number per second of the current batch size search is less than the sample number per second of the last search, determining the last batch size as the final batch size, if the sample number per second of the current batch size search is greater than the sample number per second of the last search, increasing the batch size and training the deep learning model based on the increased batch size, and continuing to compare the change of the batch size until the sample number per second of the current batch size search is less than the sample number per second of the last search, and stopping the search of the batch size and continuing the search of the number of parallel threads; for the number of parallel threads, in the case that the number of parallel threads is not greater than a maximum number of threads, comparing the sample number per second of the current thread number search with the sample number per second of the last search, if the sample number per second of the current search is greater than the sample number per second of the last search, adding 1 to the current thread number and retraining the deep learning model, until the sample number per second of the current search is less than the sample number per second of the last search, determining the thread number of the last search as the final thread number, and stopping the search of the number of parallel threads. a training module configured to retrain the deep learning model based on the determined individual final parameters to obtain an optimized performance result.

Images