GPU test method and device, electronic equipment and storage medium
By performing iterative training on GPUs and analyzing model loss differences, this method solves the problem of the inability to effectively evaluate the training effect of GPUs in existing technologies. It realizes a method for selecting a suitable GPU in large model training, thereby improving the stability and generalization ability of training results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NEW H3C TECH CO LTD
- Filing Date
- 2026-01-19
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies rely solely on performance metrics such as GPU memory size and floating-point operation speed when selecting GPUs, which cannot effectively reflect the GPU's performance in training large models, resulting in unstable training results and insufficient generalization ability.
By using training data in a fixed order to perform a preset number of iterations on the GPU under test and the reference GPU, the relative deviation of the model loss is calculated, a difference representation value is generated, and the test results of the GPU are generated based on the difference representation value. This eliminates the difference in model loss caused by different training data and evaluates the training effect of the GPU.
This study effectively evaluated the performance of GPUs in large model training, helping technicians select suitable GPUs, reduce training instability and insufficient generalization ability, and improve the stability and effectiveness of model training.
Smart Images
Figure CN122019286A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a GPU testing method, apparatus, electronic device, and storage medium. Background Technology
[0002] With the rapid development and widespread application of artificial intelligence, GPUs (Graphics Processing Units), which support the computing power of artificial intelligence, are receiving increasing attention.
[0003] With a wide variety of GPUs available on the market, selecting the right GPU is crucial for training large models. Currently, when choosing a GPU for large model training tasks, the main focus is on performance metrics such as GPU memory size and floating-point operation speed. However, these metrics often fail to effectively reflect the GPU's training performance on large models. Therefore, GPU testing is necessary to help technicians understand the training effect and select the appropriate GPU. Summary of the Invention
[0004] The purpose of this invention is to provide a GPU testing method, apparatus, electronic device, and storage medium to enable GPU testing, facilitating technicians to understand the training performance of the GPU and select a suitable GPU. The specific technical solution is as follows:
[0005] This invention first provides a GPU testing method, the method comprising:
[0006] Using the GPU under test, a preset large model is trained on a preset number of iterations with training data in a fixed order, and the model loss of the preset large model in each iteration is obtained as the target model loss.
[0007] Obtain the difference representation value between the sequence of the target model loss and the reference sequence; wherein, the reference sequence includes: the model loss obtained by performing the preset number of iterations of training on the preset large model using the training data in the fixed order using a reference GPU;
[0008] The test results for the GPU under test are generated based on the difference characterization values.
[0009] In one embodiment of the present invention, obtaining the difference between the sequence of the target model loss composition and the reference sequence includes:
[0010] For each iteration, the relative deviation between the target model loss of that iteration and the model loss of that iteration in the reference sequence is calculated, and the relative deviation of each iteration is used as the difference characterization value between the sequence composed of the target model loss and the reference sequence.
[0011] In one embodiment of the present invention, the relative deviation of each iteration is calculated based on the following formula:
[0012] deviation=|loss1-loss0| / loss0;
[0013] Where deviation is the relative deviation of this iteration; loss1 is the target model loss of this iteration; and loss0 is the model loss of this iteration in the reference sequence.
[0014] In one embodiment of the present invention, generating the test result of the GPU under test based on the difference characterization value includes:
[0015] The number of iterations with a relative deviation less than a preset threshold is determined as the number to be utilized;
[0016] The test results for the GPU under test are generated based on the number of GPUs to be utilized.
[0017] In one embodiment of the present invention, generating the test results of the GPU to be tested based on the number of GPUs to be utilized includes:
[0018] Based on the pre-defined correspondence between multiple number intervals and test results, the test result corresponding to the number interval to which the number to be used belongs is determined, and used as the test result of the GPU to be tested.
[0019] In one embodiment of the present invention, before using the GPU under test to perform a preset number of iterations of training on a preset large model with training data in a fixed order, the method further includes:
[0020] The preset large model is initialized using a preset random seed; wherein the model loss contained in the reference sequence is obtained by training the preset large model using the preset random seed.
[0021] And / or,
[0022] The specified training function of the preset large model is to disable the specified training function, wherein the specified training function is to randomly deactivate neurons in the preset large model during the training process; the model loss contained in the reference sequence is obtained by training with the specified training function disabled.
[0023] This invention also provides a GPU testing device, the device comprising:
[0024] The model training module is used to train a preset large model with training data in a fixed order using the GPU under test for a preset number of iterations, and to obtain the model loss of the preset large model in each iteration, which is used as the target model loss.
[0025] A difference acquisition module is used to obtain the difference representation value between the sequence of the target model loss and the reference sequence; wherein, the reference sequence includes: the model loss obtained by using a reference GPU to train the preset large model with the training data in the fixed order for the preset number of iterations;
[0026] The result generation module is used to generate test results for the GPU under test based on the difference characterization values.
[0027] In one embodiment of the present invention, the difference acquisition module is specifically used for:
[0028] For each iteration, the relative deviation between the target model loss of that iteration and the model loss of that iteration in the reference sequence is calculated, and the relative deviation of each iteration is used as the difference characterization value between the sequence composed of the target model loss and the reference sequence.
[0029] This invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0030] Memory, used to store computer programs;
[0031] When a processor executes a program stored in memory, it implements any of the GPU testing method steps described above.
[0032] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the GPU testing method steps described above.
[0033] This invention also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the GPU testing methods described above.
[0034] Beneficial effects of the embodiments of the present invention:
[0035] The GPU testing method provided in this invention utilizes the GPU under test to train a preset large model using training data in a fixed order for a preset number of iterations, obtaining the model loss of the preset large model in each iteration as the target model loss; obtaining the difference representation value between the sequence of target model losses and the reference sequence; wherein, the reference sequence includes: the model loss obtained by training the preset large model using a reference GPU with training data in a fixed order for a preset number of iterations; finally, generating the test result of the GPU under test based on the difference representation value. It can be seen that this scheme, by inputting the training data into the large model trained by the GPU under test and the large model trained by the reference GPU in the same order, can eliminate the factor of different model losses caused by different training data. Furthermore, based on the difference between the model loss obtained by the GPU under test and the model loss obtained by the reference GPU in each iteration, the GPU under test is evaluated. When faced with a wide variety of GPUs, it can select a GPU whose model loss during the training iteration process is highly similar to that of mainstream GPUs, thus facilitating technicians to understand the training effect of the GPU and select a suitable GPU.
[0036] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.
[0038] Figure 1 This is a schematic diagram of the first process of the GPU testing method provided in an embodiment of the present invention;
[0039] Figure 2 This is a schematic diagram of a second process of the GPU testing method provided in an embodiment of the present invention;
[0040] Figure 3 This is a schematic diagram of the third process of the GPU testing method provided in the embodiments of the present invention;
[0041] Figure 4 This is a schematic diagram of the structure of the GPU testing device provided in an embodiment of the present invention;
[0042] Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation
[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.
[0044] Training large models relies heavily on GPUs, making the selection of a suitable GPU crucial. Choosing a GPU with subpar training performance will result in models trained on that GPU with insufficient generalization ability, making them unsuitable for real-world applications. Currently, there is a lack of effective testing methods for evaluating the training performance of GPUs in large model training. Simply using the ability to train the large model normally and the convergence of the model loss as metrics for GPU training performance is insufficient to effectively assess the chosen GPU's effectiveness in large-scale model training. Relying solely on these metrics may lead to high instability due to the selection of an unsuitable GPU, resulting in significant variations in training results and increased model randomness. It may also cause insufficient generalization ability. All of these factors contribute to poor performance of large models trained in practical applications, failing to meet expectations.
[0045] To address the aforementioned issues and facilitate technicians' understanding of GPU training performance, thereby enabling them to select suitable GPUs, embodiments of this invention provide a GPU testing method, apparatus, electronic device, and storage medium. This method can be used in electronic devices capable of installing GPUs, such as computers and servers. Figure 1 As shown, the method includes the following steps:
[0046] S101: Using the GPU under test, a preset large model is trained on a preset number of iterations with training data in a fixed order, and the model loss of the preset large model in each iteration is obtained as the target model loss.
[0047] S102, obtain the difference representation value between the sequence of target model loss and the reference sequence; wherein, the reference sequence includes: the model loss obtained by using a reference GPU to train a preset large model with training data in a fixed order for a preset number of rounds of iterations;
[0048] S103 generates test results for the GPU under test based on the difference characterization values.
[0049] In this embodiment, training data is input into the large model trained by the GPU under test and the large model trained by the reference GPU in the same order. This can eliminate the factor that different training data will cause different model losses. Then, based on the difference between the model loss obtained by the GPU under test and the model loss obtained by the reference GPU in each iteration, the GPU under test is evaluated. When faced with a wide variety of GPUs, it is possible to select a GPU whose model loss during the training iteration process is very similar to that of mainstream GPUs. This makes it easier for technicians to understand the training effect of the GPU and select a suitable GPU.
[0050] For step S101, the preset large model can be a large language model, a large visual model, a multimodal large model, or a basic scientific large model, etc. For example, the preset large model can be a large language model, specifically the Llama-2-7B (an open-source large language model) model. During training, the Megatron framework can be used. Megatron is a large-scale deep learning framework based on PyTorch (an open-source Python machine learning library) designed for training and deploying large-scale language models. It successfully achieves efficient training of ultra-large-scale language models through a combination of model parallelism, data parallelism, and distributed communication.
[0051] To implement the embodiments of the present invention, a GPU to be tested can be installed on the motherboard of a computer or server to iteratively train a pre-defined large model. The number of training iterations can be set as needed; for example, 1000 iterations can be performed.
[0052] Training data can come from publicly available datasets. If the pre-defined large model is a large language model, the training data can be natural language text; if the pre-defined large model is a large visual model, the training data can be images, etc.
[0053] Before training, the training data can be processed using preprocessing tools provided by the Megatron framework to make the processed data readable by the predefined large model. In the specific implementation, a GPU testing program for executing this embodiment of the invention can be developed. This program can include a training data loading module, which can transmit the training data in the dataset to the large model training module in a predetermined order before each iteration. This module ensures that the training data is provided to the large model training module in the same order during each test, thus eliminating the possibility of different model losses due to different training data during subsequent comparative analysis of model loss.
[0054] In one specific implementation, the GPU testing program may further include a large model training module for automatically training a large model. This module automatically terminates after a preset number of iterations. The GPU testing program may also include a real-time loss value acquisition module, which automatically collects and saves the model loss corresponding to each iteration after the large model training module completes each iteration. After the large model training is complete, this module obtains a preset number of ordered model loss data points. This module stores these ordered model loss data points sequentially into a CSV (Comma-Separated Values) file, allowing testers to easily query intermediate results as needed.
[0055] Regarding step S102, the reference GPU can be a GPU that has been used for a long time and has achieved good training results when training large models, or a GPU produced by a mainstream manufacturer. Mainstream manufacturers' GPUs have high floating-point precision and relatively small rounding errors when processing floating-point operations. Furthermore, mainstream manufacturers usually have more mature hardware optimizations. Differences in GPU floating-point precision and different hardware optimization strategies often lead to differences in the actual computational precision during large model training. For example, the reference GPU could be an A100.
[0056] The aforementioned GPU testing program may also include an automatic loss value analysis module. The real-time loss value acquisition module can trigger the automatic loss value analysis module after completing model loss acquisition. The automatic loss value analysis module can have the aforementioned reference sequence built-in. Furthermore, the automatic loss value analysis module can automatically load the model loss acquired by the real-time loss value acquisition module, compare and analyze it with the reference sequence to obtain difference representation values, and then generate the test results for the GPU under test based on these difference representation values.
[0057] In this embodiment, the sequence composed of the target model loss can be compared with the reference sequence as a whole to obtain the difference characterization value. For example, in one implementation, the DTW (Dynamic Time Warping) algorithm can be used to calculate the difference between the two sets of sequences to obtain the difference characterization value.
[0058] In another implementation, the difference between the sequence comprising the target model loss and the reference sequence includes:
[0059] For each iteration, the relative deviation between the target model loss of that iteration and the model loss of that iteration in the reference sequence is calculated, and the relative deviation of each iteration is used as the difference characterization value between the sequence composed of the target model loss and the reference sequence.
[0060] In other words, the model loss obtained by training using the GPU under test is compared sequentially with the model loss obtained by training using the reference GPU. Specifically, the relative deviation between the target model loss in the first iteration and the model loss obtained in the first iteration of the reference sequence is calculated; the relative deviation between the target model loss in the second iteration and the model loss obtained in the second iteration of the reference sequence is calculated; and so on, until the relative deviation between the target model loss in the preset number of iterations and the model loss obtained in the preset number of iterations of the reference sequence is calculated.
[0061] There are several ways to reflect the relative deviation between the losses of two models; for example, one can calculate the difference between the losses of the two models. In one specific implementation, the relative deviation for each iteration can be calculated based on the following formula:
[0062] deviation=|loss1-loss0| / loss0;
[0063] Where deviation is the relative deviation in this iteration; loss1 is the target model loss in this iteration; and loss0 is the model loss in the reference sequence for this iteration. The above formula can effectively characterize the differences in model loss between each iteration.
[0064] Since the number of iterations performed on the preset large model using the GPU under test and the reference GPU is the same, calculating the relative deviation between the target model loss of each iteration and the model loss of the reference sequence for that iteration can conveniently reflect the difference in training performance between the large model of the GPU under test and the reference GPU.
[0065] For step S103, the test results of the GPU to be tested can be generated based on the following principles: the greater the difference between the sequence of the target model loss and the reference sequence, the worse the large model training effect of the GPU to be tested is; the smaller the difference represented by the difference value, the better the large model training effect of the GPU to be tested is.
[0066] In this embodiment, training data is input into the large model trained by the GPU under test and the large model trained by the reference GPU in the same order. This can eliminate the factor that different training data will cause different model losses. Then, based on the difference between the model loss obtained by the GPU under test and the model loss obtained by the reference GPU in each iteration, the GPU under test is evaluated. When faced with a wide variety of GPUs, it is possible to select a GPU whose model loss during the training iteration process is very similar to that of mainstream GPUs. This makes it easier for technicians to understand the training effect of the GPU and select a suitable GPU.
[0067] In one embodiment of the present invention, such as Figure 2 As shown, the test results generated based on the difference characterization values for the GPU under test can include:
[0068] S1031, determine the number of iterations with a relative deviation less than a preset threshold, and use them as the number to be utilized;
[0069] S1032 generates test results for the GPU under test based on the number of GPUs to be utilized.
[0070] The preset threshold can be set based on experience and needs, for example, it can be 1%. In this embodiment, the fewer the number of models to be used, the worse the large model training effect of the GPU under test is determined, and the more the number of models to be used, the better the large model training effect of the GPU under test is determined, thereby generating the test results of the GPU under test.
[0071] Specifically, the test results generated based on the number of GPUs to be utilized can include:
[0072] Based on the pre-defined correspondence between multiple number ranges and test results, the test result corresponding to the number range to which the number to be used belongs is determined, and this result is used as the test result of the GPU to be tested.
[0073] For example, qualitative test results can be given based on the percentage of epochs where the model loss deviation exceeds 1%. If the percentage of epochs where the model loss deviation exceeds 1% is less than or equal to 1%, the training performance of the GPU under test in large model training is determined to be excellent; if the percentage is greater than 1% but less than or equal to 3%, the training performance of the GPU under test in large model training is determined to be good; if the percentage is greater than 3% but less than or equal to 5%, the training performance of the GPU under test in large model training is determined to be average; and if the percentage is greater than 5%, the training performance of the GPU under test in large model training is determined to be poor.
[0074] In this embodiment, by determining the number of iterations with a relative deviation less than a preset threshold as the number to be utilized, and then generating the test results of the GPU to be tested based on the number to be utilized, the difference between the large model training effect of the GPU to be tested and the reference GPU can be effectively reflected, which makes it easier for technicians to understand the training effect of the GPU and then select a suitable GPU.
[0075] In one embodiment of the present invention, such as Figure 3 As shown, before using the GPU under test to train a pre-defined large model with a pre-defined number of iterations using training data in a fixed order, the method also includes:
[0076] S301, Initialize the preset large model using a preset random seed; wherein, the model loss contained in the reference sequence is obtained by training the preset large model using the preset random seed.
[0077] By setting random seeds for Python (a programming language), NumPy (Numerical Python, an open-source Python scientific computing library), and PyTorch, and deterministic settings for CUDA (Compute Unified Device Architecture, a general-purpose parallel computing platform and programming model), the random seed can be fixed to a preset random seed. Fixing the random seed eliminates random variations during testing, ensuring a consistent initial state for training large models. This provides a basis for comparative analysis of model losses obtained during training, thus ensuring the accuracy of test results.
[0078] Furthermore, weight data from a pre-set training weight data file can also be loaded. Loading the pre-set weight data ensures that the initial weights are consistent for each test. Since the initial weights of the large model are also fixed after the random seed is fixed, the step of loading the pre-set weight data can be omitted when initializing the pre-set large model using a pre-set random seed.
[0079] S302, disable the specified training function of the preset large model; wherein, the specified training function is: randomly deactivate neurons in the preset large model during the training process; the model loss contained in the reference sequence is obtained by training with the specified training function disabled.
[0080] The specified training function can be dropout (random deactivation). Dropout is a regularization technique that temporarily sets the output of neurons in a neural network to 0 with a certain probability during training. In the training of large language models, dropout is usually enabled to improve the final performance and generalization ability of the model. Although this embodiment uses a large model training module, the purpose is not to train a high-performance, highly generalizable model, but to use the loss value during the large model training process to evaluate the performance of the tested GPU in the large model training. Therefore, to prevent the randomness and uncertainty brought about by enabling dropout, this embodiment needs to ensure that dropout is disabled.
[0081] Specifically, when training large models using the Megatron framework, you can check the dropout switch and turn it off if it is on.
[0082] In this embodiment, a preset large model is initialized using a preset random seed, and / or a specified training function of the preset large model is disabled. The specified training function is to randomly deactivate neurons in the preset large model during training. The model loss contained in the reference sequence is also obtained by training with the preset large model initialized using the preset random seed and / or the specified training function disabled. This can further control the randomness of the large model trained by the GPU under test and the reference GPU to ensure the accuracy of the test results.
[0083] The method provided by this invention can effectively determine the training performance of different GPUs in large models, providing more basis and reference for GPU selection, thereby effectively reducing the probability of problems arising from using GPUs whose training performance does not meet the requirements in large model training tasks. Using GPUs with excellent training performance for large model training ensures that, under the same control of other factors, the model loss during the intermediate process is very similar to that of mainstream GPUs, resulting in a model that is not significantly different from that trained using mainstream GPUs, effectively reducing the instability of model training. Furthermore, since GPUs with excellent training performance, under the same control of other factors, have a model loss during the intermediate process that is similar to that of mainstream GPUs, the model performance trained by them is also close to that of mainstream GPUs, making it less prone to problems with insufficient generalization ability.
[0084] Based on the same inventive concept, embodiments of the present invention also provide a GPU testing device, such as... Figure 4 As shown, the device includes:
[0085] The model training module 401 is used to train a preset large model with training data in a fixed order using the GPU to be tested for a preset number of iterations, and to obtain the model loss of the preset large model in each iteration, which is used as the target model loss.
[0086] The difference acquisition module 402 is used to obtain the difference representation value between the sequence of the target model loss and the reference sequence; wherein, the reference sequence includes: the model loss obtained by using the reference GPU to train the preset large model with the training data in the fixed order for the preset number of rounds of iteration;
[0087] The result generation module 403 is used to generate test results for the GPU under test based on the difference characterization values.
[0088] In one embodiment of the present invention, the difference acquisition module 402 is specifically used for:
[0089] For each iteration, the relative deviation between the target model loss of that iteration and the model loss of that iteration in the reference sequence is calculated, and the relative deviation of each iteration is used as the difference characterization value between the sequence composed of the target model loss and the reference sequence.
[0090] In one embodiment of the present invention, the difference acquisition module 402 is specifically used to calculate the relative deviation of each iteration based on the following formula:
[0091] deviation=|loss1-loss0| / loss0;
[0092] Where deviation is the relative deviation of this iteration; loss1 is the target model loss of this iteration; and loss0 is the model loss of this iteration in the reference sequence.
[0093] In one embodiment of the present invention, the result generation module 403 includes:
[0094] The number determination submodule is used to determine the number of iterations in which the relative deviation is less than a preset threshold, as the number to be utilized;
[0095] The test result generation submodule is used to generate test results for the GPU under test based on the number of GPUs to be utilized.
[0096] In one embodiment of the present invention, the test result generation submodule is specifically used to determine the test result corresponding to the number interval to which the number to be used belongs, based on the correspondence between a plurality of preset number intervals and test results, and use it as the test result of the GPU to be tested.
[0097] In one embodiment of the present invention, the device further includes:
[0098] An initialization module is used to initialize the preset large model with a preset random seed before the model training module 401 uses the GPU under test to train the preset large model with training data in a fixed order for a preset number of iterations; wherein, the model loss contained in the reference sequence is obtained by training the preset large model using the preset random seed.
[0099] And / or,
[0100] The function shutdown module is used to disable a specified training function of the preset large model before the model training module 401 uses the GPU under test to perform a preset number of iterations of training data in a fixed order on the preset large model; wherein, the specified training function is: randomly deactivating neurons in the preset large model during the training of the preset large model; the model loss contained in the reference sequence is obtained by training with the specified training function disabled.
[0101] This invention also provides an electronic device, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.
[0102] Memory 503 is used to store computer programs;
[0103] When the processor 501 executes the program stored in the memory 503, it implements any of the GPU testing methods described above.
[0104] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0105] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0106] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0107] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0108] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described GPU testing methods.
[0109] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of any of the GPU testing methods described above.
[0110] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0111] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0112] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.
Claims
1. A GPU testing method, characterized in that, The method includes: Using the GPU under test, a preset large model is trained on a preset number of iterations with training data in a fixed order, and the model loss of the preset large model in each iteration is obtained as the target model loss. Obtain the difference representation value between the sequence of the target model loss and the reference sequence; wherein, the reference sequence includes: the model loss obtained by performing the preset number of iterations of training on the preset large model using the training data in the fixed order using a reference GPU; The test results for the GPU under test are generated based on the difference characterization values.
2. The method according to claim 1, characterized in that, The step of obtaining the difference between the sequence comprising the loss of the target model and the reference sequence includes: For each iteration, the relative deviation between the target model loss of that iteration and the model loss of that iteration in the reference sequence is calculated, and the relative deviation of each iteration is used as the difference characterization value between the sequence composed of the target model loss and the reference sequence.
3. The method according to claim 2, characterized in that, The relative deviation for each iteration is calculated using the following formula: deviation=|loss1-loss0| / loss0; Where deviation is the relative deviation of this iteration; loss1 is the target model loss of this iteration; and loss0 is the model loss of this iteration in the reference sequence.
4. The method according to claim 2, characterized in that, The process of generating test results for the GPU under test based on the difference characterization values includes: The number of iterations with a relative deviation less than a preset threshold is determined as the number to be utilized; The test results for the GPU under test are generated based on the number of GPUs to be utilized.
5. The method according to claim 4, characterized in that, The process of generating test results for the GPU under test based on the number of GPUs to be utilized includes: Based on the pre-defined correspondence between multiple number intervals and test results, the test result corresponding to the number interval to which the number to be used belongs is determined, and used as the test result of the GPU to be tested.
6. The method according to any one of claims 1-5, characterized in that, Before using the GPU under test to train a pre-defined large model on a pre-defined number of iterations with training data in a fixed order, the method further includes: The preset large model is initialized using a preset random seed; wherein the model loss contained in the reference sequence is obtained by training the preset large model using the preset random seed. And / or, The specified training function of the preset large model is to disable the specified training function, wherein the specified training function is to randomly deactivate neurons in the preset large model during the training process; the model loss contained in the reference sequence is obtained by training with the specified training function disabled.
7. A GPU testing device, characterized in that, The device includes: The model training module is used to train a preset large model with training data in a fixed order using the GPU under test for a preset number of iterations, and to obtain the model loss of the preset large model in each iteration, which is used as the target model loss. A difference acquisition module is used to obtain the difference representation value between the sequence of the target model loss and the reference sequence; wherein, the reference sequence includes: the model loss obtained by using a reference GPU to train the preset large model with the training data in the fixed order for the preset number of iterations; The result generation module is used to generate test results for the GPU under test based on the difference characterization values.
8. The apparatus according to claim 7, characterized in that, The difference acquisition module is specifically used for: For each iteration, the relative deviation between the target model loss of that iteration and the model loss of that iteration in the reference sequence is calculated, and the relative deviation of each iteration is used as the difference characterization value between the sequence composed of the target model loss and the reference sequence.
9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-6.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.