Precision test method, electronic equipment and storage medium
By determining the target module to be tested in the model and its calculation unit and using the output of the previous module obtained by model training as the reference input for accuracy testing, the problem of being unable to accurately locate the error module in the model in the prior art is solved, and high reliability and accuracy accuracy testing is achieved.
Patent Information
- Application Number
- CN202411751348.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-18
AI Technical Summary
The prior art cannot accurately locate the modules causing errors in the model, resulting in the accuracy alignment that can only measure end-to-end model errors, and it is impossible to accurately locate the accuracy problems caused by a certain module in the model.
By determining the target module to be tested in the model and its running calculation unit, using the output of the previous module obtained by model training as the reference input of the target module, the parallel control calculation unit performs forward propagation and back propagation of the target module, obtains the test output, and then determines the test accuracy of the target module.
Independent accuracy testing for a single module is realized, the accuracy error of the preamble module is avoided from entering the testing process, the reliability and accuracy of the accuracy test are improved, and it is suitable for models with large parameter scales.
Smart Images
Figure CN119645865A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an accuracy testing method, electronic equipment and storage medium. Background Art
[0002] In the field of artificial intelligence, model adaptation, especially large language models (LLM) model adaptation, is a crucial link. In order to meet the model adaptation requirements, a lot of precision alignment work is usually required.
[0003] Currently, precision alignment includes module-based precision alignment solutions. In the module-based precision alignment solution, each module in the model can be used as the basic unit for comparison, and the accuracy can be compared one by one. However, in this solution, the input of each module comes from the output of the previous module. If the output of the previous module fails to achieve precision alignment, then there will be differences in the input of the current module, and the output of the current module will naturally fail to achieve precision alignment. Therefore, the above precision alignment method can only measure the end-to-end model error, and cannot accurately locate the precision problem caused by a module in the model. Summary of the invention
[0004] The present invention provides an accuracy testing method, an electronic device and a storage medium, which are used to solve the defect in the related art that a module causing an error in a model cannot be accurately located.
[0005] The present invention provides an accuracy testing method, comprising: Determining a target module to be tested in the model, and determining each computing unit for running the target module; Based on the reference input of the target module, the computing units are controlled in parallel to perform forward propagation and backward propagation on the target module to obtain test outputs of the computing units, wherein the reference input is the training output of a previous module of the target module, and the training output of the previous module is obtained by training the model; Determining the test accuracy of each computing unit based on a benchmark output of the target module and a test output of each computing unit, wherein the benchmark output is a training output of the target module and the training output of the target module is obtained by training the model; Based on the test accuracy of each computing unit, the test accuracy of the target module is determined.
[0006] According to an accuracy testing method provided by the present invention, the step of determining each computing unit used to run the target module includes: At least one computing communication group for running the target module is determined, the computing communication group comprising a plurality of computing units communicating with each other.
[0007] According to an accuracy test method provided by the present invention, based on the reference input of the target module, the computing units are controlled in parallel to perform forward propagation and reverse propagation on the target module to obtain the test output of each computing unit, including: Based on the input data in the reference input of the target module and the local weights in the reference input allocated to the computing units in each computing communication group, the computing units are controlled in parallel to perform forward propagation and reverse propagation on the target module to obtain the test outputs of the computing units; The local parameters of all computing units in the computing communication group constitute the reference weight of the target module.
[0008] According to an accuracy test method provided by the present invention, based on the reference input of the target module, the computing units are controlled in parallel to perform forward propagation and reverse propagation on the target module to obtain the test output of each computing unit, which also includes: Determine the target test rounds for the target module to be tested; The benchmark input and benchmark output of the target module under the target test round are determined, and the model training input when the model is trained under different test rounds is different.
[0009] According to an accuracy testing method provided by the present invention, the test accuracy of the target module is determined based on the test accuracy of each computing unit, and then further includes: When the test accuracy of the target module does not meet the accuracy requirement and the target test round exceeds the preset round, determining a test round adjacent to the target test round as a new target test round; An accuracy test is performed based on the benchmark input and benchmark output of the target module in the new target test round.
[0010] According to an accuracy test method provided by the present invention, based on the reference input of the target module, the computing units are controlled in parallel to perform forward propagation and reverse propagation on the target module to obtain the test output of each computing unit, which also includes: Based on the environmental variables and hyperparameters when the model is trained, the environmental variables and hyperparameters of each computing unit for forward propagation and backward propagation of the target model are configured.
[0011] According to an accuracy testing method provided by the present invention, the reference input includes input data and a reference weight, and the reference output includes output data and a weight gradient.
[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned accuracy testing methods is implemented.
[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the accuracy testing method described in any one of the above is implemented.
[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the accuracy testing method described above is implemented.
[0015] The precision testing method, electronic device and storage medium provided by the present invention realize precision testing for the target module by using the output of the previous module obtained by model training as the reference input of the target module, thereby realizing independent precision testing for a single module, and the test process will not introduce the precision error of the preceding module, and can ensure the reliability and accuracy of more fine-grained precision testing. In addition, in the process of precision testing for the target module, each computing unit of the target module is controlled in parallel to perform forward propagation and reverse propagation on the target module, which can adapt to the precision testing of modules in models with larger parameter scales, and can realize the designation of computing units, thereby further improving the adaptability and flexibility of the precision testing method. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1 It is a schematic diagram of module-based accuracy comparison provided by related technologies.
[0018] Figure 2 This is one of the flow charts of the accuracy testing method provided by the present invention.
[0019] Figure 3 It is a schematic diagram of module testing under the distributed parallel strategy provided by the present invention.
[0020] Figure 4 It is a schematic diagram of module operation provided by the present invention.
[0021] Figure 5 It is a configuration schematic diagram of the accuracy test provided by the present invention.
[0022] Figure 6 This is the second flow chart of the accuracy testing method provided by the present invention.
[0023] Figure 7 It is a structural schematic diagram of the accuracy testing device provided by the present invention.
[0024] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] In the field of artificial intelligence, model adaptation, especially model adaptation of large language models, is a crucial link. In order to meet the needs of model adaptation, a lot of precision alignment work is usually required.
[0027] The accuracy alignment work generally uses the real data obtained by training the model on the benchmark hardware as the benchmark data, and then uses the input in the benchmark data as the input data, obtains the output of the operator by calling the operator calculation in the model, and then obtains the absolute error or relative error of the output by comparing the output of the operator with the output in the benchmark data. For operators with large errors, the operator can be repaired by fine-tuning parameters, so that the accuracy of the repaired model can be consistent with the accuracy running on the benchmark hardware.
[0028] By performing precision alignment, we can ensure that the performance of the model is consistent in different operating environments, thereby improving the reliability of the model and making it easier to reuse the model in different operating environments, effectively reducing the complexity and cost of model deployment.
[0029] In related technologies, common precision alignment methods for models include the following two: One is the precision alignment based on torch API operators. torch API is an API based on the PyTorch framework and is an open source machine learning tool set. In this way, the precision of individual operators in the model can be compared one by one through the torch API to ensure the calculation accuracy of the model during the adaptation process. However, this method is not suitable for the adaptation process of large language models. In large language models, in order to improve computing efficiency and performance, single operators in the model are often fused, that is, multiple single operators are merged into one fused operator. This fusion operation makes it difficult to accurately align the number and position of operators between the adaptation model (ModelAdaption) and the benchmark model (Golden), thus increasing the difficulty of precision alignment.
[0030] The other is based on the accuracy alignment of torch.nn.Module. torch.nn.Module is a core class used to build neural network models in the PyTorch deep learning framework. For ease of description, torch.nn.Module is referred to as module below. In this way, each module in the model can be used as the basic unit for comparison and accuracy comparison can be performed one by one. For example, Figure 1 This is a schematic diagram of module-based accuracy comparison provided by related technologies, such as Figure 1 As shown in Figure 1, the input and output shapes of each module in the adaptation model can be aligned with the benchmark model (Golden) to facilitate comparison and calculation of accuracy errors. Figure 1 In the test, the adapted model and the baseline model can be compared module by module. Specifically, the attention module in layer 0 of the adapted model can be compared with the attention module in layer 0 of the baseline model, and the multi-layer perceptron (MLP) module in layer 0 of the adapted model can be compared with the multi-layer perceptron module in layer 0 of the baseline model.
[0031] For example, Figure 1 In the model, precision alignment is performed module by module in the order of layer 0 → layer 1 → layer 2 → ... → layer N, where the input of each module comes from the output of the previous module.
[0032] Therefore, the module-based precision alignment method can achieve the alignment of input shapes and output shapes between modules. Here, the input shape and output shape refer to the shapes of input data and output data respectively, and the shapes are specifically used to describe the dimension and size of the data. It can be understood that the input shape alignment and output shape alignment between modules facilitates the comparison of data and the calculation of precision errors between modules, thereby better solving the alignment problem caused by operator fusion in the precision alignment method based on the torch API operator.
[0033] However, in the implementation of this method, if the output of the previous module fails to achieve precision alignment, then the input of the current module in the adaptation model will be different from that of the baseline model, and the output of the current module will naturally fail to achieve precision alignment.
[0034] Therefore, although the module-based precision alignment method can measure the end-to-end model error, it cannot accurately locate the precision problems caused by a certain module in the model.
[0035] In view of the above problems, an embodiment of the present invention provides an accuracy testing method. Figure 2 It is one of the flow charts of the accuracy testing method provided by the present invention, such as Figure 2 As shown, the method includes: Step 210, determining a target module to be tested in the model, and determining each computing unit for running the target module.
[0036] Specifically, the target module is the module in the model to be tested for accuracy. The target module can be any module in the model. The number of target modules can be one or more. One or more modules can be selected from the model as the target module, or each module in the model can be used as the target module. The embodiment of the present invention does not specifically limit this.
[0037] After determining the target module, the computing unit used to run the target module when performing the accuracy test on the target module can be determined. The computing unit running the module here is the computing unit to be tested, that is, the computing unit in the operating environment that the model needs to adapt. Here, the computing unit running the module can be an artificial intelligence chip, such as a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), an NPU (Neural network Processing Unit), a DPU (Deep learning Processing Unit), an APU (Accelerated Processing Unit), and a GPGPU (General-Purpose computing on Graphics Processing Unit), which is not specifically limited in the embodiments of the present invention.
[0038] In addition, since the parameter scale of some models is large, a single computing unit cannot carry the complete parameters of the target module. It is necessary to adopt a distributed parallel strategy to divide the complete parameters of the target module into multiple parts, and implement the operation of the target module in parallel through multiple computing units, where each computing unit is responsible for the calculation of a part of the target module, thereby reducing the memory load of each computing unit. This operation mode is also called tensor model parallelism. Therefore, in an embodiment of the present invention, the number of computing units used to run the target module can be one or more.
[0039] Step 220, based on the benchmark input of the target module, control the computing units in parallel to perform forward propagation and backward propagation on the target module to obtain the test output of each computing unit, wherein the benchmark input is the training output of the previous module of the target module, and the training output of the previous module is obtained by training the model.
[0040] Specifically, for a target module in a model, a reference input and a reference output of the target module may be acquired in advance.
[0041] Here, the reference input and reference output are obtained by training the model. The training of the model here can be implemented by a default reliable computing unit, for example, it can be implemented by a computing unit of a certain specified brand or specified model that has been verified, and the input and output of each module obtained by the training here can be recorded as the training input and training output of each module. In the model, there is a relationship between input and output between each module, which is specifically reflected in that, for any module, the output of the previous module of the module is the input of the module, and the output of the module is the input of the next module of the module. Therefore, for each module, the training output of the previous module of each module is equal to the training input of each module, and the training output of each module is equal to the training input of the next module of each module. Therefore, the training input and training output of each module can be used as the reference value of the input and output of each module for precision testing on other computing units, that is, the reference input and reference output. It can also be understood that the training output of the previous module of each module is used as the reference input of each module; the training output of each module is used as the reference output of each module. It can be understood that model training occurs before accuracy testing, that is, before accuracy testing, the benchmark input and benchmark output of each module in the model have been determined.
[0042] In the process of training the model, you can specify the sample data required for model training and use the sample data as input for model training. After the sample data is input into the model, the model training is achieved by performing forward propagation and back propagation on the model.
[0043] Among them, forward propagation and backpropagation are two core concepts in neural networks, which together constitute the basis of neural network model training. Forward propagation refers to the process of starting from the input data, calculating and transmitting data layer by layer through each layer of the neural network model, and finally obtaining the output result of the model; backpropagation refers to the process of reversely calculating the gradient through the chain rule according to the difference between the model's prediction result and the true label, and propagating the gradient from the output layer back to each layer of the neural network model to update the model's parameters.
[0044] It can be understood that, through the forward propagation of the model, the input of each module in the model in the forward propagation can be obtained, that is, the output of the previous module of each module in the model in the forward propagation, thereby, the reference input of each module in the model can be determined, that is, the output of the previous module of each module in the model in the forward propagation, that is, the reference input of each module is the training output of the previous module. In addition, through the forward propagation and backward propagation of the model, the output of each module in the model in the forward propagation, as well as the calculated differences, gradients, etc. of each module in the model in the backward propagation can also be obtained, all of which can be used as the reference output of each module in the model, that is, the reference output of each module is the training output of the module itself.
[0045] It is clear that the sample data is real training data, and the benchmark input and benchmark output of each module obtained based on this data are all obtained through real model training and can reflect the actual operating conditions of each module.
[0046] On this basis, after determining the target module to be tested and the computing units used to run the target module, the benchmark input of the target module can be obtained from the pre-stored benchmark inputs of each module, and the benchmark input of the target module can be used as the real input for accuracy testing of the target module, controlling each computing unit to perform forward propagation and backward propagation on the target module.
[0047] In this process, the benchmark input of the target module is used as the test input of the target module, and the forward propagation and backward propagation of the target module under the test input are realized in parallel through the computing units, thereby obtaining the test output of each computing unit for the target module. The test output here may include the output of the forward propagation of the target module, and may also include the difference, gradient, etc. calculated by the target module during backward propagation, and each computing unit corresponds to a test output.
[0048] It can be understood that, for the target module, the benchmark input of the target module is the output of the previous module obtained by model training, that is, when testing the target module, the benchmark input of the target module can be directly obtained without running the modules before the target module, thereby avoiding the problem of error accumulation when testing the target module.
[0049] It should be noted that in step 220, for the target module, the reference input applied by the target module during forward propagation and backward propagation is the input of the target module during model training. Therefore, the input of the target module during model training is the same as the input of the target module during precision testing. Since model training is implemented at the computing unit under a default reliable operating environment, and precision testing is implemented at the computing unit under an operating environment to which the model needs to adapt, therefore, under the condition of the same input and different operating environments, there may be errors between the reference output of the target module obtained based on model training and the test output obtained by forward propagation and backward propagation for the target module in step 220.
[0050] In addition, model training is performed in the default reliable operating environment, with forward propagation and backward propagation performed on a model-by-model basis. After the model training is completed, the benchmark input and benchmark output of each module in the model can be obtained for subsequent accuracy testing on a module-by-module basis.
[0051] In step 220, forward propagation and backward propagation are performed on the target module in the operating environment to be adapted. Since each module is independent of each other, there is no model training operation in step 220. After forward propagation and backward propagation are performed on a single target module, the test output of the target module can be obtained, and there may be an error between the test output and the benchmark output of the target module obtained by model training.
[0052] Step 230, based on the benchmark output of the target module and the test output of each computing unit, determine the test accuracy of each computing unit, the benchmark output is the training output of the target module, and the training output of the target module is obtained by training the model.
[0053] Specifically, after obtaining the test output of each computing unit for the target module, the test output of each computing unit can be compared with the benchmark output of the target module obtained in advance through training, thereby determining the test accuracy of each computing unit based on the error between the test output of each computing unit and the benchmark output of the target module. For example, for one of the computing units, at least one of the absolute error, relative error, number of elements with errors greater than the relative error, and proportion of elements with errors greater than the relative error between the test output of the computing unit and the benchmark output of the target module can be used as the test accuracy of the computing unit.
[0054] Step 240: Determine the test accuracy of the target module based on the test accuracy of each computing unit.
[0055] Specifically, the operation of the target module is implemented in parallel by multiple computing units. Therefore, when measuring the test accuracy of the target module, it is also necessary to refer to the test accuracy of each computing unit. Specifically, the test accuracy of each computing unit used to run the target module can be integrated to determine the test accuracy of the target module. For example, the test accuracy of each computing unit can be used to calculate at least one of the maximum value of the absolute error, the maximum value of the relative error, the number of elements with errors greater than the relative error, and the proportion of elements with errors greater than the relative error as the test accuracy of the target module.
[0056] In addition, when each module in the model is used as a target module for accuracy testing, the test accuracy of each module can be obtained, thereby locating the module in the model that causes the error.
[0057] The method provided by the embodiment of the present invention realizes the precision test for the target module by using the output of the previous module obtained by model training as the reference input of the target module, thereby realizing the independent precision test for a single module, and the test process will not introduce the precision error of the previous module, which can ensure the reliability and accuracy of the precision test of finer granularity. In addition, in the process of precision test for the target module, each computing unit of the target module is controlled in parallel to perform forward propagation and reverse propagation on the target module, which can adapt to the precision test of the module in the model with a large parameter scale, and can realize the designation of the computing unit, thereby further improving the adaptability and flexibility of the precision test method.
[0058] Based on the above embodiment, in step 210, determining each computing unit for running the target module includes: At least one computing communication group for running the target module is determined, the computing communication group comprising a plurality of computing units communicating with each other.
[0059] Specifically, under the distributed parallel strategy, the parameters of the target module can be divided into multiple computing units, where the multiple computing units can constitute a computing communication group, and in the case of multi-machine training, the operation of the target module can be implemented by multiple computing communication groups, and no data communication is performed between the computing communication groups.
[0060] Therefore, when determining the computing units for running the target module, specifically, the computing communication group for running the target module can be determined, and the number of computing communication groups running the target module can be one or more. For one computing communication group, the computing units in the computing communication group can communicate with each other, thereby realizing data communication during parallel operation.
[0061] For example, the parameters of the target module can be divided into 8 computing units, and these 8 computing units can constitute a computing communication group. For example, the computing units with the order Rank={0, 1, 2, 3, 4, 5, 6, 7} can constitute a computing communication group. For another example, in the case of multi-machine training, the computing units with the order Rank={0, 1, 2, 3, 4, 5, 6, 7} can constitute a computing communication group, and the computing units with Rank={8, 9, 10, 11, 12, 13, 14, 15} can also constitute a computing communication group. When performing accuracy testing on the target module, one or more computing communication groups can be specified as the computing communication group for running the target module.
[0062] For example, Figure 3 FIG. 1 is a schematic diagram of module testing under the distributed parallel strategy provided by the present invention. Figure 3 As shown, assuming that the computing communication group used to run the target module (Module) includes a total of 8 computing units numbered 0 to 7, the benchmark input Input of the target module can be divided into 8 parts, namely, benchmark input slice 0 to benchmark input slice 7, which are respectively input into the above-mentioned 8 computing units, and the 8 computing units each obtain the complete benchmark input Input through the AllGather communication operation, and then the 8 computing units can run (Operator) the target module in parallel based on the benchmark input Input to obtain their respective test outputs Output.
[0063] Based on any of the above embodiments, in step 220, based on the reference input of the target module, controlling the computing units in parallel to perform forward propagation and reverse propagation on the target module to obtain the test output of the computing units includes: Based on the input data in the reference input of the target module and the local weights in the reference input allocated to the computing units in each computing communication group, the computing units are controlled in parallel to perform forward propagation and reverse propagation on the target module to obtain the test outputs of the computing units; The local parameters of all computing units in the computing communication group constitute the reference weight of the target module.
[0064] Specifically, when performing an accuracy test on a target module, it is necessary not only to apply the input data in the benchmark input of the target module, but also to apply the benchmark weight in the benchmark input of the target module. Here, the benchmark weight of the target module is the initial weight parameter of the target module as a whole when the model is trained. In different training rounds, the benchmark weight of the target module may be different. The determination of the benchmark weight here depends on the training round specified for this accuracy test, that is, the benchmark weight is specifically the initial parameter of the target module as a whole when it is trained under the training round specified for this accuracy test.
[0065] For example, Figure 4 It is a schematic diagram of the module operation provided by the present invention, such as Figure 4 As shown, in order to realize module operation, the input data (Golden Input) and the reference weight (Golden weight) of the module can be loaded into the computing unit used to run the module.
[0066] Under the distributed parallel strategy, the target module runs in parallel at each computing unit in the computing communication group. Therefore, it is necessary to initialize parameters for each computing unit respectively. Specifically, the benchmark weight of the target module can be split into multiple parts of local weights, and then the local weights of each part can be loaded into each computing unit in the corresponding computing communication group. Therefore, based on the input data in the benchmark input of the target module and the local weights of the target module at each computing unit, the forward propagation and reverse propagation of the target module at each computing unit can be executed in parallel.
[0067] It can be understood that since there is no data transmission between the computing communication groups, but data transmission is performed between the computing units within a single computing communication group, the local weight combination of all computing units in each computing communication group constitutes the benchmark weight of the target module.
[0068] Based on any of the above embodiments, before step 220, the method further includes: Determine the target test rounds for the target module to be tested; The benchmark input and benchmark output of the target module under the target test round are determined, and the model training input when the model is trained under different test rounds is different.
[0069] Specifically, when performing an accuracy test on a target module, in addition to specifying a computing unit for running the target module, a training round to be tested can also be specified, where the training round to be tested is recorded as a target test round. It is understandable that the target test round can be any one or more rounds of the model training rounds.
[0070] It can be understood that in multiple rounds of training for the model, the model training input, that is, the sample data, used for model training is different in different training rounds, and the initial parameters of each module in the model are also different. Therefore, in different training rounds, the input and output of each module in the model are also different, and accordingly, the benchmark input and benchmark output of each module determined thereby are also different.
[0071] Therefore, after determining the target test round, it is necessary to extract the benchmark input and benchmark output under the target test round from the benchmark input and benchmark output of each module obtained by the model in multiple rounds of training, so as to facilitate the accuracy test of each computing unit of the target module under the target test round.
[0072] Based on any of the above embodiments, after step 240, the method further includes: When the test accuracy of the target module does not meet the accuracy requirement and the target test round exceeds the preset round, determining a test round adjacent to the target test round as a new target test round; An accuracy test is performed based on the benchmark input and benchmark output of the target module in the new target test round.
[0073] Specifically, the preset round is used to divide the rounds of initial training and mid-term training in the multi-round training of the model, that is, the rounds before the preset round belong to the initial training, and the rounds after the preset round belong to the mid-term training. The difference between initial training and mid-term training is that in the initial training, factors such as the model's initialization weights and learning rate have a greater impact on the training loss, so it is necessary to focus on comparing the forward and reverse data in the test output generated by the test with the benchmark output. For example, the rounds at this time can be {1,2,3,4,5}, etc. During mid-term training, the training loss is more susceptible to abnormal samples in the training samples, or samples within a special data range.
[0074] In response to this situation, when it is determined that the test accuracy of the target model in the target test round does not meet the accuracy requirements, for example, due to a sharp increase in the loss in the test output, resulting in the problem of non-convergence of the training accuracy, it is not ruled out that the test accuracy of the target model does not meet the accuracy requirements due to abnormal samples in the test input. Therefore, the test round adjacent to the target test round can be used as a new target test round. For example, the target test round step=7033, the test round steps={7031, 7032, 7033, 7034, 7035} adjacent to the target test round can be used as a new target test round, and the target module is tested for accuracy under the new target test round, thereby determining whether the previous accuracy test of the target module did not meet the accuracy requirements due to abnormal samples.
[0075] Based on any of the above embodiments, before step 220, the method further includes: Based on the environmental variables and hyperparameters when the model is trained, the environmental variables and hyperparameters of each computing unit for forward propagation and backward propagation of the target model are configured.
[0076] Specifically, before controlling each computing unit in parallel to perform forward propagation and backward propagation on the target module, it is necessary to configure environment variables and hyperparameters for each computing unit. Here, for the configuration of environment variables and hyperparameters, the environment variables and hyperparameters used for model training can be reused, thereby ensuring that when unit testing is performed on the target module, the behavior of each computing unit can be consistent with the behavior of model training, thereby avoiding the inconsistency between the behavior of each computing unit and the behavior of model training, resulting in different computing branches of the operator in the target module being called, resulting in the generation of precision test results that are meaningless for the precision check of model training.
[0077] In order to configure the environment variables and hyperparameters at each computing unit, the sources of the environment variables and hyperparameters can be set in the process of parallel controlling each computing unit to perform forward propagation and backward propagation on the target module.
[0078] For example, the torchrun tool that runs the PyTorch training script in a distributed environment can be used to parallelly control the forward propagation and backpropagation processes of each computing unit on the target module, so as to facilitate the accuracy test of the target module.
[0079] In torchrun, the distributed hyperparameters are specified by distributed_args. The environment variables for the accuracy test of the target module reuse the environment variables env.sh in the end-to-end training script of the model. The training hyperparameters for the accuracy test of the target module reuse the hyperparameters args.sh in the end-to-end training script of the model. The size of nproc_per_node in the distributed hyperparameter distributed_args is the parallel size of the tensor model TP_SIZE, that is, the number of computing units contained in a set of computing communication groups. Distributed_args specifically includes three parameters, among which nproc_per_node is the number of parallel processes, specifically TP_SIZE; nodes is the number of nodes, which can be set to 1, indicating that the test for the target module is performed on a single node, and the node here can also be understood as a computing communication group; node_rank is the ranking of the nodes, and in the case of a single node, it is usually taken as 0.
[0080] After the torchrun command is executed, the processes of each computing unit can execute the accuracy test of each target module in parallel, and then give the test accuracy of each target module in turn according to the order of each target module in the model.
[0081] For example, Figure 5 It is a schematic diagram of the configuration of the accuracy test provided by the present invention, such as Figure 5 As shown, when modules 1 to e4 are respectively used as target modules, the parallel processes in torchrun can be configured (distributed torchrun) based on the distributed hyperparameter distributed_args, the environment variable env.sh during model training, and the hyperparameter args.sh during model training, thereby achieving the accuracy test of modules 1 to 4.
[0082] Based on any of the above embodiments, the reference input includes input data and reference weights, and the reference output includes output data and weight gradients.
[0083] Specifically, in the benchmark input, the input data, that is, the data used as the input of the target module, can be recorded as forward_input, and the benchmark weight, that is, the parameter weight of the target module, can be recorded as weight; in the benchmark output, the output data, that is, the output obtained by the target module after completing the forward propagation, can be recorded as forward_output, and the weight gradient, that is, the gradient obtained by the target module after completing the back propagation, can be recorded as weight_grad.
[0084] In addition, the benchmark output may also include a reverse input, that is, the input of the target module when performing backward propagation, which may be recorded as backward_input; in addition, the benchmark output may also include a reverse output, that is, the output of the target module when performing backward propagation, which may be recorded as backward_output.
[0085] Based on any of the above embodiments, Figure 6 This is the second flow chart of the accuracy testing method provided by the present invention. Figure 6 The model shown has modules such as Token Embedding, Norm Layer, Attention Layer, Feed-Forward Neural Network (FFN), Linear Layer and Decoder.
[0086] In the end-to-end training of the model, you can obtain the module benchmark data (Module Golden Data) of each module in the model, which contains the forward propagation and backward propagation data, including the input (input), weight (weight), output (output) applied during forward propagation, and the input (grad_input) and output gradient (grad_output) during backward propagation. The module benchmark data can be split into benchmark input and benchmark output, and each module has its own benchmark input and benchmark output.
[0087] Subsequently, each module (for example, module 1 to module N) can be used as a target module, and a unit test (Unit Test) can be performed on the target module based on the benchmark input of the target module. The test output output by the unit test can be compared with the benchmark output to obtain the test accuracy (Unit Test Report) of the target module.
[0088] Based on any of the above embodiments, the accuracy test method may further include a step of obtaining a reference input and a reference output, which may also be understood as a training step for the model, and includes: Specify the computing unit used to run the model, and the sample data for the rounds of model training. Here, the computing unit can be identified by dump-ranks, and the round can be identified by dump-steps.
[0089] In addition, you can also specify the path for data export, which can be recorded as dump-dir.
[0090] In this way, the parameters of model training can be initialized based on the specified computing units and sample data.
[0091] After initialization, model training can be implemented to obtain benchmark data for each module, including benchmark input and benchmark output. Here, the benchmark data can be stored in the directory format of step→rank→realdata, where step represents the round, rank represents the computing unit, and realdata represents the training data under the specified round and specified computing unit, which can specifically include the weights, inputs and outputs of each module's forward propagation, as well as the inputs and weight gradients of each module's reverse propagation.
[0092] Based on any of the above embodiments, a precision testing method may include the following steps: First, the target module to be tested can be determined from the model, and the computing unit used to run the target module, the target test round of the target module to be tested, and the benchmark input and benchmark output under the target test round can be determined.
[0093] Secondly, the environment variables and hyperparameters for forward propagation and backward propagation of the computing units running the target module can be configured. Specifically, the environment variables and hyperparameters of the accuracy test of the target module can be configured by reusing the environment variables and hyperparameters of the model during training, thereby ensuring that the behavior of each computing unit of the target module during the accuracy test is consistent with that during model training.
[0094] The target module can then be initialized and configured, where configuration options may include the benchmark input, the keyword pattern of the file where the benchmark output is located, and the error threshold tolerance applied when comparing the benchmark output with the test output.
[0095] Next, the benchmark weights in the benchmark input are applied to initialize the weights of the target module. In addition, the input data in the benchmark input is used as the input of the target module, and each computing unit is controlled to perform forward propagation and backward propagation on the target module in parallel, thereby achieving distributed collaborative accuracy testing of the target module.
[0096] The test output obtained by forward propagation and reverse propagation of the target module by each computing unit can be compared with the reference output, thereby obtaining at least one of the absolute error AbsErr, the relative error RelativeErr, the number of elements with errors greater than the relative error mismatch_cnt, and the proportion of elements with errors greater than the relative error mismatch_percent as the test accuracy of each computing unit. In addition, based on the test accuracy of each computing unit, the test accuracy of the target module can be determined, which can specifically include at least one of the maximum value of the absolute error MaxAbsErr, the maximum value of the relative error MaxRelativeErr, the number of elements with errors greater than the relative error mismatch_cnt, and the proportion of elements with errors greater than the relative error mismatch_percent.
[0097] In the method provided by the embodiment of the present invention, the benchmark input and benchmark output of each module in the model are derived from the real model training, which makes the test accuracy based on the benchmark input and benchmark output more able to reflect the real accuracy error. And in the model training scenario, the accuracy test for the target module includes forward propagation and reverse propagation, and the data of the computing unit and round can be specified for comparison, which greatly improves the flexibility of the accuracy test. Moreover, when the model training strategy includes tensor parallelism or sequence parallelism, the module accuracy test of multiple computing units can be realized in a distributed and collaborative manner, and the accuracy comparison of multiple computing units with the benchmark output can be realized simultaneously.
[0098] The accuracy testing device provided by the present invention is described below. The accuracy testing device described below and the accuracy testing method described above can be referenced to each other.
[0099] Figure 7 is a schematic diagram of the structure of the accuracy testing device provided by the present invention, such as Figure 7 As shown, the device comprises: A designation unit 710, used to determine a target module to be tested in the model, and to determine each computing unit for running the target module; A test unit 720 is used to control the computing units in parallel to perform forward propagation and backward propagation on the target module based on a reference input of the target module to obtain a test output of each computing unit, wherein the reference input is a training output of a previous module of the target module, and the training output of the previous module is obtained by training the model; A comparison unit 730, configured to determine the test accuracy of each computing unit based on a benchmark output of the target module and a test output of each computing unit, wherein the benchmark output is a training output of the target module, and the training output of the target module is obtained by training the model; The output unit 740 is used to determine the test accuracy of the target module based on the test accuracy of each computing unit.
[0100] The device provided by the embodiment of the present invention realizes the precision test for the target module by using the output of the previous module obtained by model training as the reference input of the target module, thereby realizing the independent precision test for a single module, and the test process will not introduce the precision error of the previous module, which can ensure the reliability and accuracy of the precision test of finer granularity. In addition, in the process of precision test for the target module, each computing unit of the target module is controlled in parallel to perform forward propagation and reverse propagation on the target module, which can adapt to the precision test of the module in the model with a large parameter scale, and can realize the designation of the computing unit, thereby further improving the adaptability and flexibility of the precision test method.
[0101] Based on any of the above embodiments, the designating unit is specifically used for: At least one computing communication group for running the target module is determined, the computing communication group comprising a plurality of computing units communicating with each other.
[0102] Based on any of the above embodiments, the testing unit is specifically used for: Based on the input data in the reference input of the target module and the local weights in the reference input allocated to the computing units in each computing communication group, the computing units are controlled in parallel to perform forward propagation and reverse propagation on the target module to obtain the test outputs of the computing units; The local parameters of all computing units in the computing communication group constitute the reference weight of the target module.
[0103] Based on any of the above embodiments, the designating unit is further used for: Determine the target test rounds for the target module to be tested; The benchmark input and benchmark output of the target module under the target test round are determined, and the model training input when the model is trained under different test rounds is different.
[0104] Based on any of the above embodiments, the designating unit is further used for: When the test accuracy of the target module does not meet the accuracy requirement and the target test round exceeds the preset round, determining a test round adjacent to the target test round as a new target test round; An accuracy test is performed based on the benchmark input and benchmark output of the target module in the new target test round.
[0105] Based on any of the above embodiments, the testing unit is further used for: Based on the environmental variables and hyperparameters when the model is trained, the environmental variables and hyperparameters of each computing unit for forward propagation and backward propagation of the target model are configured.
[0106] Based on any of the above embodiments, the reference input includes input data and reference weights, and the reference output includes output data and weight gradients.
[0107] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the accuracy test method, which includes: Determining a target module to be tested in the model, and determining each computing unit for running the target module; Based on the reference input of the target module, the computing units are controlled in parallel to perform forward propagation and backward propagation on the target module to obtain test outputs of the computing units, wherein the reference input is the training output of a previous module of the target module, and the training output of the previous module is obtained by training the model; Determining the test accuracy of each computing unit based on a benchmark output of the target module and a test output of each computing unit, wherein the benchmark output is a training output of the target module and the training output of the target module is obtained by training the model; Based on the test accuracy of each computing unit, the test accuracy of the target module is determined.
[0108] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0109] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the accuracy test method provided by the above methods, the method includes: Determining a target module to be tested in the model, and determining each computing unit for running the target module; Based on the reference input of the target module, the computing units are controlled in parallel to perform forward propagation and backward propagation on the target module to obtain test outputs of the computing units, wherein the reference input is the training output of a previous module of the target module, and the training output of the previous module is obtained by training the model; Determining the test accuracy of each computing unit based on a benchmark output of the target module and a test output of each computing unit, wherein the benchmark output is a training output of the target module and the training output of the target module is obtained by training the model; Based on the test accuracy of each computing unit, the test accuracy of the target module is determined.
[0110] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the accuracy test method provided by the above methods, the method comprising: Determining a target module to be tested in the model, and determining each computing unit for running the target module; Based on the reference input of the target module, the computing units are controlled in parallel to perform forward propagation and backward propagation on the target module to obtain test outputs of the computing units, wherein the reference input is the training output of a previous module of the target module, and the training output of the previous module is obtained by training the model; Determining the test accuracy of each computing unit based on a benchmark output of the target module and a test output of each computing unit, wherein the benchmark output is a training output of the target module and the training output of the target module is obtained by training the model; Based on the test accuracy of each computing unit, the test accuracy of the target module is determined.
[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A precision testing method, characterized in that: include: Determining a target module to be tested in the model, and determining each computing unit for running the target module; Based on the reference input of the target module, the computing units are controlled in parallel to perform forward propagation and backward propagation on the target module to obtain test outputs of the computing units, wherein the reference input is the training output of a previous module of the target module, and the training output of the previous module is obtained by training the model; Determining the test accuracy of each computing unit based on a benchmark output of the target module and a test output of each computing unit, wherein the benchmark output is a training output of the target module and the training output of the target module is obtained by training the model; Based on the test accuracy of each computing unit, the test accuracy of the target module is determined.
2. The accuracy testing method according to claim 1, characterized in that: The determining of each computing unit for running the target module includes: At least one computing communication group for running the target module is determined, the computing communication group comprising a plurality of computing units communicating with each other.
3. The accuracy testing method according to claim 2, characterized in that: Based on the reference input of the target module, the computing units are controlled in parallel to perform forward propagation and reverse propagation on the target module to obtain the test output of each computing unit, including: Based on the input data in the reference input of the target module and the local weights in the reference input allocated to the computing units in each computing communication group, the computing units are controlled in parallel to perform forward propagation and reverse propagation on the target module to obtain the test outputs of the computing units; The local parameters of all computing units in the computing communication group constitute the reference weight of the target module.
4. The accuracy testing method according to claim 1, characterized in that: Based on the reference input of the target module, the computing units are controlled in parallel to perform forward propagation and reverse propagation on the target module to obtain the test output of each computing unit, which also includes: Determine the target test rounds for the target module to be tested; The benchmark input and benchmark output of the target module under the target test round are determined, and the model training input when the model is trained under different test rounds is different.
5. The accuracy testing method according to claim 4, characterized in that: The method further comprises determining the test accuracy of the target module based on the test accuracy of each computing unit, and then: When the test accuracy of the target module does not meet the accuracy requirement and the target test round exceeds the preset round, determining a test round adjacent to the target test round as a new target test round; An accuracy test is performed based on the benchmark input and benchmark output of the target module in the new target test round.
6. The accuracy testing method according to any one of claims 1 to 5, characterized in that: Based on the reference input of the target module, the computing units are controlled in parallel to perform forward propagation and reverse propagation on the target module to obtain the test output of each computing unit, which also includes: Based on the environmental variables and hyperparameters when the model is trained, the environmental variables and hyperparameters of each computing unit for forward propagation and backward propagation of the target model are configured.
7. The accuracy testing method according to any one of claims 1 to 5, characterized in that: The reference input includes input data and a reference weight, and the reference output includes output data and a weight gradient.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the accuracy testing method as described in any one of claims 1 to 7 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the accuracy testing method according to any one of claims 1 to 7 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the accuracy testing method according to any one of claims 1 to 7 is implemented.